Latex under normal clothes keeps dragging and bunching

I have started wearing a latex top under ordinary clothes on days when I want it and nobody needs to know about it. Mostly this works and I like it more than I expected to.

The problem is friction. Anything I put over it drags, so a shirt sleeve rides up my arm and stays there, and by lunchtime the top has bunched somewhere it should not be. Taking a jacket off is a whole operation because the lining sticks and pulls the sleeve up with it.

Is there a way round this or is it just the nature of the material? I am not asking to be invisible, I just want to be able to take a coat off in public without a wrestling match.