I know this is a very different kind of use but Ireally recommend this site for care of all latex (though probably doesn't extend to foam latex fully
Latex Care | Making Latex Clothing
I suspect it may well be the paint. It sounds like there was a petroleum product in the paint used. For my latex appliances I am extremely careful about making sure as much is protected from my skin as possible and also it affects what paint/makeup/primer and even facial cleanser I use for fear of this. It's all acrylic and latex all the way.
How bad is it? I know there was a recent discussion (I think in the screen used props section?) Recommending one of the members here as the go to person for repair. Otherwise a quick read of this may help pinpoint the problem:
Latex Mask Central