You are soooo much better off using a decent amt of silicone for the mold, then put some little pieces of foam to make the outside of the mold lumpy. Next, use a rigid outer shell to hold the silicone.
I would NOT skimp on the silicone. If it tears, you will spend more time trying to fix it and probably have to start over.
Get a good 1/4 inch of silicone. Brush on the first layer, try to avoid bubbles.