I did some research and it seems like silicone mold rubber can't handle UV light, but I wanted to confirm before completely writing off the material for this project.
Edit: The only UV stable rubbery material I could find is Clear Flex from Smooth On.