As long as there is something for the primer to key to it makes no real difference.
I tend to fill first then prime, mainly so that everything has an equal coat of primer to ensure a good paint finish.
Some people fill, prime and then fill up anything the prime coat picks out then prime again before painting.
Good luck with your build.