When I was making miniature wargame buildings I used a mix I came up with that basically had white glue, latex paint, and then a texture that ranged from sifted dirt, unsifted dirt, sand, to kitty litter, depending on how rough I wanted it. If I wanted it a little thicker/less defined I would add Durham's Water Putty. Anyway, something like that would give you the texture you're looking for. It's much more uneven, like real stone or concrete, rather than a uniform surface that you get from flocking with sand or whatever. I stippled it on, but on such a big surface I'd probably go with a roller
-Rog