Good point.
The series has shown that they could beam something far enough away if it threatened the ship (such as Nomad exploding). So the question now becomes: could they beam it far enough away that the creation of an entire planet would not affect the ship?
I guess that depends on the size of Genesis, and exactly how far away are they able to beam something.
According to Mr Scott's Guide to the Enterprise (because I'm pretty sure there is no mention of the maximum range of the transporters in the first two films or the series... so we really have nothing else to go on for reference) the maximum safe range is 19,500 miles or 31,200 kilometers.
The transporter would allow them to beam Genesis over 31,000km away. And while getting away the Enterprise had travel 4,000km at impulse. So say at best they could put 36,000km bewteen themselves and the explosion.
How big was Genesis? Again there is no mention of it in the films, so going on that it appears to at least have the same gravity of Earth, let's say it is about the same size which is 12,742km. How about making it easy and rounding it up to 13,000.
So really they would only have to escape half that distance because they would need to be outside of half the diameter, but let's say to avoid any debris or the shockwave itself, they would have to travel twice that distance- 26,000km.
Exvept there is no real way to know what the "safe" distance truly was, AND the transporters were not working at optimum capacity. Scotty said he "barely" had transporter power before Kirk and company beamed down to Regula, and they had only restored partial main power when they returned.
Maybe they "could" have beamed it far enough away, but it is just as likely that they could not have, and the risk would be too great if they were wrong.
So going on this... I would say that Bryan and Michael are right that "not attempting" to beam Genesis off the Reliant, and concentrating on restoring warp drive "was" the right descision by Kirk!
**** that's awesome!
Kevin