Well, I tried the step bits, but the PVC I'm drilling is too thick. I end up getting a larger hole than I wanted on the outside, and the inside gets the size I wanted.
I picked up some spade bits on sale at Home Depot. They worked pretty well. Instead of having a point, and then a flat cutting surface, the cutting surface points down at the ends. This scribes a line in the plastic before it actually cuts. This reduces the chipping.
An you guys are right. Slow and steady is the way to go.