I had a chat with my CFI today. We discussed the possible cause and determined it's my lack of focusing on a distant point, coupled with anticipating p-factor before it happens and immediately pressing right rudder, followed by left rudder. Overcompensating.
One of the posters here (forget who at the moment) suggested to simply start the takeoff and not do any rudder until I start to drift, then only compensate minimally. If I start to drift too far to the right, ease up on the right rudder and let p-factor bring me back toward the left. Basically, it appears that my nerves are kicking in and I'm overcompensating with the opposite rudder and when the swerve happens suddenly, I freak and revert to what I know, and that's trying to steer like a car.
We're going to work on this and practice until it's easily done every time.