On this subject, it's fortunate that it's not up to the states. In Shapiro v. Thompson, when the Supreme Court summarized previous rulings in various cases, they quoted many precedents that referred to a right to travel, and they concluded that
"This Court long ago recognized that the nature of our Federal Union and our constitutional concepts of personal liberty unite to require that all citizens be free to travel throughout the length and breadth of our land uninhibited by statutes, rules, or regulations which unreasonably burden or restrict this movement."
Shapiro v. Thompson
I'm not an attorney, but I think that most people recognize that testing drivers (and pilots) for competence, and requiring them to obey properly adopted laws and regulations, are reasonable. Consequently, I think driving and piloting easily fit within the scope of what the Court was talking about above.
Whether you prefer to call travel a right or a privilege, if the government made decisions about who may drive a car or pilot an airplane that were sufficiently arbitrary for the Court to recognize them as unreasonable, my bet is that they would have no trouble invalidating them under precedents like the one quoted above.
Calling things "privileges," especially things that are as basic to modern life as driving, makes me nervous, because I think it's an invitation to arbitrary government.