I'm not calling it a privilege, that is in fact what it is. The government could and has grounded aircraft and pilots with and without just cause, and has the regulatory authority to do so. Yes, if you comply with the government's requirements, they will grant you a medical and/or pilot certificate, both of which are subject to suspension or revocation by the same government who issued them.
Likewise, the proven risks of operating an aircraft are far greater than perhaps with any other vehicle. Yes, pilots do have medical events and crash, sometimes endangering or fatally injuring passengers. There is no coasting to the side of the road in an aircraft. Accident statistics for light GA aircraft are far worse than just about every other form of transportation, which reflect the presence of the additional risk.
Given the substantial increase in risk which accompanies flight, the medical suitability of the pilot is material the determination of whether a pilot may safely operate an aircraft, especially since that operation may involve passengers. There are multiple reasons why the same standard does not apply to automobiles, among them the substantially reduced risk involved with their operation, and political pressure to maintain a privilege to which many feel entitled.
Regardless of varying opinions on this topic, I don't see the 4th Ammendment issue here.
JKG
Until a Federal judge intervenes:
FAA: "This guy said mean stuff about us, so we denied his ticket"
Judge: "You have 30 days to show cause or issue."
Admittedly, probable a very abbreviated version, I know.
And you gotta have the money to get 'em in court. But no, as I said, I believe the FAA is legally, statutorily, bound to issue unless there is cause not to. And they'll do what they're told by a court. And they do, of course, otherwise they'd bring down a steel storm from elected reps and the courts, if they were routinely capricious and arbitrary - which did get them the PBOR. And reversals (occasionally) by the NTSB. The FAA is never the final arbiter, unless you choose not to pursue remedy. Or can't afford to. And when they are reversed, it's by a Fed entity, as I noted - same gov't.
And people have also been killed by lightening and meteors, but we don't require lightening rods on every tree, or a anti-meteor missile system. At some point, the stats have to support the effort. And money. And intrusion. And crashes from medical incapacitation are about as rare, or rarer, than those events, in low-end GA. It just doesn't happen. Don't quote me, and I'm sure there have been some, but I believe I read there have been intervals of several years, or more, between medical-induced crashes in low-end GA.
I concede the fourth amendment may not be compromised (I'm not really qualified to have an opinion on that) - I can't concede there is any significant additional risk to Grandma if I take her up on Sunday afternoon in our 172, without a III Class. Or any medical at all, actually.
If I fly 175 hours this year, that's about 2% of the hours available. That big blood vessel in my brain may be set to explode, but it's got a lot more opportunity to so somewhere other than my airplane. Not that a III Class would have detected it anyway.