You're missing something. Something pretty big.
Your physician does not report anything whatsoever to the FAA. Only to you. And you don't report any medical info to the FAA, either. Your doc signs a form and you keep it in your logbook; it does not go to the FAA. The FAA medical folks are completely out of the picture.
If your physician refuses to sign due to some condition you have, the two of you can agree on a treatment plan that will get his signature, or you can go to another doctor (it doesn't have to be your primary care physician), or you can fly as a SP, or you can decide your doc is right and stay on the ground. But NONE of that goes to the FAA! You cannot get a denial out of a Basic Med exam like you can a class 3.
You could go get a Basic Med exam today, complete a free online class, and be medically legal to fly with zero risk of a denial, zero risk of losing Sport Pilot privileges.
That seems like a pretty big advantage to me.