I'm sure other posts you've found the general pattern of things, but just in case, I would think about this from a few different directions:
First, the medications themselves. The FAA asks "Do you currently use any medication?" and then requires you to list them. In my understanding (though others may disagree), if you never took the SSRI, you don't have to list that. And if you no longer take the Xanax, especially if you no longer have a current prescription for it "as needed," I wouldn't list that either. That being said, assume that if you are investigated for any reason, those prescriptions will come up, and the burden will be on you to prove that they're not medications you currently use.
The bigger issue is that the FAA asks "Have you ever in your life been diagnosed with, had, or do you presently have . . . mental disorders of any sort; depression, anxiety, etc." If you've been prescribed Xanax for flying, there's very little chance your doc didn't diagnose you with (at least) situational anxiety and (at most) generalized anxiety disorder. Similarly, if you were prescribed an SSRI, you were probably diagnosed with some sort of depressive disorder, likely (just because it's easy to write down) major depressive disorder but maybe (if you were lucky) adjustment disorder with depressed mood, sometimes called "situational depression."
So let's take those in reverse order: The SSRI issue probably isn't a big deal. Depending on your AME, it might be easier or harder to satisfy them that this was a single episode not requiring pharmacologic treatment. But if you were able to do so, the guide for AMEs allows AMEs to issue medicals in the office to people with adjustment disorder or minor depression who have never required medications (or required medications for less than 6 months, and now have been off meds for 3 months). There are two other big potential "gotchas" here -- even if your AME approves and issues the medical, they send the records along to the FAA, which might turn around and say "wait a minute, the original diagnosis record said major depression, and now the AME is issuing for adjustment disorder; we need to look very deeply into this" which could cost both time and money. Finally, one thing to have in the back of your mind is that someone (the AME, the FAA, who knows) could say "wait a minute, this person was prescribed an SSRI and then didn't take it? Sounds to me like a patient with a psychiatric disorder who didn't follow through with prescribed treatment." While I'm sure we'd all agree that's a ridiculous framing, it only takes one person to think/say it before things become practically impossible.
Finally, as many others have said, the big/weird one is the anxiety/Xanax issue. Even if you don't have to report the Xanax, you'll have to report whatever anxiety diagnosis came with it. Tellingly, the little chart of things AMEs can issue doesn't include anxiety. One comment on anxiety in the AME guide says:
Although they may be rare in occurrence, severe anxiety problems, especially anxiety and phobias associated with some aspect of flying, are considered significant. Organic mental disorders that cause a cognitive defect, even if the applicant is not psychotic, are considered disqualifying whether they are due to trauma, toxic exposure, or arteriosclerotic or other degenerative changes.
I suspect there's almost no way to get a medical with a history of anxiety requiring medication in situations related to flight. I say "almost" because the only way I could think of would be to spend many many thousands of dollars to prove that whatever anxiety you once had, you no longer have. But I don't know of any commonly offered way to prove that. Short of taking a test flight with the federal air surgeon while hooked up to vital sign monitors, I don't know how it would happen. Maybe someone out there does.
That leaves you with a few options:
1. BasicMed - mentioning first because it would be the most convenient, but probably doesn't apply to you. If you're 44 now (2021) and got your last medical when you were 19 (1996), it was probably good until 2001, which is still too early for the BasicMed cutoff of 2006.
Edit: Just re-read your post and saw you were actually never issued a medical. So this option is out.
2. Sport Pilot - this is probably the big winner. You can look up the restrictions (2-seat airplanes with a speed and weight limit, day VFR only), but lots of people are very happy flying this way.
3. Gliders, balloons, ultralights or other "non-medical" options - These are out there too, and some "self-launching motor gliders" aren't much different than light airplanes.
4. Fly dual - maybe / probably not an option long term, but if you last flew when you were 18 and got anxiety around it and now you're 44 and don't know how you would feel about it, maybe fly several lessons just to see if you like it. Training hours could still go towards a sport pilot license later. Or, maybe you'll find that you don't have much use for a license, but want to take a "fly with an instructor or other pilot" fun splurge every once in a while.
5. Lie - I certainly don't advocate this, but you'll find people who say "just find an AME, don't tell them any of the above, and hope for the best." Let's be honest, you probably won't get caught for a good long while. When you do get caught, two things will happen: first, you'll lose your medical and all of your pilots' licenses. Second, and more important to some, your insurance policy will probably say "you're out of luck." So if the way you "get found out" is to bump into something with your plane, it will be incredibly expensive. There's also a possibility that the FAA will fine you or the DOJ will get interested enough to try to send you to jail, but that doesn't seem to happen unless there's something else going on.
