Can you fly IFR with only one VOR? Yes. Would you want to? Single pilot, single VOR IFR will be a handful, especially for a typical private pilot in a system that is now GPS-centric, and where many VORs are decommissioned or frequently to permanently "out of service." Personally, I would want more than one functional, IFR-approved nav device in any plane operated IFR. And to be honest, WAAS (IFR) GPS is de rigueur these days. It makes IFR so much simpler and more efficient, and even here in the NE, you can often get direct to many destinations after initial contact.
Near my home airport, virtually all the formerly nearby VORs have been decommissioned due to MON. IFR departures are now directed to a more distant VOR fix that is usually in the opposite direction than the most common directions of flight. It's really annoying. If you have GPS, you can usually get direct destination or a more suitable enroute fix as soon as radar contact. Otherwise, it's off to the distant VOR or vectors until you can get direct to an operational VOR or intercept a functional Victor airway, which are now fewer and farther between in the northeast. GPS is easier, and almost certainly safer for IFR operation, especially for the part-time IFR pilot. I can still fly the old way (how I learned pre-GPS) but it's now relegated to emergency, not routine ops. We do have GPS outages, but I haven't been caught in any of them yet. If GPS is out, and my field is IFR, I'm not getting in, as all we got are GPS approaches. (And that's another utility issue with having only VOR/LOC equipment.)