I really like the 182 for a first airplane: There's a crap-ton of them out there, so they're easy to find, easy to buy, easy to maintain, and easy to sell. They aren't the best at anything, but they're pretty darn good at everything.
I own a Mooney M20R Ovation - Obviously, you won't get one of those for $60K, but the older Mooneys are great for efficiency. I think you'd do well with an M20E. It sounds to me like your mission is going to be majority solo/long trips, with occasional family participation. The Mooney will save you a lot of money on gas.
If you can get a bird that can carry the load you want plus four hours of fuel, that is PLENTY. Don't make the family do legs over 3 hours if you want them to come with you again, regardless of what type you're flying.
A few other posters have mentioned it,
@Kristin gave the best detail, but you should definitely consider a Comanche. The PA24-180 would be analagous to an M20C in terms of speed and efficiency, maybe a few knots slower. However, it does have a larger cabin and higher useful load. A Comanche 250 would have operating costs more like those of the 182, but would also give you a much higher speed.
The other thing the Comanche has going for it is price. About a year ago, I helped someone who was looking for a plane in a slightly lower price range than yours, who wanted good cross country ability, and good useful load (they were going to be carrying scuba tanks frequently) find a Comanche 250 for $45K that already had WAAS GPS and ADS-B, and this was well before the market tanked. You can get a lot of bang for your buck here, so well worth considering.