To clarify it's for 2 adults and 2 growing kids. I'd rather not have to upgrade as the kids get older. Baggage - well you know how women pack.
Then we're talking a Cessna 182 or Piper Dakota as a minimum, and a 6-place will probably be necessary for baggage volume (if not for seating) as the kids get to teen age.
I don't really know how long the trips would be as I've never owned an airplane. If it's sitting at the airport I'm sure I'd find some places to take it.
We'll leave that out for the time being.
Budget - well, whatever I can talk the wife into spending
For arguments sake I'll say under $150k.
How about your annual flying budget? A light simple single like a 172 will probably run you $10K a year or more, and a twin over twice that, with a complex single somewhere in between.
My experience - substantial...
That's a bit vague. Complex? Multi?
but I haven't flown GA in a while.
I'll assume that means you're flying professionally for air carriers or the like, so anything up to a cabin class twin would be within your capability.
Then turbocharging/pressurization probably isn't a necessity.
Experimental a/c - I had not considered because I thought they were a little out of the mission parameters.
I think so, too. Yes, there are a few out there like the RV-10 which might fit the bill, but there are a lot of issues involved in E-AB aircraft ownership which lead me to recommend against one for a first ownership experience.
I've heard good things about a lot of the planes you've thrown out...
How is the payload on the Twin Comanche and the C-310?
With nearly twice the horsepower, the 310 has a good bit more payload, but either should be adequate for your mission.
Any advantages between the Cherokee 6-300 and the Saratoga?
Sixes are older and inherently lighter, giving you more payload. In particular, the newest PA32's with all the modern conveniences, soundproofing, and super-comfy interior have full-fuel payloads down near 600 lb, which may be a bit short for your needs. OTOH, Sixes are noiser, older, and usually less well equipped, although with your budget, you could buy a skimpily-equipped Six and upgrade it yourself.
I like the Senecas but I always thought they had much lower payload then the Aztecs.
Somewhat lower, but not enough so to be limiting with your planned loads. Also, the back door and club seating may be very nice for your family station wagon mission.
How are those two in comparison with maintenance costs?
The Seneca's turbocharged Continental engines are going to be significantly more expensive to maintain than the Aztec's Lycoming IO-540's. Other than that, about the same.
Compared to an older Baron?
Baron maintenance will be about the same as an Aztec or 310.
Pros/cons between the Cessna 206 and 210?
210's are a good bit faster, but will cost somewhat more to operate. 206's have the back door which the 210 lacks, and that may have implications for your family station wagon mission.
As always, if a twin is within your financial and pilot skill reach, I'd suggest, and the Seneca you mentioned seems an excellent fit for your mission. You might also consider a Cessna 303 Crusader, which has much the same capability, although there weren't many built. Both have turbocharged engines, which dramatically increases your single-engine service ceiling, and that's a significant safety advantage. The 310/Baron/Aztec class seems a bit more than you need. On the single-engine side, the Cessna 206 and Piper Cherokee Six seem good choices for fixed gear, with the Cessna 210/Piper Lance or Saratoga HP/Beech 36 Bonanza for retractables.
So, how much can you afford, and how much can you fly safely?
I agree with iHenning that not everything would be more expensive but when there are two of some components it will be naturally more expensive. 
Doesn't seem to be much problem on 310 parts. There was an issue with the torque tube for the gear, but I understand that's been solved. Personally I would fabricate my own reinforced one if I needed one.