It's not the poor vis, it's the pilots... if I'm not mistaken, "pilot-related" mishaps are most common among fatal VFR accidents, and in that subgroup continued flight into not only legal IMC but just plain poor vis conditions is the #1 pilot-decision-related cause of fatal accidents.
Flight into weather unsuitable for the plane, pilot, mission, or all three is also the leading cause of fatals among instrument-rated pilots.
Poor vis may be tempting when it's a bit cruddy but you are doing fine at first ("gee, this isn't so bad- what's the big deal?"), but if you miss the chance to make the right call and end up in stuff you can't handle, it's your own damn fault. Pilot error... sometimes starting the plane and deciding to go is the error that kills. The other factors, even equipment malfunctions, are just stage props, to set the scene. But it's always a comedy of human errors that leads to the tragedy.
I believe the number one cause of any airplane-bending accident, injuries or no (meaning a larger number of accidents than vis-related mishaps), is adverse wind conditions (during takeoff and landing)... but again, if you just don't try to land in that crosswind, the crosswind is no threat to you.
The weather is not your enemy, unless you're a fool... the cleverest humans with the best gear can easily be defeated by weather. It's not your friend, either... it's just the weather, and it will never care about you or your plans. But it cannot plot, scheme or reason, whereas we can. We can decide, for example, to not make that trip, or land at that airport.
If I ever land in a crosswind or gusts that I should know will leave me off the runway with a damaged airplane, somebody smack me unless there was an onboard fire, structural damage, or not a single drop of fuel left.
As for flight into IMC, I would really have no excuse at all... as the OP said, nothing is more important than getting on the ground in one piece!
The greatest peril any pilots face is their ability to deceive themselves about that, IMHO.