You should have been along for my PPL checkride - you'd have seen it then.
I hear that- I was lucky enough to encounter a little rain and snow in otherwise good VMC during my training... and the closest I've ever come to really running afoul of stuff I shouldn't have been out in was during my first solo XC.
To the OP, I'd also recommend doing some pattern or known practice area work in such conditions, or on days when the wind is tricky... that might help break through the barrier that your good common sense has built up for you.
The other stuff mentioned above is good advice- slow down if you end up lower and in poor visibility, forget your planned destination if it doesn't look good in that direction, etc.
In the end, based on my own experience, for better or worse, I think most pilots don't really get to expand their comfort zone by design... it just happens. It usually works out, though, with good training and good (decision-making) habits.
The idea is not to say "what the hell, I'm gonna go for it" so much as to consider beforehand "what are my options?". Presented with the same risks, the latter philosophy will serve you better, especially if you have been aloft long enough and flown far enough to find very different conditions than when you took off.
If the thought of being foolish about wind or weather scares you, that is good. But the more you consider any wind or weather-related decision, the less foolish it will be. Always have an out, and never let the "mission" force your decisions, and you should expand your comfort zone safely.
I don't have a lot of time logged, but I am at a point where there's no anxiety or drama about going to "have a look" if the go/no go decision isn't obvious before departure. But I also I don't get bent out of shape if I have to abort.
It helps to have minimal expectations when it comes to traveling by air. I don't care what you fly or what you know about flying, if your plans require an iron-clad guarantee that you can get where you want to go, on schedule, despite all odds, you are asking for trouble... and you will find it eventually.