If you have to think about it more than 3 minutes, don't go.
I'm with Shepherd. If you have to think about it, don't go. It's easy and prevents making bad calls. Sure there are some days that you will end up sitting on the ground when you could have completed your flight, but, to quote an overused saying, "it's better to be on the ground wishing you were in the air than in the air wishing you were on the ground".
Set your own personal minimums and stick to them.