Quit now, it's good to know your limitations and respect them. Unless you are flying for a living or on a tight business schedule, you'll never need to go IFR. In 25+years and 2500 hrs I have rarely flown IFR and have gotten through VFR where IFR would not have been a good idea by flying low. This is not without its own risks.
If your flying is for enjoyment, and you look at the book and say, "I don't think I'm going to enjoy this" the you should quit now at it and maintain VFR. I promise you, you won't have fun doing your instrument ticket, it's not about fun, it's about work and getting work done. I would suggest however that you maintain proficiency at doing a level 180° turn and a wings level climb under the hood as a minimum for an escape maneuver.