I haven't been scared too badly by anyone else - It's a VERY rare thing to find me in a GA airplane and not at a control station. The only time I can think of right offhand was with Tony Condon and Matt Sawhill occupying the front seats of the 182RG. I don't trust anyone 'til I've flown with 'em! Luckily, being 6'4" gives me a really good excuse to sit in the front seat - I don't fit well in the back of very many GA airplanes.
That said, I've never had to take over from anyone. I guess the people that give me any reason to not trust them, I just don't get in the plane in the first place.
As far as scary moments in a GA airplane where I was in control, I've had a couple:
Once, I flew into a cloud that decided to pick that particular moment to become upwardly mobile - Nary a bump, but my VSI suddenly did a backflip and my altimeter started winding up like crazy. Lesson: Even if I'd had on-board radar, I wouldn't have seen that one coming, as I hit the developing phase, before precip happens. Luckily, that always results in UPdrafts.
The other was out west, where I got into a situation where I picked up some ice and my only out was to make a 180 on an airway. (I didn't complete the turn, as ZLC advised me that I was nearly out of the weather.) I didn't expect ice since it was August, but I had air-filed IFR to get through a particular "pass" (not really a pass in the traditional sense of the word as those aren't big enough for airways to go through). OROCA in the area is 15,300 while MEA on the airway was 12,000 and terrain immediately below the airway rises above 8,000. I was at 12K and on oxygen. I picked up a bit of ice, and immediately started mentally kicking myself because I didn't have any other good out than the 180: Climb wasn't a good option due to already-limited performance and I had no idea how high the tops were, plus didn't want to expose more of the wing to the ice. I couldn't descend because I was at the MEA, I couldn't turn because I was well under the OROCA and there was higher terrain both left and right. Luckily it was only a trace of ice, and it stopped accreting almost right away - It just looked like someone had "fogged a mirror" on the leading edges. A couple minutes later I was in the clear and got one of the best shots of the trip: