I think some of it is a matter of two (or more) perspectives going on.
Slow driver thinks "What's your point in driving faster than me, we'll both get to our destinations within a minute of each other, and your way is more dangerous. I'll just shake my head at you as we arrive at the stoplight together."
Fast driver thinks "I feel comfortable going five over, if you can tell you're holding me up, will you please move over?"
They are both right; but the difference is in the perspective. Think of it from a pedal bike perspective; you are pedaling along at 15 mph. You catch somebody who is more comfortable (and safer!) only driving 10 mph. You communicate "On your left!" to the other pedal biker. They hear you, and move over to the right. AWESOME! That's just how it's supposed to go. They don't give you the finger, they recognize you wanted to go faster, you're taking the danger of said action into your own hands. You could possibly head-on hit another biker, or crash on gravel, but it's not 10 mph guy's job to police the bike trail. They are courteous and move over. Everybody is happy.
If the pedal biker tried to block you, and pedaled along at 10 mph along with somebody in the right side of the path, gave you the finger and yelled "EFF YOU!!!! What's your damned hurry? We'll get to the end of the path within minutes of each other!", you might respond "It's not about getting there one minute early, it's that I feel comfortable and gives me sanity traveling at 15 mph".
Now back to cars; you can't yell "on your left". You can't flash your lights (VERY offensive, don't you know), you can't honk. All you can do is catch up to the "mobile roadblock", and hope the guy in the left lane sees you catching up, and has some sense of courtesy to get out of your way. That's your way of saying "on your left". If they don't notice you, or don't care to go faster (for one reason or another), you have no recourse. There is no courteous way to say "on your left", at that point. Again, flashing lights, getting close to the guy, honking, all cause the slow guy to rage, and he/she will now try to punish you for being a jerk.
It's not about getting somewhere quicker, it's that one person has maybe a more capable vehicle, better tires, brakes, etc, and feels more comfortable going just a teensy bit faster than you'd care to go. Nothing wrong with that.
If somebody catches up to me, even one person, I'll speed up before they get to my rear end, if possible, and get in the right lane and let them by. I DON'T want them on my tush, I have nothing to gain by keeping them there. In my experience, if somebody caught me, they want to go faster than me, I move over, let them by, and never see them again.