It's much harder to judge altitude and relative position when you can only see directly in front of you. It's actually one of the things that makes simulator dead-reckoning flying harder in the sim than in real life (even though you can pan around for your view). I'm only slightly jealous of folks who have multi-monitor setups for the sim.
TrackIR, my friend.