Well, there are those who believe that when a non-current pilot manipulates the controls, he becomes PIC and this, the current pilot is now "merely a passenger..."
I agree with you, my local FSDO agrees with you, another FSDO I called isn't sure and didn't want to commit to answer the question yet.