Being the sole manipulator of the controls does not automatically result in you being considered the PIC. Think of a trainee flying with an instructor, or an airline co-pilot flying his/her leg. It does, of course, permit you to log PIC time (as long as you are rated in that aircraft), but it doesn't make you the PIC and it doesn't result in you being "considered" the PIC. There are many circumstances in which someone would be the sole manipulator of the controls while flying under the hood in which it would not be possible for you to be considered PIC. Examples include a pilot who is rated in the aircraft but does not have instrument privileges, or a pilot who does not have a current medical or flight review. In both cases, the safety pilot would have to be fully PIC-qualified
and acting as PIC as well as acting as safety pilot for the flight to be legal.
Logging PIC time does not require you to be the PIC. For example, I was flying today with a PP-ASEL in his Tiger on his long IFR XC for his instrument rating. As the sole manipulator of the controls of an aircraft in which he is rated, he properly logged the entire flight as PIC time. OTOH, I as an authorized instructor giving him training and the only pilot in the plane qualified to be the PIC under IFR would most definitely the PIC by the FAA and the NTSB.
Now that is correct -- if the safety pilot is PIC-qualified and acts as the PIC, then the safety pilot is that final authority responsible for the flight and gets to log PIC time for the entire time his/her presence is required (i.e., while the pilot flying is hooded).
For anyone other than an instructor giving training, that is correct. But the reason the pilot flying is considered the PIC in that case is not that s/he is the sole manipulator of the controls, but rather that s/he is the final authority responsible for the flight regardless of whether s/he is manipulating the controls or not. IOW, s/he would remain the PIC even if s/he allowed the non-PIC safety pilot to manipulate the controls for part of the flight.
The fundamental barrier to keeping all of this straight is the being the PIC and logging PIC time are different issues governed by different regulations, so there are times when you can be the PIC without being able to log PIC time, and times when you can log PIC time without being the PIC. Once you separate these two issues in your mind (which I admit isn't easy to do thanks to the FAA's insistence on using the same words "pilot in command" to describe two different things -- the person in command, and a form of aeronautical experience), it's a lot easier to understand the rules on these issues.