Class E is an area where "Air Traffic Control" is exercised. It is exercised over pilots on IFR clearances.
In areas where "Air Traffic Control" is exercised over VFR pilots, the VFR pilot, must, under regulation be in communication with ATC.
There is a difference between an "Advisory service" and "Air Traffic Control".
My view is that when a controller gives an instruction that does not comply with the FAA's orders to controllers, that still constitutes air traffic control being exercised.
I'm not saying that my view is in any way definitive; I'm just explaining why, in my own flying, I choose to take 91.123(b) literally.
If the FAA doesn't want VFR pilots to obey ATC instructions in class E, then they really need to clarify the reg, IMO