I see your point.
91.123 (a) is about clearances. (b) is just about following directions.
So while normally when flying VFR you’re not assigned an altitude, and you’re not even legally required to be in communication with ATC if you’re not in B, C, or D airspace and flying VFR, IF you are talking to them and they tell you anything - such as go to this altitude - you get to do that.
So what happens when you switch frequencies and simply stop talking to them?
§ 91.123 Compliance with
ATC clearances and instructions.
(a) When an
ATC clearance has been obtained, no
pilot in command may deviate from that clearance unless an amended clearance is obtained, an emergency exists, or the deviation is in response to a traffic alert and collision avoidance system resolution advisory. However, except in Class A airspace, a pilot may cancel an
IFR flight plan if the operation is being conducted in
VFR weather conditions. When a pilot is uncertain of an
ATCclearance, that pilot shall immediately request clarification from
ATC.
(b) Except in an emergency, no
person may operate an
aircraftcontrary to an
ATC instruction in an area in which
air traffic control is exercised.