Agreed.
I think you should re-read 91.123 again, especially paragraph (b).
91.123(a) addresses clearances, but (b) applies any time an ATC instruction is given, and that's a different thing than a clearance. If you are instructed to "advise prior to any altitude change," then you are obliged to do so if you are "in an area in which air traffic control is exercised," which includes all controlled airspace plus the G-space around towers with no E- or D-space, and there is no exception for aircraft not operating IFR.
Read 91.123(b) again -- if you are talking to ATC "in an area in which air traffic control is exercised," absent an emergency situation, you are legally obligated to comply with any ATC instructions received, and there is no exception based on VFR vs IFR.
It's black-letter law, folks.