I don't have the regulatory answer but here's how I would do it. The Cessna's of that vintage could by ordered without radios, or 'radio delete'. This was done sometimes for foreign markets. Now, the radios are gone, the speaker could still be ordered installed and the wires stubbed, or it could be a speaker delete too.
To convince the A&P, remove all radios which have an audible output(incl nav, ADF, etc). Remove the speaker, remove the headset jacks. Fill out a 337 like we do for a radio swap - "all radios and trays with audible source removed from SN xxxxxx, speaker and headset removed from aircraft. aircraft weighted and balanced and a new W&B computed." Then, go back and put what ever radios you want in, and install the headset jacks and not the speaker and fill out the 337 just like a radio install - "Installed tray and radio blah blah IAW blah blah installation instructions, and installed headset jacks in panel to audio source(s). Compute new W&B."
No reason to have a speaker in a plane which now has no audio devices. Then, go back and install what you want on the 337 just as you did before. Now, you're under the stipulations of the radio install document. If it requires a speaker(doubt it), then you put a speaker in, if it doesn't, then don't.