So, I was wrong about the controller being the decision point. Someone posted a reference making it clear that it's 100% on the pilot.
That said, the controller is altering the flight rules. They are to give IFR priority, but it seems like it's also true that once they've given the SVFR to the first pilot, they will have to delay the IFR traffic. So, if the controller knows about both, the IFR gets priority. But if the SVFR is already in effect, they are going to have no choice but to deconflict the IFR traffic with a delay.
7−5−2. PRIORITY
a. SVFR flights may be approved only if arriving and departing IFR aircraft are not delayed.
EXAMPLE−
1. A SVFR aircraft has been cleared to enter a Class B, Class C, Class D, or Class E surface area and subsequently an IFR
aircraft is ready to depart or is in position to begin an approach. Less overall delay might accrue to the IFR aircraft if the
SVFR aircraft is allowed to proceed to the airport and land, rather than leave, a Class B, Class C, Class D, or Class E
surface area or be repositioned to provide IFR priority.
2. A SVFR aircraft is number one for takeoff and located in such a position that the number two aircraft, an IFR flight, cannot
taxi past to gain access to the runway. Less overall delay might accrue to the IFR aircraft by releasing the SVFR departure
rather than by having the aircraft taxi down the runway to a turnoff point so the IFR aircraft could be released first.
NOTE−
The priority afforded IFR aircraft over SVFR aircraft is not intended to be so rigidly applied that inefficient use of airspace
results. The controller has the prerogative of permitting completion of a SVFR operation already in progress when an IFR
aircraft becomes a factor if better overall efficiency will result.
b. Inform an aircraft of the anticipated delay when a SVFR clearance cannot be granted because of IFR traffic.
Do not issue an EFC or expected departure time.
PHRASEOLOGY−
EXPECT (number) MINUTES DELAY, (additional instructions as necessary).