There are reasons why there are restrictions to baggage and those reasons are based on physics, available space, and other practical concerns, not some arbitrary decision by the FAA, airline, or airline employees.
Each certificate holder (airline) does write their own carry-on baggage program but it must be one that the FAA will accept. If you say that each passenger can bring an unlimited amount of bags into the cabin then you can't use the standard 190/195 pound passenger weight. Either your plan will be to use higher standard weights or use actual weights for passengers and bags. What the FAA will accept is generally known and this process likely starts with choosing one of the known templates.
One area where we do see baggage programs that differ noticeably is in the 50-seat RJ category as they are more susceptible to weight restrictions when at, or near, their maximum seating capacity and have less overhead space per passenger. Instead of using the standard 190/195 passenger weights, which allow both a personal item and a carry-on bag, some operators have chosen to use a plan that excludes wheeled carry-on bags from the cabin in exchange for a lower per-passenger average weight and a lowered standard weight for each gate-checked carry-on as compared to the standard weight for a normal checked bag. Whichever method the airline chooses, and FAA accepts, is enforceable under 14 CFR 121.589 just as any other FAA regulation.
There is no option that allows gate agents and flight attendants to make different decisions on carry-on baggage on a per-case basis. The employees are required by 14 CFR 121.589 to comply with the carrier's accepted carry-on baggage program. Very few front line employees will know the details of this process. They will only know the rules with which they must comply and many will correctly assume that their compliance is required by FAA regulations.
Gate availability is an issue at all hubs at all airlines. I've also sat and waited for gates at non-US airlines where their airport authority, not the airlines, have control over gate assignment. Larger airlines typically have a dedicated employee to manage gates at their hubs.
I'm attaching a picture of the Gate Manager's screen from a morning UA bank at DEN. Each gate is listed along the Y-axis with time along the X-axis. The horizontal bars indicate the aircraft at, or scheduled to be at, each gate. Color codes show the type aircraft as each gate can only support certain aircraft types. There's also notations for gates with inoperative components (ground power unit or preconditioned air) and one shows a gate blocked (between banks) for repainting. While I was watching, the Gate Manager was non-stop working on reallocating gates, and making all the arrangements for personnel, equipment, baggage, and cargo rerouting due to gate swaps, so as to use the available gates as efficiently as possible.
One reason why your connecting flight may not be held is that an arriving flight was already there waiting for that gate so those passengers can deplane and make their connections.
One way that my airline works to manage gate space is managing early arrivals. Shortly after takeoff, our gate message (from ACARS) shows the time that our gate is scheduled to become available. When we are running early we can slow down so as to not arrive well before our gate is scheduled to be open. I've even received ACARS messages from the destination's station operations control when we're forecast to arrive before our gate will be available to ensure that we are aware that there'll be nowhere to park us at our ETA.
That helps to reduce the number of early arrivals driving gate swaps but you still have to deal with late departures occupying gates beyond their scheduled departure time. This comes from mechanical delays, waiting for late passengers and bags, ATC delays, late inbound aircraft or crew, and all of the other delay causes. As you can see from the Gate Manager's display, there isn't a lot of leeway in the gate allocations to absorb disruptions before it starts to affect gate availability for other flights.
View attachment 91305