You didn't say how far out you were when this started, nor the field elevation of your home airport, so the information isn't there to simply figure out time, speed, distance, and descent rate to see if you were doing something sane or dumb.
First things first: What the tower told you to do, has zero bearing on it. You're the pilot. Don't ever use what a tower told you to do as an excuse for poor piloting and planning. Aviate. Always.
Lets assume the worst on your numbers, and do some head math.
Let's just say you wanted a sedate 500 ft/min descent, and your field is at sea level.
You needed to start your descent 11 minutes out. 5500/500=11.
Doesn't matter what speed you're going. 500 ft/min to a 0 MSL airport from 5500 MSL takes 11 minutes. Period.
Simple head math. Always look for ways to make it simple head math. KISS.
Now if you have a GPS, that's a no-brainer. It'll tell you how many minutes until you arrive. But you don't so let's figure it out.
You need the math for your ground speed to see how much distance you'll cover in 11 minutes.
(If you know you have light winds, your airspeed will be a close enough SWAG.)
Now let's make the head math easy for speed, first.
Always choose methods you can do in flight with almost no brain cells operating and expand on it.
60 knots. (It makes the math easy to show. Watch.)
At 60 knots, you'll fly one minute per nautical mile.
You know you need 11 minutes to descend at 500 ft/min.
At 60 knots, you'll cover ... 11 nautical miles.
At 120 knots it takes 1/2 a minute per nautical mile. Or two nautical miles per minute, whichever way is easiest to remember for you.
Double the distance for the same amount of time. Time is fixed here because we want to come down at 500 ft/min. So...
At 120 knots over 11 minutes, you'll cover twice the distance: 22 nautical miles.
So unless you were 22 nautical miles out, you weren't going to do a nice 500 ft/min descent at 120.
Bzzzt. Unable.
(You also mentioned the yellow arc of your airplane but didn't say how fast that is in your plane.)
Ok. Let's try coming down at 1000 ft/min to help the controller out, and hopefully not hurt our ears...
Now you only need 5500 feet/1000 feet per minute=5.5 minutes to descend. The number of minutes now doesn't change for the rest.
5.5 minutes at 60 knots: 5.5 nautical miles.
5.5 minutes at 120 knots: 11 nautical miles.
5.5 minutes at 180 knots: 17.5 nautical miles.
You need to be that far out to make it down to 0 MSL from 5500 MSL at each of those speeds.
See how this works? Figure out the descent rate you want and get the number of minutes that will take and lock the minutes in your remaining calculations if your GPS doesn't have an ETE to the final waypoint, the airport.
Groundspeed just determines how far away you have to start to do it, because it changes how many miles you're covering in a minute.
You can tell quick and easy if you need to come up with a better plan.
Bonus points: Pattern altitude is generally 1000 AGL so you only need to get to the pattern altitude most times when you're doing this calculation.
In your scenario you're headed for the runway altitude though.
For most of what we all fly 120 knots is plenty fast and you don't need to be going faster than that.
And 1000 feet per minute is as fast as you typically need to come down.
So the rule of thumb is:
AGL / 1000 feet per min. Gives you minutes.
If you're 120 knots or slower, you're doing 2 nautical miles per minute or slower.
You need a minimum of [minutes needed to decend] * 2 = nautical miles to make that descent at a reasonable rate and speed.
Controller asks you inside that distance, you need to extend, make a pattern, make S-turns, whatever ... to get that much distance between you and the airport.
Pick a descent rate and lock it. Then the math becomes easy to do in your head is you know 60 knots is one nautical mile per minute. 120 knots is two. 180 knots is three.
Comes down like a lead-coated brick.