Nationwide Verizon 4G/LTE outage

TangoWhiskey

TangoWhiskey

Touchdown! Greaser!
Joined
Feb 23, 2005
Messages
14,210
Location
Midlothian, TX
Display Name
Display name:
3Green
http://www.gottabemobile.com/2011/12/07/verizon-suffering-4g-lte-outage/

Verizon 4G down nationwide; and if you have an LTE phone, it's not failing back to 3G like it should. NO data. In fact, if I try to call people, that doesn't work, either, and tells me I will be connected with an operator so I can make a call with a credit card. Glad this doesn't happen every day! Their service is usually rock solid. I remember only one other widespread outage in the past 10 years of service with them.
 
I guess "nationwide" means "spotty nationwide"? Here's what I got on my phone when it went down:

attachment.php


The in-Verizon-store reps thought my SIM card had failed, and replaced it before they found out that "everything is down".

I'm not the only RAZR user getting that error:

http://www.phonesreview.co.uk/2011/12/07/droid-razr-sim-card-problem-not-authorized-for-lte-service/
 

Attachments

  • ScreenShot064.png
    ScreenShot064.png
    196.2 KB · Views: 141
Last edited:
I've not had any problems in the Atlanta area other than the usual spotty coverage.
 
Update: it is now down. Upload speed of 0.14 Mps and download speed of 0.22 Mps. Blech.
 
TangoWhiskey said:
Somebodies are having a SEV1 bridge and all-hands-on-deck moment.
Click to expand...

I bet I know which conference bridge number and location of the gear it is/was happening on, too... from my "former life"... :)

"Welcome to the Verizon Conferencing Center. Enter your conference ID code followed by the pound sign. Thank you, please state your name after the tone, followed by the pound sign. Thank you. This conference is being recorded. If you do not wish to be recorded, please hang up now. You will now be placed into the conference. There are... X participants already in conference. [Recorded Name]... has joined the conference..." ;)
 
denverpilot said:
I bet I know which conference bridge number and location of the gear it is/was happening on, too... from my "former life"... :)

"Welcome to the Verizon Conferencing Center. Enter your conference ID code followed by the pound sign. Thank you, please state your name after the tone, followed by the pound sign. Thank you. This conference is being recorded. If you do not wish to be recorded, please hang up now. You will now be placed into the conference. There are... X participants already in conference. [Recorded Name]... has joined the conference..." ;)
Click to expand...

Replace "Verizon" with any of several other company names, and it sounds exactly the same. They must all contract through the same ASP.
 
TangoWhiskey said:
Replace "Verizon" with any of several other company names, and it sounds exactly the same. They must all contract through the same ASP.
Click to expand...

Heh... they all bought our gear. And our competitors copied our call flow, which was copyrighted. Which led to a couple of lawsuits...

There's only one major carrier in the U.S. not running that gear.

Now the call-flow has been around so long it's not protected in any way.

ConferTech International -- first company to build a commercial 4-talker audio bridge. They demoed it with a barbershop quartet a few years before I joined them the first time. :)

Later we learned that the most natural-sounding algorithm for conferencing was "two-talker plus interrupter". Two lines were always "hot" in the conference, with any line that became louder (digitally measured) than any other "stole" a "talk-slot". Only two TDM streams needed between DSP cards, and a messages on the shared data buss for "loudness" of all the lines as they changed.

AT&T had their privately built Bell-Labs designed DVB "Digital Voice Bridge" before CTI but it wasn't available for sale commercially and it switched a single "talk-slot" around.

They're STILL in use today, discreet components, they have board houses process piles of the cards for hardware errors. No features, no security... used for their very bottom-dollar applications. Take up a HUGE amount of CO floor-space and power. But by now all the development costs have been paid off for decades, and maintenance is cheap on them... swap boards.

Anyway, the company history was...

CTI acquired by ALS (Allnet)

Allnet acquired by Frontier Communications

Frontier acquired by Global Crossing

(I left to go play in the Dot Com boom in building data centers.)

Global Crossing version 1.0 goes bankrupt

CTI Engineering spins out "equipment manufacturer" division, Voyant Communications

Voyant acquires a couple of competitors

(I came back. The Dot Bomb had blown up in my face fiscally. Tons of fun though.)

Voyant acquired by Polycom

Audio bridge gear for carriers seen as competition to video products sold to Enterprise.

VoIP changing the world...

Audio bridge product instantly end-of-sale, end-of-lifed as fast as law will allow. Support contracts continue for the time being. Not a drop in the bucket in Polycom's overall budget, but significant cash to the Support organization that would be silly to walk away from.

Life in the audio support group got slow, but was still a core of great people who'd been there clear back to the CTI days, mixed with some new folks who were excited about fixing things in video that we'd fixed a decade before in audio.

(Sometimes in meetings when we heard what bugs had gotten fixed/design changes had been made, we just chuckled and thought... "Yep, I remember that problem in 1996 on the audio side...")

I left for the second time. Back to Linux and running around keeping stuff running in data centers in the virtual call center world...
 
denverpilot said:
AT&T had their privately built Bell-Labs designed DVB "Digital Voice Bridge" before CTI but it wasn't available for sale commercially and it switched a single "talk-slot" around.
Click to expand...

Fascinating history. Thanks for sharing. Funny, too.. The company I work for now, a software based full lifecycle data management platform, started life as a Bell Labs internal product.


denverpilot said:
Life in the audio support group got slow, but was still a core of great people who'd been there clear back to the CTI days, mixed with some new folks who were excited about fixing things in video that we'd fixed a decade before in audio.

(Sometimes in meetings when we heard what bugs had gotten fixed/design changes had been made, we just chuckled and thought... "Yep, I remember that problem in 1996 on the audio side...")
Click to expand...

I'll be glad when HDMI video / audio is automatically synched... The new standard supports it, but in a lousy fashion that doesn't work well. When will they start adding 'key frames' to both streams?
 
Not segmenting the network appears to be the biggest engineering design failure. Nobody in telecom runs networks nationwide. Nobody. Even SS7 can handle unintentional segmentation and limp along.

The whole article screams "Data weenies trying to do telecom engineering" to me. Seen it before.

A "backup database" took out live Production traffic? That's a sign something's very badly engineered, too.

Verizon's always kinda been this way internally. They've got massively good procedures on how to handle outages internally, but it's because they NEED them. They're also the most forthright about mistakes, as evidenced by the interview being granted in the first place.

They also kick the living crap out of their staff with on-call duties. Their Unix team members would regularly tell stories of being up basically 24/7 for their week of on-call. Painful.

If you caught an outage as a vendor on-call guy toward the end of the week rotation, your number one priority was to make sure the exhausted, distracted, systems people didn't do something excessively stupid. They partner very tightly with vendors. (Again, evidenced by their unwillingness to name a particular vendor's problems as root cause in public, but guaranteed there's a vendor writing a root-cause analysis document for them and it's due about thirty minutes after the outage ends. Not that they always get it that fast, but that's the expectation.)

They were one of only two carriers who'd completely destroyed a production database, requiring a reload from what backups there were, and the only one to do it more than once.

On the flip side, their staff was far less difficult on a personal level to deal with and would actually discuss things rationally and not start by quoting "policy" when real problems arose. They were also far more up-to-date on industry technical knowledge and far less "stuck in the past" than other telcos.

Remote access and recovery gear was the best in the industry and the most organized. You could see and access everything needed. Some telcos relied on an on-sight tech to plug in a terminal and read and type commands which was god-awful on the vendor side of things.

So in the end, it's a trade-off. VZ has great tech and people but they push the engineering too close to the edge of disaster and get bit by it more often. I use 'em as my personal Carrier of choice simply because they bust butt to recover when down. Other carriers have a more "avoidance of outages" policy internally which works better up to a point, but when a new massive outage hits, they tend to twiddle their thumbs for a while and then act, which usually takes longer in the end.

Those conference calls must have been interesting.
 
VZN is my carrier of choice for the same reasons. When I call their FIOS support line (rarely), I get a guy who speaks English and knows the equipment he's supporting. And when it becomes clear to the Tier 1 guy I know what I'm talking about, I get put through to Tier 2 or 3 quickly, and after about 5 minutes the problem is solved.
 
TMetzinger said:
VZN is my carrier of choice for the same reasons. When I call their FIOS support line (rarely), I get a guy who speaks English and knows the equipment he's supporting. And when it becomes clear to the Tier 1 guy I know what I'm talking about, I get put through to Tier 2 or 3 quickly, and after about 5 minutes the problem is solved.
Click to expand...

Back a while ago, when I was doing a lot of Novell server work, I had a Novell server set up at home as my firewall (yes, I know...ubergeek).

Well, I called the cable company one day and got the typical Tier One droid and they started asking all the standard stuff...asked what OS I was running and I answered "Netware 6"...total silence on the line...Tier 2 came quickly after that.
 
JeffDG said:
Back a while ago, when I was doing a lot of Novell server work, I had a Novell server set up at home as my firewall (yes, I know...ubergeek).

Well, I called the cable company one day and got the typical Tier One droid and they started asking all the standard stuff...asked what OS I was running and I answered "Netware 6"...total silence on the line...Tier 2 came quickly after that.
Click to expand...

That is a dangerous thing to do. They'll just ask question #9 over and over again like you didn't understand the question.

:D When I'd have the fine, fine "Clint" in Bangalore ask me "Are you using Windows or a Mac?" I'd say "Which would you like?"

2281.strip.gif
 
Last edited:
denverpilot said:
A "backup database" took out live Production traffic? That's a sign something's very badly engineered, too.
Click to expand...

When I read "backup database" I presumed they meant "production failover database" and dumbed it down for the media. Maybe not...

denverpilot said:
Those conference calls must have been interesting.
Click to expand...

Indeed!
 
You must log in or register to reply here.
Back
Top