United grounds all flights

Thanks a lot IT Goobers ! Wish I could make as many mistakes in my job and still stay employed !!!!
 
From the ATCSCC advisory:

ATCSCC ADVZY 024 DCC 07/08/15 UAL AND UAL SUB CARRIERS ONLY GROUND STOP
DESTINATION AIRPORT: ALL AIRPORTS
FACILITIES INCLUDED: ALL FACILITIES
GROUND STOP PERIOD: 08/1200Z - 08/1315Z
REASON: USER REQUEST AUTOMATION ISSUES
REMARKS: GROUND STOP IS FOR UAL AND UAL SUBS ONLY (SKW LOF ASQ GJS
TCF RPA ASH UCA CHQ)
081225-081345
 
Art VanDelay said:
Thanks a lot IT Goobers ! Wish I could make as many mistakes in my job and still stay employed !!!!
Click to expand...


It is a niche that's good to be in.
We break stuff but we are also the only ones that can fix them.
 
Art VanDelay said:
Thanks a lot IT Goobers ! Wish I could make as many mistakes in my job and still stay employed !!!!
Click to expand...


Come give it a try, it's not as easy as you think.

Generally, situations like this are caused by an update to some system software. Network switch and particularly firewall software updates are notorious for causing these sorts of situations, sometimes not every vendor's product plays that well with every other vendor's.
 
(assuming this was caused by an update) What does it indicate about the update evaluation process when fielded systems get cromped?
 
Art VanDelay said:
Thanks a lot IT Goobers ! Wish I could make as many mistakes in my job and still stay employed !!!!
Click to expand...

Not United, but... That kind of revenue loss? I'm sure heads will roll. If I were to cause that much revenue loss, I'd lose my job.

It's a tougher job than a lot of people seem to believe. Who else works 24 hours a day to take care of the little things that everybody's businesses and lives depend on nowadays?

Realistically, when we (and most everybody else) have outages, I can legitimately fault the application(s). The business units want to spend money on infrastructure to hold the line, but that doesn't work. When you refuse to spend money on talent to fix your application, you can't just buy bigger boxes and suggest that your resiliency will improve. Systems are a sum of everything involved! I love buying new and better things, but it's not always the best course of action!
 
Cpt_Kirk said:
You mean like Professional Weather Guessers?
Click to expand...

No. The weather guessers get to blame the weather. We just have to blame those "other IT dweebs!" :D

Computer based systems are, by quite a large margin, the most complex systems mankind has ever built. It is possible to build highly reliable systems by designing for graceful failure, employing line-by-line code inspection, extensive unit testing, and exhaustive full up system testing. But that costs money. LOTS of money. Systems built that way cost orders of magnitude more to build. So that only gets done where the cost of failure warrants it. And sometimes it doesn't get done where it should. :(

John
 
FormerHangie said:
Come give it a try, it's not as easy as you think.

Generally, situations like this are caused by an update to some system software. Network switch and particularly firewall software updates are notorious for causing these sorts of situations, sometimes not every vendor's product plays that well with every other vendor's.
Click to expand...

Bla bla bla bla - quite making excuses and just OWN IT !!! I could crash a 777 tomorrow and not do as much damage as these IT clowns have done ! And if I did crash a jet I'd likely NEVER work again - IT Clowns not so much.
 
IT - do your job well and nobody notices. Screw up just once and the whole world knows.
 
Art VanDelay said:
Bla bla bla bla - quite making excuses and just OWN IT !!! I could crash a 777 tomorrow and not do as much damage as these IT clowns have done ! And if I did crash a jet I'd likely NEVER work again - IT Clowns not so much.
Click to expand...

:rolleyes2:
 
Bob Noel said:
(assuming this was caused by an update) What does it indicate about the update evaluation process when fielded systems get cromped?
Click to expand...

There can be a couple of reasons. It's very difficult to simulate the real world. Unless you have a dedicated lab set up for load testing, you won't really know until you turn the update loose. An outfit as big as United could have such a setup, but that doesn't mean they do. Sometimes this is farmed out to the vendor, and that vendor will not have complete knowledge of everything your company does or has, and they just don't test for the condition that winds up breaking the system. There's also the ever popular senior management says to get it done by this date with this many people, and that's not always enough resources to test everything.

We do make our mistakes, but we're usually able to test the bigger issues out of the system before deployment.
 
Art VanDelay said:
Bla bla bla bla - quite making excuses and just OWN IT !!! I could crash a 777 tomorrow and not do as much damage as these IT clowns have done ! And if I did crash a jet I'd likely NEVER work again - IT Clowns not so much.
Click to expand...


You do realize we can write a program that does your job better than you can.
 
FormerHangie said:
Come give it a try, it's not as easy as you think.

Generally, situations like this are caused by an update to some system software. Network switch and particularly firewall software updates are notorious for causing these sorts of situations, sometimes not every vendor's product plays that well with every other vendor's.
Click to expand...


Could be a breach, too. Just speculating, of course.

Rich
 
gkainz said:
IT - do your job well and nobody notices. Screw up just once and the whole world knows.
Click to expand...

Absolutely true.
However, I have never seen an IT department that was "unnoticeable". We always knew they were there because our s*** didn't work and they sucked at their job. I guess "suck" must be part of IT job description. :lol:
 
Art VanDelay said:
Thanks a lot IT Goobers ! Wish I could make as many mistakes in my job and still stay employed !!!!
Click to expand...

Maybe you could if you worked with anywhere near as many moving parts.

People who think large airliners are complex have never analyzed even a medium sized piece of code.

I assisted one of our camera control system engineers with a stack corruption yesterday. What a PITA. It's like pulling the #1 throttle back and having the #3 fire suppression system go off and the nose gear drop.
 
Whatever it was, the ground stop has been lifted.

Rich
 
MAKG1 said:
People who think large airliners are complex have never analyzed even a medium sized piece of code.
Click to expand...

To be fair, the software in a modern large airliner is pretty damn impressive, too. The software supporting our avionics impresses me every time I see it.

Compared to a CCTV solution, it's full of even more moving pieces -- quorum rules, safety of flight rules, and so on. Granted, they get their certification and don't get updated, so there's that. Also, their scope is far narrower, so they don't need to do nearly as many things. Additionally, QA is a different animal in that industry.

All that said, the IT bashing here is rather disturbing. Especially since these folks that are bashing couldn't do their jobs without us nowadays. Even with all of the griping, what's productivity look like today compared to 50 years ago as a direct result of our labor?
 
And now the sock market has halted all trading! Not a good day for the cyber/techno geeks...though they're not saying yet why trading has been halted.
 
Art VanDelay said:
Thanks a lot IT Goobers ! Wish I could make as many mistakes in my job and still stay employed !!!!
Click to expand...

When I say 'you have no clue', what I really mean is you are completely and totally without a clue.

I wish I could share 1/10 of the saves my company has made keeping things like the NYSE, NASDAQ, AA, BATF, and a few other places running.

I just came back from a trip to CT, looking into an issue with one of the largest insurers in the world. They were about 3 hours from having to shut down all investment activities due to some screw ups by non-IT goobers.

I could go into a major company right now, and make a few small changes to 3 config files(which will remain nameless) and massive parts of the internet would be down for hours. What's really amazing is that we have so FEW outages, and none that affect health and welfare.

Jeez.
 
docmirror said:
When I say 'you have no clue', what I really mean is you are completely and totally without a clue.

I wish I could share 1/10 of the saves my company has made keeping things like the NYSE, NASDAQ, AA, BATF, and a few other places running.

I just came back from a trip to CT, looking into an issue with one of the largest insurers in the world. They were about 3 hours from having to shut down all investment activities due to some screw ups by non-IT goobers.

I could go into a major company right now, and make a few small changes to 3 config files(which will remain nameless) and massive parts of the internet would be down for hours. What's really amazing is that we have so FEW outages, and none that affect health and welfare.

Jeez.
Click to expand...

As you typed this i noticed that the NYSE is down due to a TECHNICAL GLITCH...
 
Florida Cracker said:
As you typed this i noticed that the NYSE is down due to a TECHNICAL GLITCH...
Click to expand...

Yeah - sorry about that. I do know something about this one, but I can't share much. NYSE is a touchy account. They actually measure the delay on the optics so that there is no advantage in trading times. But, somehow I think you know that, along with the other brakes, and limitations on electronic trading.
 
docmirror said:
Yeah - sorry about that. I do know something about this one, but I can't share much. NYSE is a touchy account. They actually measure the delay on the optics so that there is no advantage in trading times. But, somehow I think you know that, along with the other brakes, and limitations on electronic trading.
Click to expand...

Nothing to be sorry about. I was reading about the shut down while glancing through the Ron Levy thread and thought it was a funny coincidence. Stuff breaks down all the time. It would be a much better world if we fixed problems instead of blame.
 
From watching and occasionally experiencing United/Continental merger madness over the last few years, they made some poor decisions that have lead up to this. Cost seemed to be the primary driver in dealing with redundant IT components such as choosing SHARES over Apollo. Maybe that was the right choice, but as others have pointed out, good, reliable, redundant, IT services, do not come cheap.
 
I don't blame the IT guys. I've been around the industry long enough to recognize the mayhem that ensues when bean counters step over dollars to pick up dimes. This kind of crap starts waaaaay above their pay grade.
 
FormerHangie said:
There can be a couple of reasons. It's very difficult to simulate the real world. Unless you have a dedicated lab set up for load testing, you won't really know until you turn the update loose. An outfit as big as United could have such a setup, but that doesn't mean they do. Sometimes this is farmed out to the vendor, and that vendor will not have complete knowledge of everything your company does or has, and they just don't test for the condition that winds up breaking the system. There's also the ever popular senior management says to get it done by this date with this many people, and that's not always enough resources to test everything.

We do make our mistakes, but we're usually able to test the bigger issues out of the system before deployment.
Click to expand...

Unless things have changed, I think UAL does all software inhouse, altho the original reservation system Apollo has been superceeded by Continental's SHARES.

United created Apollo then sold it to TravelPort and went into a lease agreement. Apollo was located here in Denver (I interviewed there many years ago, neither side was impressed with the other) but it's moved to ATL. If you ever wander around the business areas near KAPA, you'll see a low building with a very serious fence around it. Used to be the data center, altho crew training was (and still is for the time being) near Stapleton. (former KDEN)
 
Last edited:
kayoh190 said:
I don't blame the IT guys. I've been around the industry long enough to recognize the mayhem that ensues when bean counters step over dollars to pick up dimes. This kind of crap starts waaaaay above their pay grade.
Click to expand...

I've never seen a business decision override the technical recommendation that was a success. Or even mildly useful.
 
Gotta be a hack. I saw a tweet by Anon that said that today would be an "Interesting day in Wall Street."

These types of coincidences between multiple systems don't just happen.
 
And all this just conveniently happened when Ron up and left...

Curiouser and curiouser!
 
Best comment I saw on the CNN article: "Someone was just holding the NYSE wrong - Apple."
 
FormerHangie said:
You do realize we can write a program that does your job better than you can.
Click to expand...

No we can't. You just think we can. Until you program in judgement when things don't add up, I don't want to ride in an automated airplane.
 
You must log in or register to reply here.
Back
Top