The fun kind of power outage

denverpilot

Tied Down
Joined
Nov 8, 2009
Messages
55,483
Location
Denver, CO
Display Name

Display name:
DenverPilot
Lost a phase at the office today and beyond one transformer part of the building was getting 70VAC on regular outlets. Wheeeee.

Interesting to see what ran just fine and didn’t care about 70VAC from the wall. And what said “nope”. Quite mixed but most stuff with switching power supplies managed to just stay running. Probably warmed up their little diodes after their inductors a bit. We shall see what croaks in the coming weeks.

One of the 120VAC legs in the server room went dead, the other was 70VAC but the third was normal. A lot of extension cords and power strips later and things were totally buggered and cobbled to where it all ran again. The UPSes handled the low voltage with style and boosted it. Nice. Guess those were spec’d right. Knew they were but it’s nice to see it actually work. We don’t test the damn things with a Variac after all.

Outage was huge. Took out an entire mall up the road and every other traffic light was apparently on the dead phase.

The really interesting one was half of the gas pumps at a local 7-11 were operable. And I mean half of the pump. Not half of the number of pumps. Ha.

The pump money numbers displays and pump mechanism were alive, the payment screens were down. And nobody thought to mix up the phases for different islands or pumps. Hahaha. Because “you never have a single phase outage”.

Which... is why our little server room has all three legs available and critical things fed off of two of them. One UPS unhappy, the other no problem. Dual power supplies in the devices. Neither unhappy. Until the one UPS dies anyway. Then just a single power supply alarm. Noisy but nothing down.

The voltage drop fried one switch. Poor thing. Spare swapped in by 9PM and by then all power was back. Undid all the dodgy crap to make the phone system happy and went home.

It’d be really nice if they’d buy a damn generator. Oh well.

The one thing that didn’t work was the AC for the building or the server room. Needed three phase. No bueno. But can’t really fix that one. Didn’t get hot enough to bust out the fans or a move and cool. But we’ve got em.

One phase of the 480 dead. Quite entertaining. Just don’t see that one that often. Nor the upstream feed of the other two phases not being cut down automatically with that sort of failure.
 
Lost a phase at the office today and beyond one transformer part of the building was getting 70VAC on regular outlets. Wheeeee.

Interesting to see what ran just fine and didn’t care about 70VAC from the wall. And what said “nope”. Quite mixed but most stuff with switching power supplies managed to just stay running. Probably warmed up their little diodes after their inductors a bit. We shall see what croaks in the coming weeks.

One of the 120VAC legs in the server room went dead, the other was 70VAC but the third was normal. A lot of extension cords and power strips later and things were totally buggered and cobbled to where it all ran again. The UPSes handled the low voltage with style and boosted it. Nice. Guess those were spec’d right. Knew they were but it’s nice to see it actually work. We don’t test the damn things with a Variac after all.

Outage was huge. Took out an entire mall up the road and every other traffic light was apparently on the dead phase.

The really interesting one was half of the gas pumps at a local 7-11 were operable. And I mean half of the pump. Not half of the number of pumps. Ha.

The pump money numbers displays and pump mechanism were alive, the payment screens were down. And nobody thought to mix up the phases for different islands or pumps. Hahaha. Because “you never have a single phase outage”.

Which... is why our little server room has all three legs available and critical things fed off of two of them. One UPS unhappy, the other no problem. Dual power supplies in the devices. Neither unhappy. Until the one UPS dies anyway. Then just a single power supply alarm. Noisy but nothing down.

The voltage drop fried one switch. Poor thing. Spare swapped in by 9PM and by then all power was back. Undid all the dodgy crap to make the phone system happy and went home.

It’d be really nice if they’d buy a damn generator. Oh well.

The one thing that didn’t work was the AC for the building or the server room. Needed three phase. No bueno. But can’t really fix that one. Didn’t get hot enough to bust out the fans or a move and cool. But we’ve got em.

One phase of the 480 dead. Quite entertaining. Just don’t see that one that often. Nor the upstream feed of the other two phases not being cut down automatically with that sort of failure.
We put in a natural gas whole building generator. Bought a used one (from Colorado, interestingly enough). 2008 with ~500 hours on it. I'm not sure it ever got used other than weekly self tests. It went into place literally the day before Hurricane Irma killed the power for a week. (Local gas company has a captive audience and they act like it! "Send us a check for $45K and we'll eventually get your gas line in. No, we won't even give you an estimate of when." Sent the the check and BAM 15 months later (and after one attempt to test when we found out the only pressure in the line was air and they'd not hooked up the other end even though they said it was ready to test).

At any rate, it paid for itself (plus installation) the first week when the rest of the research park was down and we ran. But our transfer gear is smart and will also handle single phase outages which we've had several of. So we have ~30 seconds of UPS's beeping, then power. It's a great feeling...
 
Interesting to see what ran just fine and didn’t care about 70VAC from the wall. And what said “nope”. Quite mixed but most stuff with switching power supplies managed to just stay running. Probably warmed up their little diodes after their inductors a bit. We shall see what croaks in the coming weeks.

Some modern equipment is quite forgiving. We have new TV transmitters that can lose a phase and still make 70% power.
 
Lost a phase at the office today and beyond one transformer part of the building was getting 70VAC on regular outlets. Wheeeee.

Glad you're able to work, by the way.

We've had that happen as well. What's really fun is when you lose a phase and one of the other phases "ghosts" up to some high voltage. Our fluorescent lights absolutely hated that one. We had to replace quite a few ballasts. They were really, really bright for a while, there. :)

All of our big equipment has phase-loss detection. Same as your AC. We're rather be hot, or in the case of a transmitter, be off the air, than lose hundreds of thousands of dollars' worth of equipment ...

I'd be interested to know what brand of UPS you have there. We have a mix of Tripp Lite and APC, and they have been less than stellar. (Being charitable.)
 
I'd be interested to know what brand of UPS you have there. We have a mix of Tripp Lite and APC, and they have been less than stellar. (Being charitable.)

The rack mount APCs and one older HP mostly. The batteries die every three years but they seem to work “okay”. Have seen better permanent setups in real datacenters for sure but we aren’t that. Just a closet with switches, a few Hyper-V servers for on site stuff, and telecom gear. The rest is at AWS.
 
APC and TripLite here. No issues. Be we have a standby generator so the batteries don’t need to work too long.

We did have a fun issue when we switched to voip phones. The PoE switch running the phones didn’t get a new battery backup. First few outages brought down the switch and all the devices plugged into it.
 
Back
Top