I find all this incredible...
Common theme to this in summary...

Why isn't the UPS monitored ? (assuming it isn't)
Why didn't they abort their testing, back out, and restore power when things started going pear shaped ?

It's a no brainer that the UPS battery has a finite life span which (in this case) is designed to carry the load between the power outage and when the generator goes online, and 30 Minutes is a very generous life span. After five minutes, they should have aborted the test and backed out and then investigated why it failed.

But, hey, the spokeswoman from Telecom is telling the story...

Oh yeah, the proper generator test is to cut the mains supply to the essential bus to make sure the generator starts up and pickup the load.

On Wed, 2007-11-21 at 18:32 +1300, Russell Sharpe wrote:

They weren't monitoring the UPS? Why didn't they get a trap or alert the UPS didn't have a incoming feed? mains or Generator?

Most modern systems, including large (by NZ standards) have snmp traps etc to warn of impending doom

From: Scott Howard [mailto:scott@doc.net.au]
Sent: Wednesday, 21 November 2007 12:47
To: Joel Wiramu Pauling
Cc: NZNOG
Subject: Re: [nznog] An explanation of what went wrong at Mayoral Drive on 4/11

On 11/21/07, Joel Wiramu Pauling <aenertia@aenertia.net> wrote:

>However, the switch failed to connect to the generator and the systems ran down the batteries before the failure was noticed.

Is it just me... or does this sentence leave one to believe that mayhaps the switch in question was not actually plugged into the generator circuit, only the ups.?

Generally there will be a switch which controls whether the genset is feeding power to the UPS or not. If you're just testing that the generator is working you'd probably have that switch turned off, whilst in normal operation you'd have it turned on.

Some time ago I was involved in a blackout in a building which we had just moved into where this switch was in the wrong position, and as a result we saw a similar care to what happened here (UPS working fine, generator working fine, but one not feeding the other) - the difference being that we managed to get to the switch in time and enable it (Kudos to the guy who ran up about 19 flights of stairs to do it!)

Regardless of whether the switch failed, was in the wrong position, or anything else I can't see how this is anyones fault other than Telecoms - you don't run a UPS/Generator test and not actually monitor that the UPS/Generator are functioning correctly. They had around 30 minutes to detect the problem (based on the quoted life of the batteries) which should have been plenty to either fix the problem (if it was just a switch in the wrong state) or to backout the test.

Scott.

_______________________________________________
NZNOG mailing list
NZNOG@list.waikato.ac.nz
http://list.waikato.ac.nz/mailman/listinfo/nznog