I have
worked in the past for said Telco albeit 10+ years ago, and another in the not
so distant past...
I
currently work for a major NZ Data Centre provider...
When
you are testing DR power and other systems scenarios as part of your
"Annual Maintenance Plan" in our case "Monthly Critical Systems tests" you have
people to eyeball the equipment to alert to potential issues...
IMHO.
This type of incident enforces my feeling that some providers are only
interested in profits, not customer service. :(
and a
gross lack of investment
I find all this incredible...
Common theme to this in
summary...
Why isn't the UPS monitored ? (assuming it isn't)
Why
didn't they abort their testing, back out, and restore power when things started
going pear shaped ?
It's a no brainer that the UPS battery has a finite
life span which (in this case) is designed to carry the load between the power
outage and when the generator goes online, and 30 Minutes is a very generous
life span. After five minutes, they should have aborted the test and
backed out and then investigated why it failed.
But, hey, the
spokeswoman from Telecom is telling the story...
Oh yeah, the proper
generator test is to cut the mains supply to the essential bus to make sure the
generator starts up and pickup the load.
On Wed, 2007-11-21 at 18:32
+1300, Russell Sharpe wrote:
They weren't
monitoring the UPS? Why didn't they get a trap or alert the UPS didn't
have a incoming feed? mains or Generator?
Most modern systems,
including large (by NZ standards) have snmp traps etc to warn of impending
doom
From: Scott Howard [mailto:scott@doc.net.au]
Sent:
Wednesday, 21 November 2007 12:47
To: Joel
Wiramu Pauling
Cc:
NZNOG
Subject: Re:
[nznog] An explanation of what went wrong at Mayoral Drive on
4/11
On 11/21/07, Joel Wiramu Pauling <aenertia@aenertia.net> wrote:
>However, the switch failed
to connect to the generator and the systems ran down the batteries before
the failure was noticed.
Is it just me... or does this sentence leave one to believe
that mayhaps the switch in question was not actually plugged into the
generator circuit, only the ups.?
Generally there will be a
switch which controls whether the genset is feeding power to the UPS or
not. If you're just testing that the generator is working you'd probably
have that switch turned off, whilst in normal operation you'd have it turned
on.
Some time ago I was involved in a
blackout in a building which we had just moved into where this switch was in
the wrong position, and as a result we saw a similar care to what happened
here (UPS working fine, generator working fine, but one not feeding the other)
- the difference being that we managed to get to the switch in time and enable
it (Kudos to the guy who ran up about 19 flights of stairs to do it!)
Regardless of whether the switch failed,
was in the wrong position, or anything else I can't see how this is anyones
fault other than Telecoms - you don't run a UPS/Generator test and not
actually monitor that the UPS/Generator are functioning correctly. They
had around 30 minutes to detect the problem (based on the quoted life of the
batteries) which should have been plenty to either fix the problem (if it was
just a switch in the wrong state) or to backout the test.
Scott.
_______________________________________________
NZNOG mailing list
NZNOG@list.waikato.ac.nz
http://list.waikato.ac.nz/mailman/listinfo/nznog