Re: [nznog] An explanation of what went wrong at Mayoral Drive on 4/11

I have worked in the past for said Telco albeit 10+ years ago, and another in the not so distant past... I currently work for a major NZ Data Centre provider... When you are testing DR power and other systems scenarios as part of your "Annual Maintenance Plan" in our case "Monthly Critical Systems tests" you have people to eyeball the equipment to alert to potential issues... IMHO. This type of incident enforces my feeling that some providers are only interested in profits, not customer service. :( and a gross lack of investment _____ From: Lindsay Druett [mailto:lindsay(a)wired.net.nz] Sent: Wednesday, 21 November 2007 19:01 To: Russell Sharpe; 'NZNOG' Subject: Re: [nznog] An explanation of what went wrong at Mayoral Drive on 4/11 I find all this incredible... Common theme to this in summary... Why isn't the UPS monitored ? (assuming it isn't) Why didn't they abort their testing, back out, and restore power when things started going pear shaped ? It's a no brainer that the UPS battery has a finite life span which (in this case) is designed to carry the load between the power outage and when the generator goes online, and 30 Minutes is a very generous life span. After five minutes, they should have aborted the test and backed out and then investigated why it failed. But, hey, the spokeswoman from Telecom is telling the story... Oh yeah, the proper generator test is to cut the mains supply to the essential bus to make sure the generator starts up and pickup the load. On Wed, 2007-11-21 at 18:32 +1300, Russell Sharpe wrote: They weren't monitoring the UPS? Why didn't they get a trap or alert the UPS didn't have a incoming feed? mains or Generator? Most modern systems, including large (by NZ standards) have snmp traps etc to warn of impending doom _____ From: Scott Howard [mailto:scott(a)doc.net.au] Sent: Wednesday, 21 November 2007 12:47 To: Joel Wiramu Pauling Cc: NZNOG Subject: Re: [nznog] An explanation of what went wrong at Mayoral Drive on 4/11 On 11/21/07, Joel Wiramu Pauling wrote:

However, the switch failed to connect to the generator and the systems ran down the batteries before the failure was noticed.

Is it just me... or does this sentence leave one to believe that mayhaps the switch in question was not actually plugged into the generator circuit, only the ups.? Generally there will be a switch which controls whether the genset is feeding power to the UPS or not. If you're just testing that the generator is working you'd probably have that switch turned off, whilst in normal operation you'd have it turned on. Some time ago I was involved in a blackout in a building which we had just moved into where this switch was in the wrong position, and as a result we saw a similar care to what happened here (UPS working fine, generator working fine, but one not feeding the other) - the difference being that we managed to get to the switch in time and enable it (Kudos to the guy who ran up about 19 flights of stairs to do it!) Regardless of whether the switch failed, was in the wrong position, or anything else I can't see how this is anyones fault other than Telecoms - you don't run a UPS/Generator test and not actually monitor that the UPS/Generator are functioning correctly. They had around 30 minutes to detect the problem (based on the quoted life of the batteries) which should have been plenty to either fix the problem (if it was just a switch in the wrong state) or to backout the test. Scott. _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog