Widespread Auckland Power Outage 2009-02-03
Hi Folks, Starting around 13:06 this afternoon, a number of parts of Auckland lost power. At a glance, I saw some of our monitored subscribers with unprotected (or underprotected) devices go down in: Ellerslie Mt. Richmond Newmarket Otahuhu Parnell Penrose Westfield Regards, Jon
No outage in Grafton or the CBD. Cheers, Patrick On 3/02/2009, at 1:16 PM, Jonathan Brewer wrote:
Hi Folks,
Starting around 13:06 this afternoon, a number of parts of Auckland lost power.
At a glance, I saw some of our monitored subscribers with unprotected (or underprotected) devices go down in:
Ellerslie Mt. Richmond Newmarket Otahuhu Parnell Penrose Westfield
Regards,
Jon _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
Its in the herald, it must be true! http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10554954 From: Patrick Jordan-Smith [mailto:patpatnz(a)gmail.com] Sent: Tuesday, 3 February 2009 1:20 p.m. To: nznog Subject: Re: [nznog] Widespread Auckland Power Outage 2009-02-03 No outage in Grafton or the CBD. Cheers, Patrick On 3/02/2009, at 1:16 PM, Jonathan Brewer wrote: Hi Folks, Starting around 13:06 this afternoon, a number of parts of Auckland lost power. At a glance, I saw some of our monitored subscribers with unprotected (or underprotected) devices go down in: Ellerslie Mt. Richmond Newmarket Otahuhu Parnell Penrose Westfield Regards, Jon _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nzmailto:NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On 3/02/2009, at 1:46 PM, Simon Allard wrote:
Its in the herald, it must be true!
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10554954
Since they checked with Vector... yes, it probably is true. Major faults and outages – Current Transpower outage affecting eastern suburbs Date: 03-02-2009 01:31 pm Transpower's transformers at its Penrose substation have faulted causing an outage affecting approximately 74,000 customers in Auckland’s eastern suburbs. Must've been Megatron stealing all of Transpower's Spark... -- Juha Saarinen juha(a)saarinen.org
My monitored alarm with Safesecure (I believe they are in that area) isn't
working either, do I report that fault here? :P
Nah number just rings, makes me wonder how many other alarm companies don't
have backup power :)
Everyone affected by power and dont have UPS, go have a beer...
b.
On Tue, 3 Feb 2009 13:48:51 +1300, Juha Saarinen
On 3/02/2009, at 1:46 PM, Simon Allard wrote:
Its in the herald, it must be true!
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10554954
Since they checked with Vector... yes, it probably is true.
Major faults and outages – Current Transpower outage affecting eastern suburbs Date: 03-02-2009 01:31 pm
Transpower's transformers at its Penrose substation have faulted causing an outage affecting approximately 74,000 customers in Auckland’s eastern suburbs.
Must've been Megatron stealing all of Transpower's Spark...
Could just be they have hundreds of alarms and people ringing up and can't answer all the calls as opposed to no backup power ... then again, maybe they recognise you from caller ID. :-)
-----Original Message-----
From: Barry Murphy [mailto:barry(a)unix.co.nz]
Sent: Tuesday, 3 February 2009 1:53 p.m.
To: Juha Saarinen
Cc: nznog
Subject: Re: [nznog] Widespread Auckland Power Outage 2009-02-03
My monitored alarm with Safesecure (I believe they are in that area) isn't
working either, do I report that fault here? :P
Nah number just rings, makes me wonder how many other alarm companies don't
have backup power :)
Everyone affected by power and dont have UPS, go have a beer...
b.
On Tue, 3 Feb 2009 13:48:51 +1300, Juha Saarinen
On 3/02/2009, at 1:46 PM, Simon Allard wrote:
Its in the herald, it must be true!
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10554954
Since they checked with Vector... yes, it probably is true.
Major faults and outages – Current Transpower outage affecting eastern suburbs Date: 03-02-2009 01:31 pm
Transpower's transformers at its Penrose substation have faulted causing an outage affecting approximately 74,000 customers in Auckland’s eastern suburbs.
Must've been Megatron stealing all of Transpower's Spark...
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On Tue, Feb 3, 2009 at 1:54 PM, Dean Pemberton
Awesome - now we get to compare the reactions.
Wellington has an outage, it makes the news but we get on with it. Auckland has an outage........
74000 customers in the middle of the commercial and industrial belt of Auckland probably means 200,000+ people affected. Add traffic lights plus lunch time and it is a nice mess. You would think that Transpower would learn not do work to critical systems in the middle of a hot day when it would be a peak.
You would think that Transpower would learn not do work to critical systems in the middle of a hot day when it would be a peak.
They would have been trying to avoid doing it on a Friday after lunch. ;) Erin Salmon Managing Director - Unleash Phone: 03 365 1273 Mobile: 0275 877 913
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nicholas Lee wrote:
74000 customers in the middle of the commercial and industrial belt of Auckland probably means 200,000+ people affected. Add traffic lights plus lunch time and it is a nice mess.
You would think that Transpower would learn not do work to critical systems in the middle of a hot day when it would be a peak.
They have to do the work at some time, and it sounds like they had n+1 right up until the point that the n tripped out the +1. It happens, and getting all upppity about when they should schedule their maintenance won't change anything. It's not always reasonable to work in the middle of the night. - -- Matthew Poole "Don't use force. Get a bigger hammer." -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmHsAIACgkQTdEtTmUCdpyV/gCgoeo7VgiIgaPyfOf6htlQ7Ik8 YQ8AoN2qwbVxXImq8a2ATEl5Ha5BgKue =Lw2x -----END PGP SIGNATURE-----
On 3/02/2009, at 3:46 PM, Matthew Poole wrote:
Nicholas Lee wrote:
74000 customers in the middle of the commercial and industrial belt of Auckland probably means 200,000+ people affected. Add traffic lights plus lunch time and it is a nice mess.
You would think that Transpower would learn not do work to critical systems in the middle of a hot day when it would be a peak.
They have to do the work at some time, and it sounds like they had n+1 right up until the point that the n tripped out the +1.
If the n can trip out the +1 then you could argue they never really had n+1. -jasper
Matthew Poole wrote:
They have to do the work at some time, and it sounds like they had n+1 right up until the point that the n tripped out the +1. Stuff.co.nz wrote:
Transpower spokeswoman Adele Fitzpatrick said full restoration was expected by 5pm. She said one of the three transformers was out of action due to routine maintenance. The other two were handling the load until a malfunction in one of them cause the third to trip.
In other words, they *had* n+1, then took it down to n+0 while they worked on their +1, and then they had a fault.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Ian Batterbee wrote:
In other words, they *had* n+1, then took it down to n+0 while they worked on their +1, and then they had a fault.
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10555060 "He said three transformers usually took the load at Penrose but yesterday one of them was out for regular maintenance. Normally, two transformers could cope but this time one of them failed and this caused the remaining one to trip. Mr Strange said Penrose was now relying on one transformer while staff worked round the clock to try to have a second one in use some time today." Nope, n+1, then the n took down the +1. Otherwise it wouldn't be able to operate on a single transformer. So we'd better hope that, until the +1 is fixed, n behaves itself. My guess would be that it's roughly n+2 at this time of year, but only n+1 during winter, or maybe even once the universities all get back up to full speed in three more weeks. - -- Matthew Poole "Don't use force. Get a bigger hammer." -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmIsE4ACgkQTdEtTmUCdpwyDQCgh8SS/db22n2a07Cf0jsfcnKr zGAAoJTUjQOLx7M5UHQCXX7GD0EH0lJd =idHy -----END PGP SIGNATURE-----
Sorta. They had n+1 They took down the +1 for regular maintenance leaving them n. Then one failed leaving them n-1. Then the remaining unit could not carry the load and shutdown to protect itself. I'm not about to preach to Transpower, anyone who saw Richard Naylors talk at NZNOG will appreciate the huge job they have. But the lessons for us network people must be: If the chance of one link/router failing is x. Then if you have 3 parallel/resilient links/routers then the risk of a total outage is x/3. If you take one of those links/routers out for maintenance then the risk of a total outage is x/2. But here is the kicker. If the single remaining link/router can't take the total load without causing such significant loss or interruption that it had may as well be down... then the risk of a total outage with 2 links is really x. Have a look around your network. The same thing applies to UPSs and all sorts of things. Feel free to flame if I have my maths/stats/facts wrong. Dean Matthew Poole wrote:
Nope, n+1, then the n took down the +1. Otherwise it wouldn't be able to operate on a single transformer. So we'd better hope that, until the +1 is fixed, n behaves itself. My guess would be that it's roughly n+2 at this time of year, but only n+1 during winter, or maybe even once the universities all get back up to full speed in three more weeks.
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
At 11:13 a.m. 4/02/2009, Dean Pemberton wrote:
They had n+1 They took down the +1 for regular maintenance leaving them n. Then one failed leaving them n-1. Then the remaining unit could not carry the load and shutdown to protect itself.
Actually they may have had (n+1)x2 or some factor like 2. On big transformers is common to have oil pumps and fans on the radiators. So you can turn a 15MW tranny into a 30MW tranny (or whatever the manufacturer has calculated). Of course if you were already running the pumps and fans, then it doesn't help.....and you can't run them for too long, apart from the noise, things do like to cool. Same goes for cables. Older cables (paper-lead) could be run in over load for several hours (like 50%+ overload). "Plastic" cables aren't as tolerant - they melt. You also need to keep an eye on road temperatures, some cables have to be de-rated if the roads are running too hot. Overhead lines are so much easier, they just sag under heavy load........
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Richard Naylor wrote:
Actually they may have had (n+1)x2 or some factor like 2.
On big transformers is common to have oil pumps and fans on the radiators. So you can turn a 15MW tranny into a 30MW tranny (or whatever the manufacturer has calculated). Of course if you were already running the pumps and fans, then it doesn't help.....and you can't run them for too long, apart from the noise, things do like to cool.
Ah. So yesterday they were running at more like n+0.5, then n died and the 0.5 had to be fiddled with to get it up to n? Clearly the single transformer that was still operational when the Herald went to press this morning hasn't failed again, because I'm sitting in the affected area and my (non-UPS'd) home server is reporting an up-time that extends well into yesterday. So a margin less than 1 but greater than 0, and able to be extended to 1 in emergencies, makes sense. - -- Matthew Poole "Don't use force. Get a bigger hammer." -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmIybUACgkQTdEtTmUCdpynAQCbBbOgPim8/nbdZyEuqpR8CZ4C tFkAn0vbMBN9JEUZbbczZwvxy66LOnIE =F9bf -----END PGP SIGNATURE-----
So maybe two issues
- Equipment is installed at N+1 with 50% load for N, but now load is 80%
N. Pretending to be RAID1 with a hot spare, but which is actually RAID0.5
with a hot spare.
- When you assume you have an option to do a boost on specs - turn 15MW
into 30MW - but this is dependent on heat levels. Plan to do work when it is
cooler.
Power infrastructure is expensive and these guys try to get them moneys
worth. The transformer in my street was giving me constant low voltage,
Vector's answer was to switch me to a new phase. Now I only seem to get low
voltage at 6pm in winter when people get home. It might be ok to do this
stuff for one street - but if they doing this for core infrastructure this
issue is going to keep happening.
On Wed, Feb 4, 2009 at 11:32 AM, Richard Naylor
At 11:13 a.m. 4/02/2009, Dean Pemberton wrote:
They had n+1 They took down the +1 for regular maintenance leaving them n. Then one failed leaving them n-1. Then the remaining unit could not carry the load and shutdown to protect itself.
Actually they may have had (n+1)x2 or some factor like 2.
On big transformers is common to have oil pumps and fans on the radiators. So you can turn a 15MW tranny into a 30MW tranny (or whatever the manufacturer has calculated). Of course if you were already running the pumps and fans, then it doesn't help.....and you can't run them for too long, apart from the noise, things do like to cool.
Same goes for cables. Older cables (paper-lead) could be run in over load for several hours (like 50%+ overload). "Plastic" cables aren't as tolerant - they melt.
You also need to keep an eye on road temperatures, some cables have to be de-rated if the roads are running too hot. Overhead lines are so much easier, they just sag under heavy load........
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On 03/02/2009, at 1:13 PM, Nicholas Lee wrote:
You would think that Transpower would learn not do work to critical systems in the middle of a hot day when it would be a peak.
Isn't the weather something like 25-26C in Auckland today? (Feel free to swap - it's awesomely hot here in Adelaide this week - peaked at 45.7C). MMC -- Matthew Moyle-Croft Internode/Agile Peering and Core Networks Level 5, 162 Grenfell Street, Adelaide, SA 5000 Australia Email: mmc(a)internode.com.au Web: http://www.on.net Direct: +61-8-8228-2909 Mobile: +61-419-900-366 Reception: +61-8-8228-2999 Fax: +61-8-8235-6909
On 3/02/2009, at 3:43 PM, Nicholas Lee wrote:
You would think that Transpower would learn not do work to critical systems in the middle of a hot day when it would be a peak.
Personally I'd prefer to have the time when there is risk of power cuts during the day - at least there is a big light in the sky (most days.) Much easier than trying to find the candles when the power cuts out at 10pm and it all goes dark. Much easier to deal with traffic issues. Cheers, Patrick
On Tue, 3 Feb 2009, Patrick Jordan-Smith wrote:
On 3/02/2009, at 3:43 PM, Nicholas Lee wrote:
You would think that Transpower would learn not do work to critical systems in the middle of a hot day when it would be a peak.
Personally I'd prefer to have the time when there is risk of power cuts during the day - at least there is a big light in the sky (most days.) Much easier than trying to find the candles when the power cuts out at 10pm and it all goes dark. Much easier to deal with traffic issues.
Yes, and people are at work to deal with the ramifications, instead of being on callback/overtime and delayed to respond... After all, UPS is just brownout protection, right? :-)
After all, UPS is just brownout protection, right?
Having just moved to a new building with a backup generator that kicked in less than 20 seconds after the initial power outage i'm pleased to say thats exactly how my UPS's functioned this time. The change back to grid power was even quicker - so much so that one of my UPS's didn't even register the fact that anything had changed :D Aside from keeping the servers running, it indiscriminately powered every outlet in the entire building so the fridge stayed on and therefore the beer stayed cold!
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dean Pemberton wrote:
Awesome - now we get to compare the reactions.
Wellington has an outage, it makes the news but we get on with it. Auckland has an outage........
The outage up here didn't hit the CBD. Not sure how far south it extended, but having come from the city via Newmarket while power was out I can confirm that it "began" from Mountain Road. Looks like roughly a line from Mt Eden through to Parnell, avoiding the CBD entirely. - -- Matthew Poole "Don't use force. Get a bigger hammer." -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkmHr3MACgkQTdEtTmUCdpyw1wCfRFMN7bP1qrTyq4Mt65JvUiIC akAAnA0ZwZzzdkQ2wPecNP+h8y5DXaL4 =MaGi -----END PGP SIGNATURE-----
On Tue, 2009-02-03 at 13:54 +1300, Dean Pemberton wrote:
Awesome - now we get to compare the reactions.
Auckland: 17 posts. (and counting) Wellington: 4 posts. Hmm, what's the signal to noise ratio with either? jamie
Wellington has an outage, it makes the news but we get on with it. Auckland has an outage........
=P
Dean
Juha Saarinen wrote:
On 3/02/2009, at 1:46 PM, Simon Allard wrote:
Its in the herald, it must be true!
http://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10554954
Since they checked with Vector... yes, it probably is true.
Major faults and outages – Current Transpower outage affecting eastern suburbs Date: 03-02-2009 01:31 pm
Transpower's transformers at its Penrose substation have faulted causing an outage affecting approximately 74,000 customers in Auckland’s eastern suburbs.
Must've been Megatron stealing all of Transpower's Spark...
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On 3/02/2009, at 22:59 , jamie baddeley wrote:
On Tue, 2009-02-03 at 13:54 +1300, Dean Pemberton wrote:
Awesome - now we get to compare the reactions.
Auckland: 17 posts. (and counting) Wellington: 4 posts.
Hmm, what's the signal to noise ratio with either?
Population of Auckland region 1.3 million, Wellington region 0.4 million. Guess that makes us greater whingers down here. (:-) Can this discussion be moved to the NZPOG?
participants (18)
-
Andy Linton -
Barry Murphy -
Dean Pemberton -
Erin Salmon -
Ian Batterbee -
jamie baddeley -
Jasper Bryant-Greene -
Jonathan Brewer -
Juha Saarinen -
Mark Foster -
Matthew Moyle-Croft -
Matthew Poole -
Nicholas Lee -
Patrick Jordan-Smith -
Philip D'Ath -
Regan Murphy -
Richard Naylor -
Simon Allard