- NZNOG - lists.nznog.org

Re: [nznog] Outage update from Telecom
by Ewen McNeill 20 Jun '05

20 Jun '05

In message <Pine.LNX.4.56.0506201840470.8951(a)blue.darkmere.gen.nz>, Lin Nah writes: >http://y3m.net/files/audio/telecom.fault.interview-2005-06-20.mp3 >http://y3m.net/files/audio/telecom.fault.weldon-2005-06-20.mp3 Thanks for recording these and making them available. The spin of both appears to be that this was an unforseeable event, which is difficult to accept. Unlikely, perhaps, but not unforseeable (imagine one break, then look for single points of failure that are exposed by that break). The stock exchange in particular appears not to have done a particularly thorough single-points-of-failure analysis, or deliberately chosen not to attempt to eliminate some of them. For instance it appears that they have a single transit provider, a single firewall, etc -- the failure of any of which will take them "off the air". The same appears to be true of many other systems, some of which can probably not afford to eliminate all the single points of failure (eg, EFTPOS in a small business, even small business DSL), and some of which probably should (eg, aircraft control). It's also a reminder that as much as a provider may claim n-9s reliability for their network, at some level using a single provider for all traffic still represents a single point of failure for your own infrastructure, however unlikely it might seem. Another completely independent provider at least gives you a statistically less correlated set of risk; although ensuring it's completely independent can be non-trivial. Still, given a 99.999% guarentee, and 4.5 hours of downtime (10:48 through 16:18), that means it'll be 4.5 hours / (1/100000) / 24 / 365.25 = 51.3 years until the next outage.... >Let's not forget that TelstraClear and many other ISPs in NZ resell >Telecom (either wholesale or UBS). TCL has patches of their own network >(eg Chch and wgtn are two examples). However the last mile home for >majority of NZers is still Telecom. The "last mile" isn't so much of a problem providing that the single-provider portion really is issolated to the "last mile". If the handoff of each of those "last mile" connections is fairly local to the users, and there's redundant, independent, backhaul from there to the other infrastructure, then a "core network" failure in one provider need not necessarily take out all those users. If there's only a single handoff between the provider and the reseller then obviously it's much more vulnerable as now much larger portions of the providers network must remain operational for those end users to stay working. Hopefully a single-point-of-failure analysis will guide a design which allows as much of the network to remain functional as possible even in the face of several failures. Andy's points of multiple network exchanges, in multiple locations, is of course one good mitigating strategy. Finally it's a reminder that sometimes having the "redundant" part of your network down requires high priority attention; your systems might still be working, but if you're now only one failure away from a total failure it's not a situation you want to be in for long. (From what I heard the first break in Telecom's situation occured yesterday, and was still being worked on today. Not fatal in itself, but without that failure still being present the hungry backhoe wouldn't have caused as much chaos.) Ewen

2 1

[Fwd: Re: [nznog] Outage update from Telecom]
by Mark Foster 19 Jun '05

19 Jun '05

[meant to send to all of list, sorry... redirected:] > So is this a good time to ask about the design of a network that concentrates > so much of its core service in Auckland? Is TelstraClear's network design any > better? > > How many regional providers were unable to talk to others in the same region > because everything (?) is hauled back to Auckland? > Before you follow this train of thought too far - reverse it. For arguements sake: I'm in Auckland, able to talk to the rest of the world - but nowhere south of Palmy. So what about things hauled back to Wellington? ;-) So the bigger picture question should be 'how many providers have sufficient redundancy to provide services in both locations independent of eachother for any given period?' Obviously theres some outages you just can't protect yourself from - but backup plans and DR plans should incorporate other means of getting info from A to B. Corporates etc with multiple offices can't just be dependant on their PON or DSL. They need to be prepared to make a point-to-point dialup, or an ISP dialup at both ends, in order to be able to communicate critical data at least. (How many companies build their SME LAN's behind a DSL link but business grinds to a halt when they can't get their email because the local exchange had a DSL whoopsie? How hard is it to keep a pay-per-hour dialup plan with an alternate ISP?) Ok, upscale slightly. Corporates and PON with big VPN/WAN environments get more complex. But there are ways. GPRS, Mobile Jetstream? Dialup again, with a VPN client? How many businesses retain 'manual' procedures? Fax machines? Fax-over-Cellular? With ISPs its obviously a different can of worms, but the easiest thing to do is to have what essentially translates into two datacentres - or options on two physically diverse routes to your main datacentre from the other. I don't know if theres enough diversity of choice in the market to let the end usr dictate much of this? The other consideration is that for some businesses - and some scales of bandwidth requirement - its not economical, nor feasible, to have a fully disaster-tolerant network. But its interesting to see how we function when the manure hits the turbine, and to see how many people have actually thought it through that far. Mark.

1 0

RE: [nznog] Outage update from Telecom
by Gavin Legge 19 Jun '05

19 Jun '05

likewise. wgtn-acld frame ckt up and running. -----Original Message----- From: Craig Humphrey [mailto:Craig.Humphrey.Work(a)paradise.net.nz] Sent: Monday, 20 June 2005 3:50 p.m. To: 'NZNOG List' Subject: RE: [nznog] Outage update from Telecom We're back too. _____ From: Drew.Collins(a)apn.co.nz [mailto:Drew.Collins(a)apn.co.nz] Sent: Monday, June 20, 2005 3:24 PM To: Juha Saarinen Cc: NZNOG List Subject: Re: [nznog] Outage update from Telecom We have some of our links coming back up at the moment. So hopefully TNZ has fixed some of the problems. -- This message has been scanned for viruses and dangerous content by <http://www.mailscanner.info/> MailScanner, and is believed to be clean. MailScanner thanks transtec <http://www.transtec.co.uk/> Computers for their support. Supplied by ITS-HelpDesk. ext 4001 helpdesk(a)irl.cri.nz

5 4

RE: [nznog] Outage update from Telecom
by Gavin Legge 19 Jun '05

19 Jun '05

Well - ok. Thought it may be helpful.. I'll drop it. - back to lurking. -----Original Message----- From: James Tyson [mailto:james(a)giantrobot.co.nz] Sent: Monday, 20 June 2005 4:31 p.m. To: nznog Subject: Re: [nznog] Outage update from Telecom Do we really need all the "me too"'s? --- James Tyson He's just this guy you know. http://helicopter.geek.nz/ _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. MailScanner thanks transtec Computers for their support. MailScanner Supplied by ITS-HelpDesk. ext 4001 or email helpdesk(a)irl.cri.nz--

1 0

RE: [nznog] Outage update from Telecom
by Gavin Legge 19 Jun '05

19 Jun '05

Our wgtn->chch link was fine as well.. I guess it was below the 'fibre cut's' ??? -----Original Message----- From: Dan Clark [mailto:dan(a)synaptic.net.nz] Sent: Monday, 20 June 2005 4:21 p.m. To: Craig Humphrey Cc: NZNOG(a)list.waikato.ac.nz Subject: Re: [nznog] Outage update from Telecom maybe you went out over microwave? what sort of latency did you experience? Craig Humphrey wrote: > POP, PON, NGN, TLA of the week... we're on it, and we're back. > > BTW we were told that the barer lines to the South Island were down, > but it turns out our Welly office never lost connectivity with our > Christchurch office... (though did experience increased packet loss) > > ------------------------------------------------------------------------ > *From:* James William Butler [mailto:muh47_6(a)hotmail.com] > *Sent:* Monday, June 20, 2005 4:07 PM > *To:* NZNOG(a)list.waikato.ac.nz > *Subject:* RE: [nznog] Outage update from Telecom > > Anyone on Telecom PON have their links up yet? > > > > > > James Butler > > NewJobz > > > > > ---------------------------------------------------------------------- > -- > > *From:* Craig Humphrey [mailto:Craig.Humphrey.Work(a)paradise.net.nz] > *Sent:* Monday, June 20, 2005 3:50 PM > *To:* 'NZNOG List' > *Subject:* RE: [nznog] Outage update from Telecom > > > > We're back too. > > > >----------------------------------------------------------------------- >- > >_______________________________________________ >NZNOG mailing list >NZNOG(a)list.waikato.ac.nz >http://list.waikato.ac.nz/mailman/listinfo/nznog > > _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. MailScanner thanks transtec Computers for their support. MailScanner Supplied by ITS-HelpDesk. ext 4001 or email helpdesk(a)irl.cri.nz--

2 1

I think a rubber band just fell off.
by jamie.baddeley＠vpc.co.nz 19 Jun '05

19 Jun '05

any one else seeing a *large* number of Telecom NZ outages? FR/PON/DSL/Dial etc etc jamie

19 25

RE: [nznog] I think a rubber band just fell off.
by Philip D'Ath 19 Jun '05

19 Jun '05

In fact this was a clever move by Telecom to prevent there share price sliding with everyone wanting to sell up at the same time. :-) -----Original Message----- > "The fault, which happened around 10am, also crashed the New Zealand > Stock Exchange..." This is hyperbole, surely? I'd be fairly shocked if the NZSE internal systems themselves could be "crashed" by a fault in an external vendor. Access to the system from external points may be impacted, of course, but even then I'd have to do the Dance of Stupid[1] if the NZSE public and secure networks only used a single network provider for everything.

1 0

Fwd: UPDATE – OUTAGE ON TELECOM NETWORK
by Ian McDonald 19 Jun '05

19 Jun '05

---------- Forwarded message ---------- From: media(a)telecom-media.co.nz <media(a)telecom-media.co.nz> Date: 20-Jun-2005 13:19 Subject: UPDATE – OUTAGE ON TELECOM NETWORK To: imcdnzl(a)gmail.com UPDATE – OUTAGE ON TELECOM NETWORK Telecom technicians have identified the two locations where the North Island Telecom network has today been damaged and are working to repair them. It is anticipated that a patch will be in place by late this afternoon, enabling the restoration of services. A large number of mobile, Internet and other data services have been affected. In addition, many voice calls on fixed lines are congested. The outage has been affecting customers since late this morning. Telecom is working to repair the outage as soon as possible. Telecom regrets the inconvenience to customers and will provide more information throughout the day. The outage, which occurred at 10.48am today, was caused by physical damage to the network on both the east and west coast sections of the network. Technicians are currently working on repairing both the eastern fibre break which is in the Rimutaka area, and the western break in south Taranaki. The outage is affecting calls to Telecom's own call centres, causing overloading. Telecom is asking customers to be patient. We will provide more information as soon as possible. For more information, please contact: Sarah Berry Media Relations Executive 04 498 9481 027 470 7900 Phil Love Senior Media Relations Executive 04 498 9155 027 277 8496 Melanie Marshall 04 498 9272 027 452 6231 John Goulter Public Affairs & Government Relations Manager 04 498 9369 This message was sent to you because your email address is on the Telecom media releases mailing list. To be removed from this mailing list please visit http://www.telecom-media.co.nz/lists/?id=869&e=im IMPORTANT: Security is limited with current Internet email technology. If you have doubts about the contents or source of this message, please verify that this release comes from Telecom New Zealand by checking the Telecom website at: http://www.telecom-media.co.nz/releases.asp

1 0

Telecom has it as well
by Russell Sharpe 19 Jun '05

19 Jun '05

http://www.telecom-media.co.nz/releases_detail.asp?id=3190 <http://www.telecom-media.co.nz/releases_detail.asp?id=3190&page=index> &page=index

1 0

Fwd: SIGNIFICANT OUTAGE ON TELECOM NETWORK
by Ian McDonald 19 Jun '05

19 Jun '05

Official statement from Telecom.... ---------- Forwarded message ---------- From: media(a)telecom-media.co.nz <media(a)telecom-media.co.nz> Date: 20-Jun-2005 12:35 Subject: SIGNIFICANT OUTAGE ON TELECOM NETWORK To: imcdnzl(a)gmail.com SIGNIFICANT OUTAGE ON TELECOM NETWORK An outage on the Telecom network has been affecting a large number of customers since late this morning. Telecom is working to repair the outage as soon as possible. Telecom regrets the inconvenience to customers and will provide more information as soon as possible. The outage, which occurred at 10.48am today, has been caused by two separate incidents affecting different parts of the network. Internet, mobile and landline calls are affected. Technicians are currently working on repairing one of the fibre breaks which is in the Rimutaka area. Once it has been repaired the majority of services will be restored. Technicians are investigating the other fault and will repair it as soon as possible. The outage is affecting calls to Telecom's own call centres, causing overloading. Telecom is asking customers to be patient. We will provide more information as soon as possible. For more information, please contact: Sarah Berry Media Relations Executive 04 498 9481 027 470 7900 Phil Love Senior Media Relations Executive 04 498 9155 027 277 8496 Melanie Marshall 04 498 9272 027 452 6231 John Goulter Public Affairs & Government Relations Manager 04 498 9369 027 232 4303 This message was sent to you because your email address is on the Telecom media releases mailing list. To be removed from this mailing list please visit http://www.telecom-media.co.nz/lists/?id=869&e=im IMPORTANT: Security is limited with current Internet email technology. If you have doubts about the contents or source of this message, please verify that this release comes from Telecom New Zealand by checking the Telecom website at: http://www.telecom-media.co.nz/releases.asp

1 0