Ok, I'll run a pool for cause. Propose possible reasons and I'll come up with odds. All betting to be done in the currency of handles to be settled at the next meeting. Well Its been more than a couple of more hours now ! There had better be a good explanation and I really love the - Network Status - No known fault System Status Normal - No major problems
On 6/10/2007, at 16:40 , Tony Wicks wrote:
On 6/10/2007, at 14:04 , Nathan Ward wrote:
Ok, I'll run a pool for cause. Propose possible reasons and I'll come up with odds. All betting to be done in the currency of handles to be settled at the next meeting.
The first time I read that as NIC handles. A beer fixed that.
Well Its been more than a couple of more hours now ! There had better be a good explanation and I really love the -
Network Status - No known fault System Status Normal - No major problems
Yeah I noticed that too. :) http://tinyurl.com/37qpx4 Seems updated now - tho still not 'closed' http://www.citylink.co.nz/information/status/ http://lists.citylink.co.nz/cgi-bin/mailman/listinfo/network-events Wellington Outage System Status: Major Fault Time: 16:55 Date: 6/9/2007 Location: Wellington Description A number of customers have had their connectivity restored. We are however, still experiencing outages for a number of customers, work is on-going to restore connectivity. Unfortunately we do not have an ETR at this stage.
im putting down they need a clue LS At 04:40 p.m. 6/10/2007, Tony Wicks wrote:
Ok, I'll run a pool for cause. Propose possible reasons and I'll come up with odds. All betting to be done in the currency of handles to be settled at the next meeting.
Well Its been more than a couple of more hours now ! There had better be a good explanation and I really love the -
Network Status - No known fault System Status Normal - No major problems
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On 6/10/2007, at 21:41 , Term wrote:
im putting down they need a clue LS At 04:40 p.m. 6/10/2007, Tony Wicks wrote:
Ok, I'll run a pool for cause. Propose possible reasons and I'll come up with odds. All betting to be done in the currency of handles to be settled at the next meeting.
I'm going for 'terrorism' :) It seems Stuff got round to posting again this morning. Apart from the few blog posts - not a lot out there. Technical problems publishing Stuff.co.nz http://stuff.co.nz/4229049a28.html Stuff.co.nz, New Zealand - 10 hours ago The problem was as a result of an outage at the CityLink network and also affected a number of other websites hosted in Wellington, including the MetService ... Broadband network crash upsets punters http://www.stuff.co.nz/4229049a11.html Stuff.co.nz, New Zealand - 1 hour ago Broadband network operator CityLink's system crashed early yesterday morning causing slow or, in some cases, no service for most of the capital. ... -- Jo Booth Network Facilitator Cell: +64 21 JBOOTH (021 526684) PO Box 40732, Upper Hutt 5140. Mesh|net - http://mesh.net.nz
Morning all On Sat, Oct 06, 2007 at 04:40:48PM +1300, Tony Wicks said:
Well Its been more than a couple of more hours now !
A 6-10 hour outage for most, a couple of buildings are still out as of midnight. Not the best way to spend a sunny Saturday.
There had better be a good explanation
I'll do a more in depth explanation tomorrow. Essentially, this started happening, on every interswitch trunk on most switches on the network: Oct 6 14:09:07: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on FastEthernet0/24. Oct 6 14:09:07: %PM-4-ERR_DISABLE: loopback error detected on Fa0/24, putting Fa0/24 in err-disable state Oct 6 14:09:08: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/24, changed state to down Oct 6 14:09:09: %LINK-3-UPDOWN: Interface FastEthernet0/24, changed state to down Oct 6 14:09:50: %PM-4-ERR_RECOVER: Attempting to recover from loopback err-disable state on Fa0/24 Oct 6 14:09:55: %LINK-3-UPDOWN: Interface FastEthernet0/24, changed state to up Oct 6 14:09:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/24, changed state to up Oct 6 14:09:57: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on FastEthernet0/24. a problem that takes a while to resolve, when switches are only reachable for 5 of every 45 seconds. It wasn't a fibre cut, it wasn't a power outage, it wasn't a hardware failure, it wasn't obviously a spanning-tree wigout, we had no scheduled (or unscheduled) work on at that time.
and I really love the -
Network Status - No known fault System Status Normal - No major problems
Priorities. When the network is so hosed that I can't see the website from the other end of the DMZ, making sure the website is uptodate takes a back seat to actually fixing the network. Once we got Internet connectivity back to our office, we got the page updated pretty smartly - just before 5pm, from memory. Cheers Si
Si It would be good to hear what caused the problem... I'm sure many of us will learn from it... Sounds like a nasty :( Russell Sharpe -----Original Message----- From: Simon Blake [mailto:simon(a)citylink.co.nz] Sent: Sunday, 7 October 2007 00:56 To: Tony Wicks Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Citylink problems? Morning all On Sat, Oct 06, 2007 at 04:40:48PM +1300, Tony Wicks said:
Well Its been more than a couple of more hours now !
A 6-10 hour outage for most, a couple of buildings are still out as of midnight. Not the best way to spend a sunny Saturday.
There had better be a good explanation
I'll do a more in depth explanation tomorrow. Essentially, this started happening, on every interswitch trunk on most switches on the network: Oct 6 14:09:07: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on FastEthernet0/24. Oct 6 14:09:07: %PM-4-ERR_DISABLE: loopback error detected on Fa0/24, putting Fa0/24 in err-disable state Oct 6 14:09:08: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/24, changed state to down Oct 6 14:09:09: %LINK-3-UPDOWN: Interface FastEthernet0/24, changed state to down Oct 6 14:09:50: %PM-4-ERR_RECOVER: Attempting to recover from loopback err-disable state on Fa0/24 Oct 6 14:09:55: %LINK-3-UPDOWN: Interface FastEthernet0/24, changed state to up Oct 6 14:09:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/24, changed state to up Oct 6 14:09:57: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on FastEthernet0/24. a problem that takes a while to resolve, when switches are only reachable for 5 of every 45 seconds. It wasn't a fibre cut, it wasn't a power outage, it wasn't a hardware failure, it wasn't obviously a spanning-tree wigout, we had no scheduled (or unscheduled) work on at that time.
and I really love the -
Network Status - No known fault System Status Normal - No major problems
Priorities. When the network is so hosed that I can't see the website from the other end of the DMZ, making sure the website is uptodate takes a back seat to actually fixing the network. Once we got Internet connectivity back to our office, we got the page updated pretty smartly - just before 5pm, from memory. Cheers Si _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On Sun, Oct 07, 2007 at 12:56:07AM +1300, Simon Blake said:
I'll do a more in depth explanation tomorrow.
Two weeks later, I'm done writing. Finally. The in depth writeup is at: http://news.clnz.net/ Cheers Si
$quoted_author = "Simon Blake" ;
Two weeks later, I'm done writing. Finally. The in depth writeup is at:
**APPLAUSE** one of the best outage reports i've had the pleasure of reading... cheers marty -- "No GUI for you! Use lynx!!!, Come back, One year!" /avant
participants (6)
-
Jo Booth -
Martin Barry -
Russell Sharpe -
Simon Blake -
Term -
Tony Wicks