Morning all On Sat, Oct 06, 2007 at 04:40:48PM +1300, Tony Wicks said:
Well Its been more than a couple of more hours now !
A 6-10 hour outage for most, a couple of buildings are still out as of midnight. Not the best way to spend a sunny Saturday.
There had better be a good explanation
I'll do a more in depth explanation tomorrow. Essentially, this started happening, on every interswitch trunk on most switches on the network: Oct 6 14:09:07: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on FastEthernet0/24. Oct 6 14:09:07: %PM-4-ERR_DISABLE: loopback error detected on Fa0/24, putting Fa0/24 in err-disable state Oct 6 14:09:08: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/24, changed state to down Oct 6 14:09:09: %LINK-3-UPDOWN: Interface FastEthernet0/24, changed state to down Oct 6 14:09:50: %PM-4-ERR_RECOVER: Attempting to recover from loopback err-disable state on Fa0/24 Oct 6 14:09:55: %LINK-3-UPDOWN: Interface FastEthernet0/24, changed state to up Oct 6 14:09:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/24, changed state to up Oct 6 14:09:57: %ETHCNTR-3-LOOP_BACK_DETECTED: Keepalive packet loop-back detected on FastEthernet0/24. a problem that takes a while to resolve, when switches are only reachable for 5 of every 45 seconds. It wasn't a fibre cut, it wasn't a power outage, it wasn't a hardware failure, it wasn't obviously a spanning-tree wigout, we had no scheduled (or unscheduled) work on at that time.
and I really love the -
Network Status - No known fault System Status Normal - No major problems
Priorities. When the network is so hosed that I can't see the website from the other end of the DMZ, making sure the website is uptodate takes a back seat to actually fixing the network. Once we got Internet connectivity back to our office, we got the page updated pretty smartly - just before 5pm, from memory. Cheers Si