On Mon, 5 Nov 2007, Edmund A. Hintz wrote:

But, the fire service was hit fairly hard. The reporting system is also used for turnouts, and was down, so the comm centers had to page everyone out manually.

For those not au fait with how things work for the Fire Service, they use a Computer Aided Dispatch system that, amongst other things, automatically sends pages out based on appliances being attached to a call. The loss of i/CAD meant that pages had to be sent using manual paging terminals, from lists of pager numbers. It's not efficient, by any stretch of the imagination. *SNIP*

had to be verbally radioed in. I understand that in Northland the paging systems were out entirely, and they were on siren/verbal turnout only up

Not just Northland, Auckland too. No paging==no station alarms or speakers, no automatic doors, etc. It was ALL being done by RT, which meant that stations had to keep a radio on in order to be alerted to calls. It was fortunate that Murphy, having chosen to strike the Fire Service over Guy Fawkes weekend, didn't interject further with a major incident while things were running on hard-copy. Another major service provided by i/CAD is that fire appliances can use DTMF-over-RF to keep the computer appraised of the appliance's availability or its disposition in relation to a call. Losing that meant that every move by an appliance had to be spoken verbally, and confirmed verbally, greatly increasing the radio time required for absolutely everything.

there. And lastly, I hear that some Ambo comms were out entirely, though don't know any more about it (Messrs Poole or Foster may have more on that).

Only heard about the ambulance issue at second hand. They rely pretty heavily on cellphones anyway, so being deprived of radios isn't necessarily a significant imposition on their normal operations. Unsurprisingly, the media only half reported on how serious this outage was for the Fire Service. Explaining it to reporters in sufficient clarity that they can then put it in print for the great unwashed is no small ask. I've only summarised very lightly how it all works. -- Matthew Poole "Don't use force. Get a bigger hammer."

On 5/11/2007, at 2:05 PM, Neil Gardner wrote:

...

But, the fire service was hit fairly hard. The reporting system is also used for turnouts, and was down, so the comm centers had to page everyone out manually.

[snipped lots of interesting stuff about how various emergency services handled the drop back to manual processes]

Anyone else pretty pleased and impressed with how well the people and organisations involved dealt with a failure in the automated systems? Personally it makes me a bit happier to realise that some genuinely critical services have pretty solid backup processes that by all accounts so far were followed and performed satisfactorily.

I think everyone with a reliance on technology for their business / personal life needs to take a lesson from this.

If your reliance on technology is _THAT_ important, you WILL have suitable backup systems in place. And I don't mean having 2 x ADSL connections either! :-)

Do Telecom manage this service for the Fire Service, or is it managed by a third party, or even the Fire Service itself? It seems like the blame for the Fire Service outage is proposed to be Telecom by some people - but if there's a third party involved I'd put the blame on them for either: a) Not planning for an outage (which might be because of b, or incompetence). b) Deciding that an outage wasn't likely to happen. In short though - yep, their fallback appeared to work pretty well, and it's good that they were prepared for it. A similar thing for ISPs might be - what happens if you lose power to your call centre - do you have an answer phone (or something) somewhere off your network that you can have calls diverted to? Does anyone have info about how this power outage happened? The analysis of 365 Main's outage was interesting - I wonder if Telecom will provide the same level of info? -- Nathan Ward

On Mon, 5 Nov 2007, Neil Gardner wrote:

Anyone else pretty pleased and impressed with how well the people and organisations involved dealt with a failure in the automated systems? Personally it makes me a bit happier to realise that some genuinely critical services have pretty solid backup processes that by all accounts so far were followed and performed satisfactorily.

I'll chip in here with something that may or may not amuse/enrage/bemuse people on the list: Some communities, especially ones that are becoming affluent as townies move out of the cities in order to get a better lifestyle, are complaining about the traditional volunteer fire siren - the big pole-mounted beastie that is fit to wake the dead. They don't like the noise, it wakes the children, they look ugly, etc. A consequence of the paging failure in Auckland/Northland is that those sirens were the only way of notifying volunteer fire fighters in affected communities of a call. Take those sirens away, and it'd be back to the old telephone tree, assuming that the communications centres could find a phone number for someone in the brigade to kick it off in the first place. Moral? If someone objects to something because it offends their aesthetics, consider the consequences if it's removed and things break. Some "ugly" solutions have their place, if you have need for a truly robust system. -- Matthew Poole "Don't use force. Get a bigger hammer."

Well that broadened it out a bit. Re "Same goes with power pylons. Remove them all, especially the ones to their house." I suspect that the Power Pylons being referred to are of no benefit to those they affect given that the power supplies Auckland not the farms they blight. Cheers Wayne -----Original Message----- From: Phillip Hutchings [mailto:phillip.hutchings(a)sitharus.com] Sent: Monday, 5 November 2007 02:27 To: nznog Subject: Re: [nznog] BBBZZZZZTTTT: Where did all the lights go? On 5/11/2007, at 14:16, Matthew Poole wrote:

On Mon, 5 Nov 2007, Neil Gardner wrote:

...

Anyone else pretty pleased and impressed with how well the people and organisations involved dealt with a failure in the automated systems? Personally it makes me a bit happier to realise that some genuinely critical services have pretty solid backup processes that by all accounts so far were followed and performed satisfactorily.

I'll chip in here with something that may or may not amuse/enrage/ bemuse people on the list: Some communities, especially ones that are becoming affluent as townies move out of the cities in order to get a better lifestyle, are complaining about the traditional volunteer fire siren - the big pole-mounted beastie that is fit to wake the dead. They don't like the noise, it wakes the children, they look ugly, etc. A consequence of the paging failure in Auckland/Northland is that those sirens were the only way of notifying volunteer fire fighters in affected communities of a call. Take those sirens away, and it'd be back to the old telephone tree, assuming that the communications centres could find a phone number for someone in the brigade to kick it off in the first place.

Moral? If someone objects to something because it offends their aesthetics, consider the consequences if it's removed and things break. Some "ugly" solutions have their place, if you have need for a truly robust system.

I'd like to have a militant approach to such silliness. Take down the siren, set fire to their house and say "Gee, wouldn't it be great if we had some siren that could alert the firemen to the call?". Same goes with power pylons. Remove them all, especially the ones to their house. I don't have much patience for people who want all the benefits to modern life but none of the downsides, and I still look at people building on the unused land next to railway lines and wonder how long it'll take for them to figure out why nobody had that 'brilliant' idea before. -- Phillip Hutchings http://www.sitharus.com/ _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog