Hi All
Imagine any customers would have received this message already, but in
case you didn't receive it and/or are interested... see below.
Cheers,
Stephen
---------- Forwarded message ----------
From: support(a)discountdomains.co.nz
Date: Thu, Mar 25, 2010 at 6:40 PM
Subject: Network Outage 24/03/10 - 25/03/10
To:
Outage Summary
Dear Customer,
Following Tuesday night's reported outage (21.30-24.00) which was
attributed to a core switch intermission failure, last night the same
symptoms occurred (commencing 19.30). Clearly this highlighted that the
corrective action of the previous night i.e. the replacement of both
core switches deferred the issue rather than provided a permanent
resolution.
Last night the fault was again identified by our network management
software and the team reassembled consisting of the CTO, Sys-Admins and
management. The issue was immediately escalated to our external
maintenance support teams (CheckPoint firewall provider and hardware
provider) as is standard practice for an outage of this significance.
This identified that the fault appeared to be within the Checkpoint
firewall clustering software (dual redundancy).
With the assistance of Checkpoint engineers the decision was made to
split the firewall cluster and run them as individual stand alone units
to resurrect the network. This appeared to temporarily solve the issue
at 00.15. For context the firewall servers are running at 15-20% whilst
not clustered i.e. with very low levels of utilisation for the spec of
the equipment.
At 02.45 the network failed again. The team were still onsite monitoring
the network. Our firewall maintenance providers were again called who
arranged for patches to be downloaded. At 05.10 the patches were
installed and the firewall management server reconfigured to accommodate
the patch upgrade. This did not provide a permanent fix.
During more sociably acceptable hours we reached out to our friends in
Gen-i to help source checkpoint firewall hardware and to provide 'men on
the ground' to help support our technical team that had worked through
the night. In addition to this a decision was taken to move some core
applications to the old network (ASA) that was still functioning as was
not reliant on the check point firewalls. These include
DiscountDomains.co.nz, Email (inbound and outbound) and Digiweb.co.nz.
However the core network was re-established without the need to deploy
this second network with the core applications migrated.
Low level analysis with the assistance of Checkpoint engineers in the
USA identified high volumes of fragmented packets originating from one
of our shared virtual hosting servers to be the root cause of the issue.
These packets were flooding the firewalls and causing the outage. The
source of these packets was identified and blocked at 13:50. The
checkpoint firewalls then returned to normal service which finally
brought the network back on line at approximately 14:00 hours.
Like all hosting companies, we do not exercise strict control over the
content that customers upload to their websites. It appears that one
customer site was compromised, which in turn caused the flood of
malformed packets to the firewalls. Our internal network analysis
software did not identify these packets as they were not 'standard'
TCP/IP traffic.
In order to prevent this level of disruption in future we intend to move
all shared virtual hosting customers behind a separate firewall that is
issolated from the rest of our networks. This will ensure that should
there be any re-ocurrence the offending server is quarantined, and does
not cause the kind of outage we have just experienced.
We do sincerely apologise for this outage. These problems are
extraordinarily difficult to diagnose, and we are greatful for the
assistance provided by CheckPoint engineers in the USA, and local Gen-i
network engineers who have complemented the efforts of our own technical
team.
Should you require a more technical update, please contact Shaun
Williams our CTO (Shaun(a)digiweb.co.nz) or please contact me on my email
(Adrian.grant(a)digiweb.co.nz)or directly on my cell (021 626 484).
Once again our apologies for this critical issue and thank you for your
continued support.
With Regards,
Adrian Grant
Managing Director
Discount Domains Limited
Contact us by calling +64 3 961 9554 for fast and friendly service, or
simply email info(a)discountdomains.co.nz
Unsubscribe me from this mailing list
This newsletter was authorised by Brendan McNeill, the Managing Director
for Digiweb NZ LTD and can be contacted on +64 3 961 9554
http://newsletter.digiweb.net.nz/open.php?M=58994&N=199&L=52
DISCLAIMER
This e-mail is intended for the addressee only and may contain information which is subject to legal privilege. The contents are not necessarily the official view or communication of the Ministry of Education. If you are not the intended recipient you must not use, disclose, copy or distribute this e-mail or any information in, or attached to it. If you have received this e-mail in error, please contact the sender immediately or return the original message to the Ministry by e-mail, and destroy any copies. The Ministry does not accept any liability for changes made to this e-mail or attachments after sending.
All e-mails have been scanned for viruses and content by security software. The Ministry reserves the right to monitor all e-mail communications through its network.
