Outage Summary

Dear Customer,

Following Tuesday night’s reported outage (21.30-24.00) which was attributed to a core switch intermission failure, last night the same symptoms occurred (commencing 19.30). Clearly this highlighted that the corrective action of the previous night i.e. the replacement of both core switches deferred the issue rather than provided a permanent resolution.

Last night the fault was again identified by our network management software and the team reassembled consisting of the CTO, Sys-Admins and management. The issue was immediately escalated to our external maintenance support teams (CheckPoint firewall provider and hardware provider) as is standard practice for an outage of this significance. This identified that the fault appeared to be within the Checkpoint firewall clustering software (dual redundancy).

With the assistance of Checkpoint engineers the decision was made to split the firewall cluster and run them as individual stand alone units to resurrect the network. This appeared to temporarily solve the issue at 00.15. For context the firewall servers are running at 15-20% whilst not clustered i.e. with very low levels of utilisation for the spec of the equipment.

At 02.45 the network failed again. The team were still onsite monitoring the network. Our firewall maintenance providers were again called who arranged for patches to be downloaded. At 05.10 the patches were installed and the firewall management server reconfigured to accommodate the patch upgrade. This did not provide a permanent fix.

During more sociably acceptable hours we reached out to our friends in Gen-i to help source checkpoint firewall hardware and to provide ‘men on the ground’ to help support our technical team that had worked through the night. In addition to this a decision was taken to move some core applications to the old network (ASA) that was still functioning as was not reliant on the check point firewalls. These include DiscountDomains.co.nz, Email (inbound and outbound) and Digiweb.co.nz. However the core network was re-established without the need to deploy this second network with the core applications migrated.

Low level analysis with the assistance of Checkpoint engineers in the USA identified high volumes of fragmented packets originating from one of our shared virtual hosting servers to be the root cause of the issue. These packets were flooding the firewalls and causing the outage.  The source of these packets was identified and blocked at 13:50. The checkpoint firewalls then returned to normal service which finally brought the network back on line at approximately 14:00 hours.

Like all hosting companies, we do not exercise strict control over the content that customers upload to their websites.  It appears that one customer site was compromised, which in turn caused the flood of malformed packets to the firewalls.  Our internal network analysis software did not identify these packets as they were not ‘standard’ TCP/IP traffic.

In order to prevent this level of disruption in future we intend to move all shared virtual hosting customers behind a separate firewall that is issolated from the rest of our networks.  This will ensure that should there be any re-ocurrence the offending server is quarantined, and does not cause the kind of outage we have just experienced.

We do sincerely apologise for this outage. These problems are extraordinarily difficult to diagnose, and we are greatful for the assistance provided by CheckPoint engineers in the USA, and local Gen-i network engineers who have complemented the efforts of our own technical team.

Should you require a more technical update, please contact Shaun Williams our CTO (Shaun@digiweb.co.nz) or please contact me on my email (Adrian.grant@digiweb.co.nz)or directly on my cell (021 626 484).

Once again our apologies for this critical issue and thank you for your continued support.

With Regards, 

Adrian Grant
Managing Director
Discount Domains Limited

Contact us by calling +64 3 961 9554 for fast and friendly service, or simply email info@discountdomains.co.nz

Unsubscribe me from this mailing list

This newsletter was authorised by Brendan McNeill, the Managing Director for Digiweb NZ LTD and can be contacted on +64 3 961 9554