This is the formal reply from TCL on
a email I sent them two days ago.
Last
Sunday morning between 0100-0600 during the regular
planned maintanance window we upgraded 90% of the IP Networks' access
switches to a new version of code.
The
main reason for this upgrade was to introduce software
into the core switches that would support 'hitless upgrades' upon
the next
release of processor cards. This feature added resiliance to the IP Network
in that we could upgrade software on the core switches and customers would
not notice any outage.
During
the course of Sunday day/night there were a number of
customers who appeared to have unrelated issues and engineers worked through
the evening to resolve these.
On
Monday morning, when most customers started work, other
network faults were reported. Along with some unrelated issues, it also
appeared that there were issues with regard to the software upgrade on
Sunday morning. About 12% of IP customers were affected with software issues
related to the upgrade.
After
further diagnosis, it was decided to roll back to the
previous, known good, software version of code on the core access switches
in Auckland, Wellington and Christchurch. This was achieved by around 11am
Monday morning and appeared to help restore service to affected customers.
>From that point on, it was decided to work through all known affected
customers with issues and rollback the software on the affected switches
that they are connected to. This work carried on through the day on a case
by case basis.
This
process will continue, as required, and engineers will
work with any further affected customer.
While
the above is occuring, engineers are working with
suppliers to resolve the issues and then a plan can be devised to complete
the upgrade.
We
regret the impact this has had on customers. Significant
steps were taken prior to the software change to avoid this happening.
It is
important to note that this version of GA (or General availability)
code
had gone through extensive testing in the TelstraClear IP Lab and had been
running successfully online in parts of the Network for several weeks.
Russell Prince
TelstraClear Business Consultant
Drew Collins
Group Communications Manager
Group IT Services NZ
APN Holdings NZ Ltd
My DDI: +64 9 373 9573
My Mobile +64 21 823268
My Fax: +64 9 373 6411
Ph: +64 9 379 5050
My eMail: drew_collins@apn.co.nz
Website: www.apn.co.nz
***************************************************************************
This eMail may contain privileged and confidential information intended
only for the use of the intended recipient. If you are not the intended
recipient of this message, any use, dissemination, distribution or
reproduction of this message is prohibited. Any views expressed in this
message are those of the individual sender and may not necessarily reflect
the views of APN News & Media NZ Ltd.
For more information on APN News & Media NZ Ltd please visit our
web site at http://www.apn.co.nz
***************************************************************************