From that point on, it was decided to work through all known affected customers with issues and rollback the software on the affected switches
This is the formal reply from TCL on a email I sent them two days ago. Last Sunday morning between 0100-0600 during the regular planned maintanance window we upgraded 90% of the IP Networks' access switches to a new version of code. The main reason for this upgrade was to introduce software into the core switches that would support 'hitless upgrades' upon the next release of processor cards. This feature added resiliance to the IP Network in that we could upgrade software on the core switches and customers would not notice any outage. During the course of Sunday day/night there were a number of customers who appeared to have unrelated issues and engineers worked through the evening to resolve these. On Monday morning, when most customers started work, other network faults were reported. Along with some unrelated issues, it also appeared that there were issues with regard to the software upgrade on Sunday morning. About 12% of IP customers were affected with software issues related to the upgrade. After further diagnosis, it was decided to roll back to the previous, known good, software version of code on the core access switches in Auckland, Wellington and Christchurch. This was achieved by around 11am Monday morning and appeared to help restore service to affected customers. that they are connected to. This work carried on through the day on a case by case basis. This process will continue, as required, and engineers will work with any further affected customer. While the above is occuring, engineers are working with suppliers to resolve the issues and then a plan can be devised to complete the upgrade. We regret the impact this has had on customers. Significant steps were taken prior to the software change to avoid this happening. It is important to note that this version of GA (or General availability) code had gone through extensive testing in the TelstraClear IP Lab and had been running successfully online in parts of the Network for several weeks. Russell Prince TelstraClear Business Consultant Drew Collins Group Communications Manager Group IT Services NZ APN Holdings NZ Ltd My DDI: +64 9 373 9573 My Mobile +64 21 823268 My Fax: +64 9 373 6411 Ph: +64 9 379 5050 My eMail: drew_collins(a)apn.co.nz Website: www.apn.co.nz *************************************************************************** This eMail may contain privileged and confidential information intended only for the use of the intended recipient. If you are not the intended recipient of this message, any use, dissemination, distribution or reproduction of this message is prohibited. Any views expressed in this message are those of the individual sender and may not necessarily reflect the views of APN News & Media NZ Ltd. For more information on APN News & Media NZ Ltd please visit our web site at http://www.apn.co.nz ***************************************************************************