As promised, a more detailed write up about the Citylink outage in
Auckland earlier today.
Jumping straight to the chase, we deleted the native vlan from our VTP
server (and hence most of our switches) in Auckland. We believe this
caused some of our switches to stop talking spanning tree on those
trunks and resulted in a layer 2 loop/broadcast storm. Service was
restored when an engineer re-created the vlan on the VTP server.
We have been making a number of changes over the last 12 months with the
goal of improving stability on the network[1]. Most of these changes
have revolved around implementing best practise and ensuring consistency
across all our switches.
The particular changes that resulted in the outage where around the
interface configuration on Citylink switch to switch trunks.
Previously a trunk configuration would look something like:
interface GigabitEthernet0/1
switchport trunk native vlan 999
switchport trunk allowed vlan 1-4095
Under normal circumstances no traffic should be using the native vlan.
To enforce this situation we removed the native vlan from the allowed
list, e.g.
interface GigabitEthernet0/1
switchport trunk native vlan 999
switchport trunk allowed vlan 1-998,1000-4095
This change was completed without any issues.
The second part of the tidy up was to remove the native vlan completely
from the network. We lab tested this scenario and there was no impact
when removing the native vlan.
What we suspect happened is one or more switches had their spanning tree
process bound to the native vlan. When the native vlan was removed the
switches stopped talking spanning tree resulting in a loop.
Service was restored by an engineer going to site and using the console
to re-create the native vlan on the VTP server.
Also, a big thanks to John from FX who assisted us restoring service
with a very timely visit to the Sky Tower :)
Again, sorry for any inconvenience caused.
Dylan Hall
Network Engineer
CityLink Ltd
[1] Sadly the irony here hasn't gone unnoticed.
