We've now added a "maximum-prefix 4096" to every peer on the route servers.

Hopefully this should prevent a re-occurrence before we are able to upgrade the route server hardware.

Dylan


On Tue, 2010-03-16 at 16:21 +1300, Dylan Hall wrote:
Both route servers are functioning as usual again.

We've rolled back the software upgrade on rs1.

We've spoken to the peer with the full route table and they have rectified the issue at their end.

If anyone requires an incident report for management please contact me off list :)


Dylan


On Tue, 2010-03-16 at 15:43 +1300, Dylan Hall wrote:
One of the route servers (rs2) is functioning again, the other is still having issues[*].

It looks like one of our peers hit us with a full route table and both route servers ran out of memory and crashed.

We're looking into getting some larger boxes to host the route servers although this won't happen today.

We won't be processing any requests for routing updates until we have both route servers functioning again. Hopefully this won't take too long.


Dylan

[*] rs1 is unwell because we've tried to install a newer version of quagga which is now refusing to start.



On Tue, 2010-03-16 at 14:22 +1300, Dylan Hall wrote:
Something appears to have happened to both of the APE route servers shortly after 2pm today.

We're investigating at the moment.

Dylan

_______________________________________________
NZNOG mailing list
NZNOG@list.waikato.ac.nz
http://list.waikato.ac.nz/mailman/listinfo/nznog


_______________________________________________
NZNOG mailing list
NZNOG@list.waikato.ac.nz
http://list.waikato.ac.nz/mailman/listinfo/nznog


_______________________________________________
NZNOG mailing list
NZNOG@list.waikato.ac.nz
http://list.waikato.ac.nz/mailman/listinfo/nznog