One would assume however that being based at 220q they will also pickup AKL-IX in due course as the other main Auckland IX. On Wed, 19 Sep 2018 at 10:47 AM, Michael Fincham wrote:

On Wed, 19 Sep 2018 10:41:45 +1200 Michael Fincham wrote:

...

I guess hello HE and welcome to AKL-IX?

Sorry, Megaport.

-- Michael _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

-- Kind Regards Liam Farr Maxum Data +64-9-950-5302

HE peers everywhere they can, so yes I assume you'll see them on AKL-IX too... APE might depend if they see value, last I checked between MegaIX and AKL-IX you collected most of the routes on APE direct or indirect. To be honest none of the IX's in AKL have been truely domestic for a long time, there's been chunks of AU and other parts of the world, CDNs, global OTTs (Microsoft etc) presenting non NZ routes. This has then flowed into the concept of domestic transit, I've certainly seen bits of international in there a long time.. as did AU when we used to talk about domestic table too (and NZ was always in it!) Ultimately it's probably a good thing, new names in a market is new competition and even if you don't shop with them you'll probably gain from the market getting a shake. As a reminder, HE will give you even MORE routes if you bring up a bilateral with them email the peering email @ https://peeringdb.com/net/291 or talk to Walt from Hurricane.. I'll throw him on the CC as I'm not sure if he's on the list. Maybe he'll visit NZNOG next year. Gav On Wed, 19 Sep 2018 at 06:57, Liam Farr wrote:

One would assume however that being based at 220q they will also pickup AKL-IX in due course as the other main Auckland IX.

On Wed, 19 Sep 2018 at 10:47 AM, Michael Fincham wrote:

...

On Wed, 19 Sep 2018 10:41:45 +1200 Michael Fincham wrote:

...

I guess hello HE and welcome to AKL-IX?

Sorry, Megaport.

-- Michael _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

-- Kind Regards

Liam Farr

Maxum Data +64-9-950-5302 _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

-- Global Interconnection Director Megaport https://www.megaport.com +61 498 498 458

Yes, This is exactly why this happened. Vocus made a change around 0050 this morning. We dropped some peerings when the prefix limits breached. If your box isn't coping you could consider rejecting routes received from Megaport with HE in the path. Cam -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz On Behalf Of Mike Taylor Sent: Wednesday, 19 September 2018 12:06 PM To: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Hello Hurricane Electric, doubling the size of our domestic table? Eek, I wonder if this is why my NZ domestic table size had a blow out at around 0100 this morning and tipped one of my (smaller) boxes over the edge... I've spent all morning trying to triage the damn thing, knowing that the BGP table had blown out and taken all of the TCAM memory... We're still on the old style internet ISP plan from FX days; 1x International feed with a default route only 1x Domestic feed with All NZ Domestic routes, (and now a shedload more in this table) Cheers, Mike On 19/09/18 11:33, Gavin Tweedie wrote:

HE peers everywhere they can, so yes I assume you'll see them on AKL-IX too... APE might depend if they see value, last I checked between MegaIX and AKL-IX you collected most of the routes on APE direct or indirect.

To be honest none of the IX's in AKL have been truely domestic for a long time, there's been chunks of AU and other parts of the world, CDNs, global OTTs (Microsoft etc) presenting non NZ routes. This has then flowed into the concept of domestic transit, I've certainly seen bits of international in there a long time.. as did AU when we used to talk about domestic table too (and NZ was always in it!)

Ultimately it's probably a good thing, new names in a market is new competition and even if you don't shop with them you'll probably gain from the market getting a shake.

As a reminder, HE will give you even MORE routes if you bring up a bilateral with them email the peering email @  https://peeringdb.com/net/291 or talk to Walt from Hurricane.. I'll throw him on the CC as I'm not sure if he's on the list. Maybe he'll visit NZNOG next year.

Gav

On Wed, 19 Sep 2018 at 06:57, Liam Farr mailto:liam(a)maxumdata.com> wrote:

One would assume however that being based at 220q they will also pickup AKL-IX in due course as the other main Auckland IX.

On Wed, 19 Sep 2018 at 10:47 AM, Michael Fincham mailto:michael(a)hotplate.co.nz> wrote:

On Wed, 19 Sep 2018 10:41:45 +1200 Michael Fincham mailto:michael(a)hotplate.co.nz> wrote:

> I guess hello HE and welcome to AKL-IX?

Sorry, Megaport.

-- Michael _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

-- Kind Regards

Liam Farr

Maxum Data +64-9-950-5302 _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

-- Global Interconnection Director Megaport https://www.megaport.com +61 498 498 458

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

OK, so the routes are on AKL-IX now [cue a bunch more peoples import limits breaking again]. Did we not learn anything from the last time this broke a whole bunch of stuff? HE needs to be giving us some heads up before they turn up here and increase the size of NZ's domestic route table by an order of magnitude... Hello? -- Michael

On Wed, 19 Sep 2018 16:23:56 +1200 Dave Mill wrote:

Megaport AKL emailed their customers multiple times about HE turning up their services.

NZIX also emailed.

So I assume its just 'domestic' transit providers that maybe aren't notifying us?

I don't think this is the right way to look at it. Those routes are going to end up in everyone's "domestic" feeds. HE should have been here on NZNOG months ago making sure everyone was OK with this and ready. Nobody is running a 10x safety margin on their BGP limits... -- Michael

They did notify, "months ago" back in June... (as below) and again yesterday. ------------------------------- Date: Tue, 18 Sep 2018 at 11:30 Subject: Re: [Megaport Auckland IX] Upcoming large increase in prefixes from MegaIX Auckland Route Servers To: Hi AKL peers, Just a reminder regarding the increase of prefixes from the MegaIX Auckland Route Servers mentioned below. We are expecting the increase to occur in the next few days depending on peer turn-up schedule. Gavin On Fri, 29 Jun 2018 at 07:54, Gavin Tweedie wrote:

Hi Auckland Peers,

Please be aware that soon(ish) we are expecting an increase in prefixes on the Auckland MegaIX Route Servers driven by Hurricane Electric (AS6939) https://peeringdb.com/net/291 connecting to the IX within the next 60 days.

If you set a maximum prefix (MaxPFX) limit on the BGP neighbours with our route servers or if you have other limitations in terms of prefixes you can receive from the route server you should start preparing.

Our new recommended MaxPFX setting is always maintained in route-server peeringdb entries (Auckland @ https://peeringdb.com/net/12814) in case you automate this.

For reference our Auckland Route servers are AS63839,

43.243.22.1 & 43.243.22.2

2001:dea:0:40::1 & 2001:dea:0:40::2

The new recommended minimum MaxPFX settings are,

IPv4 65,000

IPv6 35,000

If you are not sure if your router is able to handle such a large table I would recommend filtering by AS-Path routes received from AS6939 from the route server while you investigate it so that you can experiment in a window that suits you, not just when the routes show up.

If you require any assistance please feel free to reach out to myself or our NOC team via our portal live chat or via email at support(a)megaport.com

Regards

Gavin

--

Global Interconnection Director

Megaport https://www.megaport.com/

+61 498 498 458

On Wed, 19 Sep 2018 at 16:25, Michael Fincham wrote:

On Wed, 19 Sep 2018 16:23:56 +1200 Dave Mill wrote:

...

Megaport AKL emailed their customers multiple times about HE turning up their services.

NZIX also emailed.

So I assume its just 'domestic' transit providers that maybe aren't notifying us?

I don't think this is the right way to look at it. Those routes are going to end up in everyone's "domestic" feeds. HE should have been here on NZNOG months ago making sure everyone was OK with this and ready.

Nobody is running a 10x safety margin on their BGP limits...

-- Michael _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

-- Kind Regards Liam Farr Maxum Data +64-9-950-5302

If your not on the exchange directly, you only buy (domestic) transit from an upstream, and your upstream didn't notify you... perhaps its time to find a new upstream or have a long hard chat with your account manager / their ops team? On Wed, 19 Sep 2018 at 16:29, Michael Fincham wrote:

On Wed, 19 Sep 2018 16:27:16 +1200 Liam Farr wrote:

...

They did notify, "months ago" back in June... (as below) and again yesterday.

Yes, but how is this meant to filter out to all the other operators not on that exchange exactly, who have spent much of the day fixing things and helping other people fix things?

-- Michael _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

-- Kind Regards Liam Farr Maxum Data +64-9-950-5302

On Wed, 19 Sep 2018 16:32:09 +1200 Nathan Ward wrote:

Through whoever you buy domestic transit from - communication for these sorts of changes, and if required protecting customers who want to run such a service on c2610s, is a core part of such a service. If your transit providers are unable to do this I would suggest looking towards other operators.

This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of stuff, doesn't even bother to post to NZNOG letting us know they're going to break things, and you're just blaming my transit provider? Also, none of the IXes posted on NZNOG? -- Michael

On Wed, Sep 19, 2018 at 4:35 PM, Michael Fincham wrote:

On Wed, 19 Sep 2018 16:32:09 +1200 Nathan Ward wrote:

...

Through whoever you buy domestic transit from - communication for these sorts of changes, and if required protecting customers who want to run such a service on c2610s, is a core part of such a service. If your transit providers are unable to do this I would suggest looking towards other operators.

This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of stuff, doesn't even bother to post to NZNOG letting us know they're going to break things, and you're just blaming my transit provider?

Also, none of the IXes posted on NZNOG?

Joe emailed NZNOG at 11.28am today. Sorry that there wasn't more notice. Dave

On Wed, 19 Sep 2018 16:39:50 +1200 Liam Farr wrote:

IMO I think its great that a large carrier has come to our neck of the woods, and they are openly peering on both our major IX's. This is a good thing for internet in NZ.

No contest. And I'm sure HE isn't here intentionally breaking the Internet. It is great they're here. I am just unhappy that nobody thought that maybe increasing the size of the table we all carry as "national" by this much wasn't worth mentioning to the operator community. I really don't think this is a "your transit provider" is bad moment. Who actually got a message about this from their transit provider? -- Michael

Don't forget that the NZ/AU concept of 'domestic table' is a fairly unique concept that isn't seen in much of the rest of the world- generally any feed is going to be default, a peering feed (customer routes), or a full table.... Doubt anyone thought of it ;). Congrats to Mike & Walt on extending to NZ... pretty cool to see a worldwide carrier finally making it there - fantastic to see how far the NZ internet has progressed! On Tue, Sep 18, 2018 at 9:47 PM Bill wrote:

Perhaps those that had issues should use such tools as prefix limits on their inbound feeds. We get notified by our edge routers quite regularly that the number of prefixes has or is about to max out.

Sent from my iPhone

On 19/09/2018, at 4:39 PM, Liam Farr wrote:

IMO I think its great that a large carrier has come to our neck of the woods, and they are openly peering on both our major IX's. This is a good thing for internet in NZ.

Notifications were sent to the affected parties who peer, months in advance.

I don't believe HE had any malicious intent to break your internet.

If your stuff broke because you have a shit upstream, fix it and move on.

On Wed, 19 Sep 2018 at 16:35, Michael Fincham wrote:

...

On Wed, 19 Sep 2018 16:32:09 +1200 Nathan Ward wrote:

...

Through whoever you buy domestic transit from - communication for these sorts of changes, and if required protecting customers who want to run such a service on c2610s, is a core part of such a service. If your transit providers are unable to do this I would suggest looking towards other operators.

This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of stuff, doesn't even bother to post to NZNOG letting us know they're going to break things, and you're just blaming my transit provider?

Also, none of the IXes posted on NZNOG?

-- Michael _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

-- Kind Regards

Liam Farr

Maxum Data +64-9-950-5302

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

Buying 'domestic' is a way of paying for the privilege of sending/receiving traffic to Spark :) On Wed, Sep 19, 2018 at 5:02 PM, Tom Paseka wrote:

If I can ask, what does a "domestic" table genuinely bring you. If you carry routes to your edge(s) that have both peering and transit, do you need to extend the DFZ all the way through your network? If you've got a single exit path (or a few) do you need to extend the domestic routes any further?

Another thing to note: BGP Sanity. Apply AS_PATH filters to all your peers. This doubly so if you're extending the routes throughout your network. Much of HE's routes might not be relevant to your network and could have easily been sanitized with some nice filters.

Also congrats to HE/Team! -Tom

On Tue, Sep 18, 2018 at 9:50 PM Tim Hoffman wrote:

...

Don't forget that the NZ/AU concept of 'domestic table' is a fairly unique concept that isn't seen in much of the rest of the world- generally any feed is going to be default, a peering feed (customer routes), or a full table.... Doubt anyone thought of it ;).

Congrats to Mike & Walt on extending to NZ... pretty cool to see a worldwide carrier finally making it there - fantastic to see how far the NZ internet has progressed!

On Tue, Sep 18, 2018 at 9:47 PM Bill wrote:

...

Perhaps those that had issues should use such tools as prefix limits on their inbound feeds. We get notified by our edge routers quite regularly that the number of prefixes has or is about to max out.

Sent from my iPhone

On 19/09/2018, at 4:39 PM, Liam Farr wrote:

IMO I think its great that a large carrier has come to our neck of the woods, and they are openly peering on both our major IX's. This is a good thing for internet in NZ.

Notifications were sent to the affected parties who peer, months in advance.

I don't believe HE had any malicious intent to break your internet.

If your stuff broke because you have a shit upstream, fix it and move on.

On Wed, 19 Sep 2018 at 16:35, Michael Fincham wrote:

...

On Wed, 19 Sep 2018 16:32:09 +1200 Nathan Ward wrote:

...

Through whoever you buy domestic transit from - communication for these sorts of changes, and if required protecting customers who want to run such a service on c2610s, is a core part of such a service. If your transit providers are unable to do this I would suggest looking towards other operators.

This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of stuff, doesn't even bother to post to NZNOG letting us know they're going to break things, and you're just blaming my transit provider?

Also, none of the IXes posted on NZNOG?

-- Michael _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

-- Kind Regards

Liam Farr

Maxum Data +64-9-950-5302

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

Hi, It’s been around here for a long time. Perhaps it’s a bit dated these days, sure. Getting through cables was expensive, so people offered connectivity to other NZ networks at a significantly cheaper price (think at least an order of magnitude). It wasn’t always delivered as seperate transit - often it would be a single connection but “domestic” was zero rated, and “international” was billed.. or, similar thing but shaping rather than billing. You get the idea. A large number of regular business Internet connections had this sort of thing on them I dunno, 10 years ago maybe? Few have it now, but it won’t have been too long ago that it got like that. I’m pretty sure my home internet had some different “international” component on it at some point, too. It’s meant we’ve always cared a lot about local content, which has meant our peering and the community around it has been strong, even when the economics aren’t quite as glaring anymore. It’s also a hard thing to take away - providers offering both “domestic" and “international” transit have to either take a stab at their costs (like they do for US vs. AU for the “international” part) and risk a loss, or they have to effectively say “domestic is the same price as international”. That’s a fast way to have your transit customers move elsewhere, I’d say. Worth noting, most “international” products also include “domestic” routes and transit. It’s really “domestic” and “global”.

On 19/09/2018, at 5:13 PM, Tom Paseka wrote:

I'd call that partial transit ;-)

(BGP sanity questions remain).

On Tue, Sep 18, 2018 at 10:05 PM Dave Mill mailto:dave(a)mill.net.nz> wrote: Buying 'domestic' is a way of paying for the privilege of sending/receiving traffic to Spark :)

On Wed, Sep 19, 2018 at 5:02 PM, Tom Paseka mailto:tom+lists(a)cloudflare.com> wrote: If I can ask, what does a "domestic" table genuinely bring you. If you carry routes to your edge(s) that have both peering and transit, do you need to extend the DFZ all the way through your network? If you've got a single exit path (or a few) do you need to extend the domestic routes any further?

Another thing to note: BGP Sanity. Apply AS_PATH filters to all your peers. This doubly so if you're extending the routes throughout your network. Much of HE's routes might not be relevant to your network and could have easily been sanitized with some nice filters.

Also congrats to HE/Team! -Tom

On Tue, Sep 18, 2018 at 9:50 PM Tim Hoffman mailto:tim(a)hoffman.net.nz> wrote: Don't forget that the NZ/AU concept of 'domestic table' is a fairly unique concept that isn't seen in much of the rest of the world- generally any feed is going to be default, a peering feed (customer routes), or a full table.... Doubt anyone thought of it ;).

Congrats to Mike & Walt on extending to NZ... pretty cool to see a worldwide carrier finally making it there - fantastic to see how far the NZ internet has progressed!

On Tue, Sep 18, 2018 at 9:47 PM Bill mailto:bill(a)wjw.nz> wrote:

Perhaps those that had issues should use such tools as prefix limits on their inbound feeds. We get notified by our edge routers quite regularly that the number of prefixes has or is about to max out.

Sent from my iPhone

On 19/09/2018, at 4:39 PM, Liam Farr mailto:liam(a)maxumdata.com> wrote:

...

IMO I think its great that a large carrier has come to our neck of the woods, and they are openly peering on both our major IX's. This is a good thing for internet in NZ.

Notifications were sent to the affected parties who peer, months in advance.

I don't believe HE had any malicious intent to break your internet.

If your stuff broke because you have a shit upstream, fix it and move on.

On Wed, 19 Sep 2018 at 16:35, Michael Fincham mailto:michael(a)hotplate.co.nz> wrote: On Wed, 19 Sep 2018 16:32:09 +1200 Nathan Ward mailto:nznog(a)daork.net> wrote:

...

Through whoever you buy domestic transit from - communication for these sorts of changes, and if required protecting customers who want to run such a service on c2610s, is a core part of such a service. If your transit providers are unable to do this I would suggest looking towards other operators.

This is a bad attitude IMO. An operator turns up in NZ, breaks a bunch of stuff, doesn't even bother to post to NZNOG letting us know they're going to break things, and you're just blaming my transit provider?

Also, none of the IXes posted on NZNOG?

-- Michael _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog https://list.waikato.ac.nz/mailman/listinfo/nznog

-- Kind Regards

Liam Farr

Maxum Data +64-9-950-5302 _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog https://list.waikato.ac.nz/mailman/listinfo/nznog

---

NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog https://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

Did the AKL-IX people send an email ahead of this? I ask because we didn't see anything so I'm wondering if we just lost it in a spam folder somewhere or if others got missed also. Thanks, Dylan On 19 September 2018 at 16:23, Dave Mill wrote:

Megaport AKL emailed their customers multiple times about HE turning up their services.

NZIX also emailed.

So I assume its just 'domestic' transit providers that maybe aren't notifying us?

Dave

On Wed, Sep 19, 2018 at 4:17 PM, Michael Fincham wrote:

...

OK, so the routes are on AKL-IX now [cue a bunch more peoples import limits breaking again].

Did we not learn anything from the last time this broke a whole bunch of stuff?

HE needs to be giving us some heads up before they turn up here and increase the size of NZ's domestic route table by an order of magnitude...

Hello?

-- Michael _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog