Async Routing across ape to pipe return via vocus & Internode speedtest

Barry Murphy

27 Feb 2012 27 Feb '12

8:52 p.m.

Hey Guys, I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus. I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF. Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth. I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all. Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported. thanks Barry

Show replies by date

Tim Price

27 Feb 27 Feb

8:59 p.m.

New subject: Async Routing across ape to pipe return via vocus & Internode speedtest

I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive. -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest Hey Guys, I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus. I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF. Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth. I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all. Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported. thanks Barry _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Barry Murphy

9:08 p.m.

I recall someone mentioning (Blair?) to perhaps tripple pre-pend PIPE IX's using vocus communities (http://tools.vocus.com.au/additionals/communities2.0.html). Not sure how productive this is either, anyone tried this? cheers Barry On Tue, 28 Feb 2012 09:59:19 +1300, Tim Price wrote:

...

I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest

Hey Guys,

I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus.

I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF.

Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth.

I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all.

Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported.

thanks Barry

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Tristram Cheer

9:09 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

Hey All, We had the same issue last year, Akamai decided to point us at Vocus's node and Akamai traffic coming from AKL server's under Vocus's control were returning our data over our International. We of course were very unhappy about this but we got nowhere with Vocus or Akamai about it (Basically told to sod off), The only way we found to fix it was to point the DNS zones for akamaitech.net and akamai.net to another DNS server in NZ that stayed pointed to CallPlus's Akamai node. Very frustrating given and difficult to resolve, The offending parties always seem to be of zero help in these sorts of cases. Cheers -- Tristram Cheer Network Architect Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tim Price Sent: Tuesday, 28 February 2012 9:59 a.m. To: 'Barry Murphy'; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive. -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest Hey Guys, I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus. I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF. Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth. I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all. Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported. thanks Barry _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Barry Murphy

9:39 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

After much debate with Akamai NOC, I finally referred them to Patrick Gilmore who posted here some time ago about akamai, they then fixed the routing and gave us callplus as priority and snap as secondary for akamai. A few months went by and all of a sudden we are back on Vocus for akamai. I've tried to contact the NOC again however after 3 emails over 2 months no luck, I then contacted Patrick again however he was at a conference for LINX in London, haven't had a fix on this. I'm tempted to hard set DNS too but don't want to break anything else :/ What DNS servers are you using that are recursive that force this traffic? I've heard offlist that there seem to be a couple of providers in the same boat with regards to Internode speedtest node. Barry On Tue, 28 Feb 2012 10:09:31 +1300, Tristram Cheer wrote:

...

Hey All,

We had the same issue last year, Akamai decided to point us at Vocus's node and Akamai traffic coming from AKL server's under Vocus's control were returning our data over our International. We of course were very unhappy about this but we got nowhere with Vocus or Akamai about it (Basically told to sod off), The only way we found to fix it was to point the DNS zones for akamaitech.net and akamai.net to another DNS server in NZ that stayed pointed to CallPlus's Akamai node.

Very frustrating given and difficult to resolve, The offending parties always seem to be of zero help in these sorts of cases.

Cheers

--

Tristram Cheer Network Architect

Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz

PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tim Price Sent: Tuesday, 28 February 2012 9:59 a.m. To: 'Barry Murphy'; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest

I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest

Hey Guys,

I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus.

I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF.

Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth.

I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all.

Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported.

thanks Barry

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Craig Whitmore

9:49 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

...

I've heard offlist that there seem to be a couple of providers in the same boat with regards to Internode speedtest node.

Looking at the pings of 59.167.169.4 (the host where the speedtest server is) it does look like the internode speedtest server IS in New Zealand (if you are connected to APE) If you are not connected via APE then I presume it goes international and then comes back to NZ. What do the traces look like from Telecom and TelstraClear? If it does go international then should internodes server really be called a NZ test server? Thanks

...

Mark Foster

9:55 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

On 28/02/12 10:49, Craig Whitmore wrote:

...

...
I've heard offlist that there seem to be a couple of providers in the same boat with regards to Internode speedtest node.

Looking at the pings of 59.167.169.4 (the host where the speedtest server is) it does look like the internode speedtest server IS in New Zealand (if you are connected to APE)

If you are not connected via APE then I presume it goes international and then comes back to NZ. What do the traces look like from Telecom and TelstraClear? If it does go international then should internodes server really be called a NZ test server?

Via TCL the route to that IP is rediculous: traceroute to 59.167.169.4 (59.167.169.4), 30 hops max, 40 byte packets 1 (etc) 2 * * * 3 (etc) 4 ie2-g-0-0-0.telstraclear.net (203.98.50.2) 21.696 ms 21.694 ms 21.691 ms 5 ge-0-2-0-1.xcore1.acld.telstraclear.net (203.98.50.251) 56.405 ms 56.890 ms 56.882 ms 6 203.167.233.10 (203.167.233.10) 21.903 ms 22.046 ms 22.033 ms 7 i-0-0-4-0.sydo-core02.bx.reach.com (202.84.142.153) 45.523 ms 48.698 ms 48.694 ms 8 TenGigE0-2-0-2.oxf-gw1.Sydney.telstra.net (203.50.13.53) 50.416 ms 50.198 ms 56.678 ms 9 Bundle-Ether1.ken-core4.Sydney.telstra.net (203.50.6.5) 57.438 ms 56.959 ms 56.191 ms 10 Bundle-Ether1.ken39.Sydney.telstra.net (203.50.6.146) 59.414 ms 49.442 ms 51.190 ms 11 optusc1.lnk.telstra.net (165.228.132.206) 42.697 ms 46.932 ms 45.686 ms 12 * * * 13 * * * 14 * * * 15 203.208.190.233 (203.208.190.233) 220.372 ms 220.128 ms 220.325 ms 16 203.208.166.61 (203.208.166.61) 219.874 ms 218.108 ms 221.121 ms 17 so-5-2-2-0.toknf-cr3.ix.singtel.com (203.208.173.94) 219.897 ms 203.208.166.174 (203.208.166.174) 220.340 ms so-5-2-3-0.toknf-cr3.ix.singtel.com (203.208.154.18) 222.360 ms 18 203.208.166.205 (203.208.166.205) 218.837 ms 219.869 ms so-5-1-2-0.toknf-cr3.ix.singtel.com (203.208.145.214) 222.107 ms 19 AS4739.ix.jpix.ad.jp (210.171.224.202) 191.834 ms 196.885 ms 203.208.166.205 (203.208.166.205) 220.863 ms 20 AS4739.ix.jpix.ad.jp (210.171.224.202) 201.378 ms 191.895 ms pos5-0-0.bdr1.syd4.internode.on.net (203.16.211.57) 315.817 ms 21 pos5-0-0.bdr1.syd4.internode.on.net (203.16.211.57) 321.073 ms 316.046 ms 314.551 ms 22 gi0-2-201.bdr1.akl1.internode.on.net (59.167.169.1) 319.556 ms * * Across APE it's a single hop.

Craig Whitmore

9:59 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

...

...
If you are not connected via APE then I presume it goes international and then comes back to NZ. What do the traces look like from Telecom and TelstraClear? If it does go international then should internodes server really be called a NZ test server?

Via TCL the route to that IP is rediculous:

New Zealand->Australia->Singapore->Japan->Australia(Again)->New Zealand(Again)

Tristram Cheer

9:53 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

That is Interesting that you got moved to CallPlus, The response from Akamai's Network Strategy guy in AU was they CAN'T alter the node selection at all since it "will result in breaking partner agreements". Needless to say we were a bit miffed that Kordia,Vocus and Akamai were all unwilling to fix the issue beyond "Buy More Int Transit" -- Tristram Cheer Network Architect Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: Barry Murphy [mailto:barry(a)unix.co.nz] Sent: Tuesday, 28 February 2012 10:40 a.m. To: Tristram Cheer Cc: Tim Price; nznog(a)list.waikato.ac.nz Subject: RE: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest After much debate with Akamai NOC, I finally referred them to Patrick Gilmore who posted here some time ago about akamai, they then fixed the routing and gave us callplus as priority and snap as secondary for akamai. A few months went by and all of a sudden we are back on Vocus for akamai. I've tried to contact the NOC again however after 3 emails over 2 months no luck, I then contacted Patrick again however he was at a conference for LINX in London, haven't had a fix on this. I'm tempted to hard set DNS too but don't want to break anything else :/ What DNS servers are you using that are recursive that force this traffic? I've heard offlist that there seem to be a couple of providers in the same boat with regards to Internode speedtest node. Barry On Tue, 28 Feb 2012 10:09:31 +1300, Tristram Cheer wrote:

...

Hey All,

We had the same issue last year, Akamai decided to point us at Vocus's node and Akamai traffic coming from AKL server's under Vocus's control were returning our data over our International. We of course were very unhappy about this but we got nowhere with Vocus or Akamai about it (Basically told to sod off), The only way we found to fix it was to point the DNS zones for akamaitech.net and akamai.net to another DNS server in NZ that stayed pointed to CallPlus's Akamai node.

Very frustrating given and difficult to resolve, The offending parties always seem to be of zero help in these sorts of cases.

Cheers

--

Tristram Cheer Network Architect

Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz

PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tim Price Sent: Tuesday, 28 February 2012 9:59 a.m. To: 'Barry Murphy'; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest

I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest

Hey Guys,

I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus.

I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF.

Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth.

I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all.

Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported.

thanks Barry

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Mark Foster

9:48 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

The problem with doing this is that Callplus may not be intending their Akamai node to be used by folks that aren't either their customer, or an organisation they have a commercial arrangement with to serve this content. In my last job we had a similar issue, we were going to wind up requiring our upstream to do some source-based-routing, so we didn't pursue that option as it wasn't considered practical. Instead perhaps ISP's need to consider setting up their own Akamai presence or formally arranging something with an ISP that does have it. (And inversely the ISPs that want to enforce this position would need to apply controls on their DNS server, which is itself annoying for other reasons...) Mark. On 28/02/12 10:09, Tristram Cheer wrote:

...

Hey All,

We had the same issue last year, Akamai decided to point us at Vocus's node and Akamai traffic coming from AKL server's under Vocus's control were returning our data over our International. We of course were very unhappy about this but we got nowhere with Vocus or Akamai about it (Basically told to sod off), The only way we found to fix it was to point the DNS zones for akamaitech.net and akamai.net to another DNS server in NZ that stayed pointed to CallPlus's Akamai node.

Very frustrating given and difficult to resolve, The offending parties always seem to be of zero help in these sorts of cases.

Cheers

--

Tristram Cheer Network Architect

Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz

PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tim Price Sent: Tuesday, 28 February 2012 9:59 a.m. To: 'Barry Murphy'; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest

I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest

Hey Guys,

I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus.

I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF.

Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth.

I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all.

Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported.

thanks Barry

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Tristram Cheer

9:55 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

Hi Mark, The CallPlus node is open to anyone at APE and we were sitting on it happily until our upstream provider Kordia changed from PacNet to Vocus. Cheers -- Tristram Cheer Network Architect - Most problems are the result of previous solutions... Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 | 53 Port Road, Whangarei tristram.cheer(a)ubergroup.co.nz mailto:tristram.cheer(a)ubergroup.co.nz |www.ubergroup.co.nz http://www.ubergroup.co.nz http://ubergroup.co.nz/fb https://twitter.com/#!/ubergroupltd From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Mark Foster Sent: Tuesday, 28 February 2012 10:49 a.m. To: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest The problem with doing this is that Callplus may not be intending their Akamai node to be used by folks that aren't either their customer, or an organisation they have a commercial arrangement with to serve this content. In my last job we had a similar issue, we were going to wind up requiring our upstream to do some source-based-routing, so we didn't pursue that option as it wasn't considered practical. Instead perhaps ISP's need to consider setting up their own Akamai presence or formally arranging something with an ISP that does have it. (And inversely the ISPs that want to enforce this position would need to apply controls on their DNS server, which is itself annoying for other reasons...) Mark. On 28/02/12 10:09, Tristram Cheer wrote: Hey All, We had the same issue last year, Akamai decided to point us at Vocus's node and Akamai traffic coming from AKL server's under Vocus's control were returning our data over our International. We of course were very unhappy about this but we got nowhere with Vocus or Akamai about it (Basically told to sod off), The only way we found to fix it was to point the DNS zones for akamaitech.net and akamai.net to another DNS server in NZ that stayed pointed to CallPlus's Akamai node. Very frustrating given and difficult to resolve, The offending parties always seem to be of zero help in these sorts of cases. Cheers -- Tristram Cheer Network Architect Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [ mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tim Price Sent: Tuesday, 28 February 2012 9:59 a.m. To: 'Barry Murphy'; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive. -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest Hey Guys, I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus. I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF. Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth. I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all. Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported. thanks Barry _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Mark Foster

10:21 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

On 28/02/12 10:55, Tristram Cheer wrote:

...

Hi Mark,

The CallPlus node is open to anyone at APE and we were sitting on it happily until our upstream provider Kordia changed from PacNet to Vocus.

By accident or by design? (If by design, then yes, that particular point would be moot, but some ISPs may well see that the Opex of operating an Akamai node is only in effect recovered by paying customers, and not by those taking advantage of free peering. There is a cost to being on the APE, and if (for example) a GigabitEthernet link to APE was put under pressure by a large number of APE folks hitting your CDN node, will your beancounters approve of paying for more APE connectivity to service people who aren't even your customers? No? How surprising?) Cheers Mark.

Barry Murphy

10:35 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

I made the suggestion back in July 2010 that perhaps we need a Skytower hosted Akamai, Jay from NZRS suggested he was actually looking into this, then Mike Jager pointed us to this… http://list.waikato.ac.nz/pipermail/nznog/2008-July/014249.html where callplus actively advertised the free use of the Akamai cluster they have. Cheers Barry On Tue, 28 Feb 2012 11:21:42 +1300, Mark Foster wrote:

...

On 28/02/12 10:55, Tristram Cheer wrote:

...
Hi Mark,

The CallPlus node is open to anyone at APE and we were sitting on it happily until our upstream provider Kordia changed from PacNet to Vocus.

By accident or by design?

(If by design, then yes, that particular point would be moot, but some ISPs may well see that the Opex of operating an Akamai node is only in effect recovered by paying customers, and not by those taking advantage of free peering. There is a cost to being on the APE, and if (for example) a GigabitEthernet link to APE was put under pressure by a large number of APE folks hitting your CDN node, will your beancounters approve of paying for more APE connectivity to service people who aren't even your customers? No? How surprising?)

Cheers Mark.

Nathan Ward

10:50 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

Who would pay for the transit to feed it? On 28/02/2012, at 11:35 AM, Barry Murphy wrote:

...

I made the suggestion back in July 2010 that perhaps we need a Skytower hosted Akamai, Jay from NZRS suggested he was actually looking into this, then Mike Jager pointed us to this… http://list.waikato.ac.nz/pipermail/nznog/2008-July/014249.html where callplus actively advertised the free use of the Akamai cluster they have.

Cheers Barry

Steve Kurzeja

11:33 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

On Feb 28, 2012 11:21 AM, "Mark Foster" wrote:

...

On 28/02/12 10:55, Tristram Cheer wrote:

...
Hi Mark,

The CallPlus node is open to anyone at APE and we were sitting on it

happily until our upstream provider Kordia changed from PacNet to Vocus.

...

...
By accident or by design? *>*

By design.

...

(If by design, then yes, that particular point would be moot, but some ISPs may well see that the Opex of operating an Akamai node is only in effect recovered by paying customers, and not by those taking advantage of free peering. There is a cost to being on the APE, and if (for example) a GigabitEthernet link to APE was put under pressure by a large number of APE folks hitting your CDN node, will your beancounters approve of paying for more APE connectivity to service people who aren't even your customers? No? How surprising?)

Cheers Mark.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Nathan Ward

11:04 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

In order for a node to be available in the Akamai systems for your prefixes (or rather the prefixes where your DNS servers live), the provider hosting the cluster needs to advertise your prefix over a BGP session to Akamai (it's multi hop over the Internet, and not used for IP routing, just as a good way to dynamically inform Akamai of what prefixes you want them to server through your cluster). If you don't want your prefixes being seen by that cluster, ask whoever hosts the cluster to not advertise them to Akamai. If it's a common request, maybe they have a community you can use, or maybe they can create one? Google GGC is similar, except it's based on client IP addresses not DNS servers, and the eBGP session goes to their local server and not over the internet. The same challenges no doubt impact you though. I'd suggest avoiding ASPATH poisoning, because in some cases your preferred CDNs aren't going to have the content you want, and your customers will go to the CDN's global servers instead of the next best. On 28/02/2012, at 10:09 AM, Tristram Cheer wrote:

...

Hey All,

We had the same issue last year, Akamai decided to point us at Vocus's node and Akamai traffic coming from AKL server's under Vocus's control were returning our data over our International. We of course were very unhappy about this but we got nowhere with Vocus or Akamai about it (Basically told to sod off), The only way we found to fix it was to point the DNS zones for akamaitech.net and akamai.net to another DNS server in NZ that stayed pointed to CallPlus's Akamai node.

Very frustrating given and difficult to resolve, The offending parties always seem to be of zero help in these sorts of cases.

Cheers

--

Tristram Cheer Network Architect

Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz

PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tim Price Sent: Tuesday, 28 February 2012 9:59 a.m. To: 'Barry Murphy'; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest

I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest

Hey Guys,

I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus.

I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF.

Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth.

I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all.

Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported.

thanks Barry

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Tristram Cheer

11:17 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

How Akamai points you to a node might as well be black magic, It's not entirely based on the IP prefix's as by simply changing our DNS servers to point Akamai zones to other DNS servers in NZ we can adjust where our Akamai traffic is coming from. Not announcing your range is only an option where the network hosting the node you want to avoid ISN'T your upstream provider's transit carrier, I'm looking at you Kordia. -- Tristram Cheer Network Architect Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: Nathan Ward [mailto:nznog(a)daork.net] Sent: Tuesday, 28 February 2012 12:05 p.m. To: Tristram Cheer Cc: Tim Price; Barry Murphy; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest In order for a node to be available in the Akamai systems for your prefixes (or rather the prefixes where your DNS servers live), the provider hosting the cluster needs to advertise your prefix over a BGP session to Akamai (it's multi hop over the Internet, and not used for IP routing, just as a good way to dynamically inform Akamai of what prefixes you want them to server through your cluster). If you don't want your prefixes being seen by that cluster, ask whoever hosts the cluster to not advertise them to Akamai. If it's a common request, maybe they have a community you can use, or maybe they can create one? Google GGC is similar, except it's based on client IP addresses not DNS servers, and the eBGP session goes to their local server and not over the internet. The same challenges no doubt impact you though. I'd suggest avoiding ASPATH poisoning, because in some cases your preferred CDNs aren't going to have the content you want, and your customers will go to the CDN's global servers instead of the next best. On 28/02/2012, at 10:09 AM, Tristram Cheer wrote:

...

Hey All,

We had the same issue last year, Akamai decided to point us at Vocus's node and Akamai traffic coming from AKL server's under Vocus's control were returning our data over our International. We of course were very unhappy about this but we got nowhere with Vocus or Akamai about it (Basically told to sod off), The only way we found to fix it was to point the DNS zones for akamaitech.net and akamai.net to another DNS server in NZ that stayed pointed to CallPlus's Akamai node.

Very frustrating given and difficult to resolve, The offending parties always seem to be of zero help in these sorts of cases.

Cheers

--

Tristram Cheer Network Architect

Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz

PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tim Price Sent: Tuesday, 28 February 2012 9:59 a.m. To: 'Barry Murphy'; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest

I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest

Hey Guys,

I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus.

I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF.

Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth.

I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all.

Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported.

thanks Barry

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

McDonald Richards

11:36 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

Hello, I seem to remember our operations team doing work to assist you in moving traffic from the Vocus network on to APE Tristram. We by no means told you to "sod off" and remain, as always, happy to assist where we can. We installed an Akamai cache to serve our customer routes in Auckland after we noticed a significant number of hits on our Sydney infrastructure coming from New Zealand. This traffic, combined with load spikes we would see when a local NZ cache ran into problems was enough to motivate us to put a local node in to try and keep the traffic closer to the end users. If I recall correctly, the load and performance of a cache is one of the metrics used in the selection. If the "free" (and I use this term very loosely as this infrastructure does cost money to run) node at CallPlus was under strain, then Akamai, whose own agreements with their content customers probably have some sort of metric for performance, would likely elect to send you to another available cache to ensure their content is delivered in line with their commercial obligations. All of our transit routes are advertised to Akamai as eligible for access to caches on our network. This is a standard route policy that exists to cover the costs of running the node (by generating some utilisation) and to ensure that if there are any downstream node failures that we are able to absorb the impact within the country (with a slight uplift in traffic required to fill the cache instead of the traffic being served from further abroad). Macca Vocus On 28/02/12 8:09 AM, "Tristram Cheer" wrote:

...

Hey All,

We had the same issue last year, Akamai decided to point us at Vocus's node and Akamai traffic coming from AKL server's under Vocus's control were returning our data over our International. We of course were very unhappy about this but we got nowhere with Vocus or Akamai about it (Basically told to sod off), The only way we found to fix it was to point the DNS zones for akamaitech.net and akamai.net to another DNS server in NZ that stayed pointed to CallPlus's Akamai node.

Very frustrating given and difficult to resolve, The offending parties always seem to be of zero help in these sorts of cases.

Cheers

--

Tristram Cheer Network Architect

Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz

PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tim Price Sent: Tuesday, 28 February 2012 9:59 a.m. To: 'Barry Murphy'; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest

I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest

Hey Guys,

I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus.

I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF.

Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth.

I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all.

Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported.

thanks Barry

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Barry Murphy

11:59 p.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

Hey Macca, Because we use vocus for transit, we seem to be getting priority to vocus akamai. Are you able to tell us what the IP address of the local nodes so we can possibly add fake latency to these nodes to try and swing the traffic to the APE and or our local transit provider (cheaper). The problem we are having is our local transit provider has their Akamai infrastructure in CHCH, the latency to this cluster would be a lot higher than that of the vocus locally hosted option. Alternatively can we possibly have an additional community to tag that we can stop the advertising of our ranges to Auckland cluster & Sydney Cluster, this way we can manage the traffic in a much better arrangement. Perhaps you can add this communities to the existing list at http://tools.vocus.com.au/additionals/communities2.0.html for all providers to use so they can swing traffic as they wish. Many thanks Barry On Tue, 28 Feb 2012 10:36:24 +1100, McDonald Richards wrote:

...

Hello,

I seem to remember our operations team doing work to assist you in moving traffic from the Vocus network on to APE Tristram. We by no means told you to "sod off" and remain, as always, happy to assist where we can.

We installed an Akamai cache to serve our customer routes in Auckland after we noticed a significant number of hits on our Sydney infrastructure coming from New Zealand. This traffic, combined with load spikes we would see when a local NZ cache ran into problems was enough to motivate us to put a local node in to try and keep the traffic closer to the end users.

If I recall correctly, the load and performance of a cache is one of the metrics used in the selection. If the "free" (and I use this term very loosely as this infrastructure does cost money to run) node at CallPlus was under strain, then Akamai, whose own agreements with their content customers probably have some sort of metric for performance, would likely elect to send you to another available cache to ensure their content is delivered in line with their commercial obligations.

All of our transit routes are advertised to Akamai as eligible for access to caches on our network. This is a standard route policy that exists to cover the costs of running the node (by generating some utilisation) and to ensure that if there are any downstream node failures that we are able to absorb the impact within the country (with a slight uplift in traffic required to fill the cache instead of the traffic being served from further abroad).

Macca Vocus

On 28/02/12 8:09 AM, "Tristram Cheer" wrote:

...
Hey All,

We had the same issue last year, Akamai decided to point us at Vocus's node and Akamai traffic coming from AKL server's under Vocus's control were returning our data over our International. We of course were very unhappy about this but we got nowhere with Vocus or Akamai about it (Basically told to sod off), The only way we found to fix it was to point the DNS zones for akamaitech.net and akamai.net to another DNS server in NZ that stayed pointed to CallPlus's Akamai node.

Very frustrating given and difficult to resolve, The offending parties always seem to be of zero help in these sorts of cases.

Cheers

--

Tristram Cheer Network Architect

Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz

PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tim Price Sent: Tuesday, 28 February 2012 9:59 a.m. To: 'Barry Murphy'; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest

I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest

Hey Guys,

I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus.

I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF.

Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth.

I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all.

Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported.

thanks Barry

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Barry Murphy

13 Mar 13 Mar

4:30 a.m.

New subject: Async Routing across ape to pipe return via vocus &Internode speedtest

Heya macca. With risk of this becoming a long thread again... Just wondering if you had given any thought to a community where we can deny routes to the akamai cluster? Many thanks Barry Sent from my iPhone On 28/02/2012, at 12:36 PM, McDonald Richards wrote:

...

Hello,

I seem to remember our operations team doing work to assist you in moving traffic from the Vocus network on to APE Tristram. We by no means told you to "sod off" and remain, as always, happy to assist where we can.

We installed an Akamai cache to serve our customer routes in Auckland after we noticed a significant number of hits on our Sydney infrastructure coming from New Zealand. This traffic, combined with load spikes we would see when a local NZ cache ran into problems was enough to motivate us to put a local node in to try and keep the traffic closer to the end users.

If I recall correctly, the load and performance of a cache is one of the metrics used in the selection. If the "free" (and I use this term very loosely as this infrastructure does cost money to run) node at CallPlus was under strain, then Akamai, whose own agreements with their content customers probably have some sort of metric for performance, would likely elect to send you to another available cache to ensure their content is delivered in line with their commercial obligations.

All of our transit routes are advertised to Akamai as eligible for access to caches on our network. This is a standard route policy that exists to cover the costs of running the node (by generating some utilisation) and to ensure that if there are any downstream node failures that we are able to absorb the impact within the country (with a slight uplift in traffic required to fill the cache instead of the traffic being served from further abroad).

Macca Vocus

On 28/02/12 8:09 AM, "Tristram Cheer" wrote:

...
Hey All,

We had the same issue last year, Akamai decided to point us at Vocus's node and Akamai traffic coming from AKL server's under Vocus's control were returning our data over our International. We of course were very unhappy about this but we got nowhere with Vocus or Akamai about it (Basically told to sod off), The only way we found to fix it was to point the DNS zones for akamaitech.net and akamai.net to another DNS server in NZ that stayed pointed to CallPlus's Akamai node.

Very frustrating given and difficult to resolve, The offending parties always seem to be of zero help in these sorts of cases.

Cheers

--

Tristram Cheer Network Architect

Tel. 09 438 5472 Ext 803 | Mobile. 022 412 1985 Fax. | tristram.cheer(a)ubergroup.co.nz | www.ubergroup.co.nz

PS: Follow us on facebook: www.ubergroup.co.nz/fb or twitter https://twitter.com/#!/ubergroupltd -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tim Price Sent: Tuesday, 28 February 2012 9:59 a.m. To: 'Barry Murphy'; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Async Routing across ape to pipe return via vocus &Internode speedtest

I've seen something similar to this whereby traffic to Vocus egresses APE and ingresses International via Kordia. I was told by various parties that it is because Vocus sets a higher local preference on it's 'customer' routes (Kordia being a customer of Vocus) than on the peering exchanges. The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE which would seem counter productive.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Barry Murphy Sent: Tuesday, 28 February 2012 9:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Async Routing across ape to pipe return via vocus & Internode speedtest

Hey Guys,

I just saw Glen Eustace's post which reminded me of an issue we have seen of late with regards to 'my internet seems slow'. Just wondering if anyone else has seen this similar issue since PIPE joined the APE, then again it's not just PIPE, but a mixture of Oz providers accepting traffic over the APE and then returning the transit via international links such as Vocus.

I have a few customers that buy dedicated international and dedicated domestic bandwidth. These customers have a stub address that resides in the Domestic VRF & another stub in the International VRF. Since PIPE came on-board we have noticed that customers trying to do a whois (whois.apnic.net) using their router that connects the stubs, that the traffic goes out thinking its Domestic, but then gets no response because it attempts to return in via the International (Transit) VRF to which has no access to stubs in the Domestic VRF.

Another issue we see where 'slow' comes into play is internode have recently been added to speedtest.net as a NZ speedtest server in Auckland. While tracing to this node it does appear to go across APE, the return path is via Vocus. When a customer attempts a speed test and they see they are only able to obtain say 30mbps rather than the 100mbps they would expect from a "NZ" server they start thinking something is broken, when actually it's because the server is International and they are in a pool of bandwidth.

I've tried to contact internode & so has a colleague of mine but with no luck. We've tried to pre-pend AS's to the vocus communities but again this hasnt helped at all.

Just wondering if anyone else has seen issues arise since ASYNC routing between Oz & NZ, to do with either STUBs getting no internet due to VRF nature, or these 'slow' speeds being reported.

thanks Barry

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Martin D Kealey

2 Mar 2 Mar

6:44 p.m.

New subject: Asymmetric routing across APE to PIPE & Internode speedtest, return via Vocus

On Tue, 28 Feb 2012, Tim Price wrote:

...

The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE

<nit> I know what asymmetric routes are, but "asynchronous" routes are a new one for me. </nit> -Martin

Tim Price

3 Mar 3 Mar

12:16 a.m.

New subject: Asymmetric routing across APE to PIPE & Internode speedtest, return via Vocus

Point taken. http://xkcd.com/386/ -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Martin D Kealey Sent: Saturday, 3 March 2012 7:44 a.m. To: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Asymmetric routing across APE to PIPE & Internode speedtest, return via Vocus On Tue, 28 Feb 2012, Tim Price wrote:

...

The only way that we could force the routes to be synchronous again was to drop all Vocus routes received across APE

<nit> I know what asymmetric routes are, but "asynchronous" routes are a new one for me. </nit> -Martin _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

4770

Age (days ago)

4784

Last active (days ago)

List overview

Download

21 comments

10 participants

participants (10)

Barry Murphy
Craig Whitmore
Craig Whitmore
Mark Foster
Martin D Kealey
McDonald Richards
Nathan Ward
Steve Kurzeja
Tim Price
Tristram Cheer