On Jul 13, 2010, at 9:13 PM, Craig Spiers wrote:

I think it's pretty common for service providers to be using DNS trickery to get to the closer, faster, less cost caches..

This trickery could be called far worse names. I'll explain more below.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Lincoln Reid Sent: Wednesday, 14 July 2010 1:08 p.m.

My question is, is there a method I can use to influence which Akamai cabinet serves up content to our networks and downstream networks?

You can ask us. But we have restrictions on what we can do. If you e-mail NetSupport-tix(a)akamai.com, that will open a ticket with our Network Support group. They will do what they can, but as I said, we do not have complete freedom on where to serve what to whom. If you e-mail them, please include the results of: host whoami.akamai.net host www.akamai.com [Note the .com & .net are not interchangeable.]

We have free peering with most networks within New Zealand. We have paid peering circuits ( at something like $150 / mbps ) with two larger NZ telco's. And finally we have multiple paid transit services with other international providers ( at something like $250 / mbps ).

Damned telcos! :)

The situation we have currently is that the default Akamai instance we get given when we make a request is over one of the most expensive circuits for us and it is outside the country.

Akamai choses where to serve people based on latency, packet loss, and throughput. If the latency over your transit provider is slightly lower than the latency over a 'free' path, we will still pick the transit provider - many times. We can influence that decision if you ask us, assuming that 1) we are allowed to serve you over the free path, and 2) the free path has acceptable performance characteristics.

Both of the larger NZ telco's have their own Akamai cabinets, that with ( some slightly distasteful ) dns hackery we can use, and appear to work well with lower latency and at a lower cost. Also one of the NZ ISP's that we have free peering with ( Callplus ) have their own Akamai instance that we can access at zero cost and has the best latency. Ideally, we would like to use that one when ever possible.

I had a bunch of feedback from my original query from network folk that were whacking in forwarders to other peoples DNS servers to use their local caches to pick better ( either $ or ms ) circuits for them. I'm sure that there will be side effects from this at times, but the trade off is worth it for them.

Most Akamai nodes are limited to serve only the network in which they reside and its customers. Think of it as peering on steroids. We give them servers, they give us rackspace, and we agree to only serve their end users. This means we cannot simply map you to the "closest" node if the network hosting that node does not want us to serve you. You can get around this using "DNS trickery". But I caution you against this. Let me explain more in-depth how the Akamai system works to explain why. First, some terminology. Akamai's secret sauce is our Mapping System. When I say "map you to", that means tell you which web server will serve you content. This is done through DNS. You type "www.akamai.com" into your browser, your computer goes to your recursive name server (aka RNS or sometimes caching NS). The RNS asks us for an IP address for that hostname. Notice at this point we have not seen the end user IP address. Moreover, even if we had, the RNS will cache the response and had it to the next end user without asking us again. As a result, we use the RNS IP address as a proxy for the end user IP address. We look up the RNS IP address in our system and respond with an IP address of a web server that is both 'close' and allowed to serve that IP address. The rest of the transaction is standard, every day HTTP GET & the like. No redirects, no anycast, no transparent proxying, nothing. If you used a sniffer on the line, you could not tell the difference between an Akamai web server and the Apache server you probably installed yourself last month. Now, back to the on-net caches. The network with the on-net node gives us an ACL (usually through a BGP session to one of our Quagga boxes) of which users are allowed to use that node. Any request from an IP address outside that ACL will not be directed to the on-net cache. Unfortunately, one of the limitations of Akamai is that we do our mapping through DNS, as I described above. This means if you use a recursive name server inside a network, your request will get mapped to the on-net cache because you look to us like you came from inside the network. So when you configure your RNS to forward to another provider's RNS without their permission, you are really circumventing the agreement we have with that provider. It is the same as sending a packet to Provider 1 which is destined for Provider 2. Even if Provider 1 & Provider 2 peer, it is still really bad Netiquette at best, and many would call it far worse. (Including me.) End result, if you want to be mapped somewhere other than where you are mapped today, please e-mail us and ask. Please do not use other networks' resources by sending DNS requests through a box you do not own or pay for. If we can move you somewhere better and save you money, we will - as long as it does not destroy performance. (Which I doubt any of you want anyway.) But we must abide by the agreement we make with networks when we place caches inside those networks. And, of course, if you know a network which has an Akamai node, you can always ask them to add your AS / prefixes to the ACL. We have no problem serving you from any node that gives us permission to serve you. Finally, Akamai ships free kit to any network with enough traffic. Unfortunately, the levels are significant, I believe it is 75 or 100 Mbps of Akamai traffic. (Hey, those servers ain't cheap!) But the requirement may be lower in .nz because of the higher cost down there. It wouldn't be 5 or 10 Mbps, but you can always ask and we'll let you know. Hopefully this answered many people's questions. If anything was unclear, please let me know. -- TTFN, patrick

On 14 July 2010 15:31, Lincoln Reid wrote:

Ideally someone from Callplus could chirp in here and clarify if they are feeding all the prefixes they learn at APE on to Akamai, or if there is a static list that will become out of date etc.

We set a community on anything we learn from APE, and send everything that has that community to Akamai. If someone can identify a particular route that isn't working as expected, let me know and I'll have a look.

Hi Ian, Patrick, We've had some fairly major problems with the Callplus Akamai cache in the past, especially with a customer of ours that makes heavy use of Apple developer resources -- these tend to have relatively little use, so the customer was often getting un-cached items, and performance of those downloads, frankly, sucked. If we got a cached item, it roared down. But if un-cached, they were experiencing sub-megabit download rates. Downloading beta OS distributions was so bad that they were downloading at home (via a different cache) and bringing them in on a flash stick. Akamai support was, uh, unhelpful. I'm being diplomatic. Now, I haven't had a complaint for a while. I don't know if that's just because the customer has learned to live with it knowing there's not a lot I can do, or if things have genuinely improved. -- don On 07/15/2010 12:21 PM, Ian Batterbee wrote:

Futher to my message below, uh, it seems there were some uh.. problems with the config after some BGP changes some time ago. I believe this has now been fixed, and we're now advertising approximately 2000 prefixes learned through APE towards Akamai's quagga box.

I guess this should put an end to the discussion about setting up an APE-connected Akamai cluster.

On 15 July 2010 10:23, Ian Batterbee mailto:ibatterb(a)gmail.com> wrote:

On 14 July 2010 15:31, Lincoln Reid mailto:lincoln(a)acsdata.co.nz> wrote:

Ideally someone from Callplus could chirp in here and clarify if they are feeding all the prefixes they learn at APE on to Akamai, or if there is a static list that will become out of date etc.

We set a community on anything we learn from APE, and send everything that has that community to Akamai. If someone can identify a particular route that isn't working as expected, let me know and I'll have a look.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

On Jul 14, 2010, at 9:20 PM, Craig Spiers wrote:

I've seen this also.. even on cached items I see download rates of at most 500Kbyte/sec - is there a limit on throughput for even cached items? Via another cache on APE - I can hit speeds in the Mbytes/sec

The servers have no rate limiting - other than standard TCP. (Actually, non-standard, as we tweak even the TCP parameters. But there are limits, since the end user's machine has its own limitations.)

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Don Stokes Sent: Thursday, 15 July 2010 1:18 p.m.

We've had some fairly major problems with the Callplus Akamai cache in the past, especially with a customer of ours that makes heavy use of Apple developer resources -- these tend to have relatively little use, so the customer was often getting un-cached items, and performance of those downloads, frankly, sucked.

If we got a cached item, it roared down. But if un-cached, they were experiencing sub-megabit download rates. Downloading beta OS distributions was so bad that they were downloading at home (via a different cache) and bringing them in on a flash stick.

That (obviously) should not happen. However, we are limited to the upstream b/w provided to us. As stated above, we tweak TCP such that downloads to the cache should be quick, but (again, obviously) it cannot be instantaneous. If you have more info, or if it reoccurs, we can look into it.

Akamai support was, uh, unhelpful. I'm being diplomatic.

I am sorry to hear that. Do you have a ticket #? (Sorry to sound like an auto-response-bot, but it really will help me to chase things down if I have a ticket #.) -- TTFN, patrick

Thanks Patrick, This is what happens when you offer to explain how things work ! :) On 15 July 2010 15:31, Patrick W. Gilmore wrote:

[Geez, I really started a storm, didn't I? :-]

On 14/07/2010, at 3:34 PM, Barry Murphy (iphone) wrote:

What about the idea of citylink or someone in skytower hosting an instance for all peering at ape ( Auckland peering exchange)?

We're thinking of doing that and have been talking to a few people to see if there would be any interest. If anyone else is then please let us know. cheers Jay

I'm sure that would generate a lot of traffic from all the small ISPs connected there without the need to have their own smaller instances

Thanks Barry Murphy

On 14/07/2010, at 1:40 PM, "Patrick W. Gilmore" wrote:

...

On Jul 13, 2010, at 9:13 PM, Craig Spiers wrote:

...

I think it's pretty common for service providers to be using DNS trickery to get to the closer, faster, less cost caches..

This trickery could be called far worse names. I'll explain more below.

...

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Lincoln Reid Sent: Wednesday, 14 July 2010 1:08 p.m.

...

My question is, is there a method I can use to influence which Akamai cabinet serves up content to our networks and downstream networks?

You can ask us. But we have restrictions on what we can do.

If you e-mail NetSupport-tix(a)akamai.com, that will open a ticket with our Network Support group. They will do what they can, but as I said, we do not have complete freedom on where to serve what to whom.

If you e-mail them, please include the results of:

host whoami.akamai.net host www.akamai.com

[Note the .com & .net are not interchangeable.]

...

We have free peering with most networks within New Zealand. We have paid peering circuits ( at something like $150 / mbps ) with two larger NZ telco's. And finally we have multiple paid transit services with other international providers ( at something like $250 / mbps ).

Damned telcos! :)

...

The situation we have currently is that the default Akamai instance we get given when we make a request is over one of the most expensive circuits for us and it is outside the country.

Akamai choses where to serve people based on latency, packet loss, and throughput. If the latency over your transit provider is slightly lower than the latency over a 'free' path, we will still pick the transit provider - many times. We can influence that decision if you ask us, assuming that 1) we are allowed to serve you over the free path, and 2) the free path has acceptable performance characteristics.

...

Both of the larger NZ telco's have their own Akamai cabinets, that with ( some slightly distasteful ) dns hackery we can use, and appear to work well with lower latency and at a lower cost. Also one of the NZ ISP's that we have free peering with ( Callplus ) have their own Akamai instance that we can access at zero cost and has the best latency. Ideally, we would like to use that one when ever possible.

I had a bunch of feedback from my original query from network folk that were whacking in forwarders to other peoples DNS servers to use their local caches to pick better ( either $ or ms ) circuits for them. I'm sure that there will be side effects from this at times, but the trade off is worth it for them.

Most Akamai nodes are limited to serve only the network in which they reside and its customers. Think of it as peering on steroids. We give them servers, they give us rackspace, and we agree to only serve their end users. This means we cannot simply map you to the "closest" node if the network hosting that node does not want us to serve you.

You can get around this using "DNS trickery". But I caution you against this. Let me explain more in-depth how the Akamai system works to explain why.

First, some terminology. Akamai's secret sauce is our Mapping System. When I say "map you to", that means tell you which web server will serve you content. This is done through DNS. You type "www.akamai.com" into your browser, your computer goes to your recursive name server (aka RNS or sometimes caching NS). The RNS asks us for an IP address for that hostname.

Notice at this point we have not seen the end user IP address. Moreover, even if we had, the RNS will cache the response and had it to the next end user without asking us again. As a result, we use the RNS IP address as a proxy for the end user IP address.

We look up the RNS IP address in our system and respond with an IP address of a web server that is both 'close' and allowed to serve that IP address. The rest of the transaction is standard, every day HTTP GET & the like. No redirects, no anycast, no transparent proxying, nothing. If you used a sniffer on the line, you could not tell the difference between an Akamai web server and the Apache server you probably installed yourself last month.

Now, back to the on-net caches. The network with the on-net node gives us an ACL (usually through a BGP session to one of our Quagga boxes) of which users are allowed to use that node. Any request from an IP address outside that ACL will not be directed to the on-net cache.

Unfortunately, one of the limitations of Akamai is that we do our mapping through DNS, as I described above. This means if you use a recursive name server inside a network, your request will get mapped to the on-net cache because you look to us like you came from inside the network.

So when you configure your RNS to forward to another provider's RNS without their permission, you are really circumventing the agreement we have with that provider. It is the same as sending a packet to Provider 1 which is destined for Provider 2. Even if Provider 1 & Provider 2 peer, it is still really bad Netiquette at best, and many would call it far worse. (Including me.)

End result, if you want to be mapped somewhere other than where you are mapped today, please e-mail us and ask. Please do not use other networks' resources by sending DNS requests through a box you do not own or pay for.

If we can move you somewhere better and save you money, we will - as long as it does not destroy performance. (Which I doubt any of you want anyway.) But we must abide by the agreement we make with networks when we place caches inside those networks.

And, of course, if you know a network which has an Akamai node, you can always ask them to add your AS / prefixes to the ACL. We have no problem serving you from any node that gives us permission to serve you.

Finally, Akamai ships free kit to any network with enough traffic. Unfortunately, the levels are significant, I believe it is 75 or 100 Mbps of Akamai traffic. (Hey, those servers ain't cheap!) But the requirement may be lower in .nz because of the higher cost down there. It wouldn't be 5 or 10 Mbps, but you can always ask and we'll let you know.

Hopefully this answered many people's questions. If anything was unclear, please let me know.

-- TTFN, patrick

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

---

NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Jay Daley Chief Executive .nz Registry Services (New Zealand Domain Name Registry Limited) desk: +64 4 931 6977 mobile: +64 21 678840

On 14/07/2010, at 3:44 PM, Mike Jager wrote:

On 14/07/10 15:34, Barry Murphy (iphone) wrote:

...

What about the idea of citylink or someone in skytower hosting an instance for all peering at ape ( Auckland peering exchange)?

http://list.waikato.ac.nz/pipermail/nznog/2008-July/014249.html

Sorry, were we talking about Akamai instances or shared DNS to simplify geo-location? cheers Jay

-Mike _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Jay Daley Chief Executive .nz Registry Services (New Zealand Domain Name Registry Limited) desk: +64 4 931 6977 mobile: +64 21 678840