On Wed, 30 Jun 2004, Matthew Poole wrote:

A number of people are reporting difficulties accessing the Debian mirror at ftp.citylink.

FTPing in from IHUG or Max (and apparently from Xtra, and I'd guess anyone else who transits via GGI) gets the following message: *SNIP*

On further contemplation, I realised that this almost certainly isn't anything to do with IHUG - Though I don't know what MaxNet's excuse is. Since I'm using Jetstream Surf, my bandwidth is provided by Telecom, subjecting me to their routing games. Which is doubly frustrating, since it means that nobody who wants anything faster than a 128k connection in the vast majority of the country has any chance to use peered traffic - Cable=Paradise=TelstraClear, and Jetstream fullrate or Jetstream Surf=Telecom. And those of us living in locations where there is a choice won't even be saved by the arrival of Bitstream, since Mr Webb's wisdom doesn't require it to be provided in locations with existing competition. *SIGH*!

On Wed, 30 Jun 2004, Matthew Poole wrote:

Now, we all expect Xtra/Telecom to play nasty games with domestic traffic, but can anyone shed light on why "good" ISPs such as the Hug or Max aren't doing the domestic peering thing? Are we starting to see preemptive peering termination by ISPs who're preparing for the day when the two Ts stop playing the game?

One problem is that citylink's ftp servers are off WIX so providers that peer on APE might not peer fully on WIX since it's costs a bit of money to haul the traffic down there. Ihug should be working however but it looks like a few ranges arn't correctly advertised so they might be causing the problem. Mathew can you please email me with the ihug IP addresses you were on when you got the problem and we should be able to fix the filters.

On the other side, are we now seeing a trend towards content providers playing a political game to try and reverse the swing away from peering? Obviously Citylink are far more involved at the peering level than nearly any other content provider, but blocking access to content mirrors under the guise of protecting the consumer(when in reality it's protecting the provider) is not a good way to make friends.

I feel it's a good way to do this. Just this morning I had a call from someone at a University asking for access to ihug's ftp server since they could no longer get to citylink's . Since the company is a Telecom customer they are paying Telecom for Internet and Telecom are turning around and expecting other companies such as Citylink or ihug to send pay to send them traffic accross links that Citylink or Ihug have to pay Telecom for. Most organisations in Auckland, Wellington or Hamilton can get connected to WIX or APE for around $500 per month. By refusing to do this and getting connected via a non-peering provider they are costing the rest of the NZ Internet money. I did a list of Networks and Organisations a few weeks ago that ( I estimated) were spending more than $5000 per month on Internet but were not peering. The list was incomplete and had several error but if somebody was interested in maintaining and fixing then I'm sure several people could use it for things like: - Blocking access to ftp or web sites - Routing traffic for those ranges via International rather than paid National links. - Traffic shaping across paid links to reduce costs. - Sending bills to to recover costs paid to T&T for National transit. -- Simon J. Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.

Various people are asking me for comment, so here goes On Wed, Jun 30, 2004 at 09:38:16AM +1200, Matthew Poole said:

On the other side, are we now seeing a trend towards content providers playing a political game to try and reverse the swing away from peering? Obviously Citylink are far more involved at the peering level than nearly any other content provider, but blocking access to content mirrors under the guise of protecting the consumer(when in reality it's protecting the provider) is not a good way to make friends.

I can't really speak for other content providers, but for us, Xtra depeering a bunch of weeks back meant that our transit pipe to Auckland filled up, to the detriment of other services that are supposed to be using that bandwidth. The ftp server, webcams, streaming servers, and other knicknacks that we run could hardly be described as big earners - when we delivered most of the traffic on WIX, then we could afford to let the remainder leak up our transit link, but when our domestic transit link gets swamped delivering free stuff to other ISP's punters, then I can't easily justify upgrading it. <soapbox> In a very real sense, all I'm doing is offering services for free, to customers of ISP's in NZ. Presumably, those customers are paying those ISP's for access to the net, and by extension, my server. Those users must be paying the full cost of their service, in order to allow them to reach Microsoft, or Google, or whomever - it should certainly cover the costs of reaching a well connected server in NZ. So I don't feel it's unreasonable to ask ISP's with customers who want some stuff I have to peer to get it, especially if they're already up and running on the same exchange. It doesn't seem reasonable (to me, at least) that I should have to pay these ISP's to serve content to their users, when I wouldn't have to pay them if my server was housed in California, or Reykjavik, or Buenos Aires. </soapbox> When our link started getting walloped, I had various options. I could turn off the mirror, but that didn't seem super constructive, especially since Citylink itself still needs access to Debian - I'd keep the costs of running a mirror, and lose all the goodwill. So I moved it to the same numbers that we used last year for the LoTR, numbers for which there is no transit contract, and which are advertised only into the WIX route servers. The upshot of this was pretty predictable, and has been discussed ad nauseum, so I won't go into it here. We figured this would upset some users, and it's not a decision we took lightly. In order to inform users who miss out about what was going on, we've setup a sorry server in the US that apologises, and gives some indications as to what the users can do about the situation: http://scorchio.pure-guava.org.nz/cgi-bin/wiki/kwiki.cgi?PeeringNotice these are currently on my personal wiki, at some point they'll get marked up onto the citylink site proper. The sorry server isn't active right now (other than for FTP), but it will be tomorrow. ISP's who can't reach the various boxes on WIX may get an increase in helpdesk requests as a result. So, what's to be done about this less than ideal situation. Obviously, Telecom could restore the peering they took away. That'd be a positive step, and an excellent solution, but I'm not holding my breath. Putting a Debian mirror at APE/Xtra will doubtless ease the grizzles from users about lack of access to Debian. However, there are other content providers on WIX looking to bolt more services onto the WIX only numbers, likewise there are service providers in Auckland setting up on a similar range on APE. So fixing one service by creating more mirrors doesn't really cure the core issue - you'll just be chasing the various content providers on APE and WIX, and some of their content isn't going to be easily mirrored. If you're a service provider using an upstream who isn't peering, then you have a choice. Moan at them until they peer (I suspect this will work for Orcon and IHUG, they seem to getting organised to peer in Wellington). If moaning doesn't work, then I suggest you should be getting yourself attached to APE and WIX and taking charge of your own domestic arrangements sooner rather than later - your upstream clearly doesn't really have your best interests at heart. If you're unclear on how you might achieve this, then get in contact with Citylink and we'll help you along. If you're an Auckland based ISP looking to peer on WIX, you should *definitely* get in touch. If you're a content provider who'd like a bit of a transit free /24 (a /29, for eg), to use for whatever ever you choose - mirrors, game servers, streamers, whatever, get in contact - the more the merrier. The obvious caveats about NZ only apply, but the price is right. I've /24's for APE and WIX ready to cut up and hand out to anybody prepared to put a service up. Special treat for those peering, check this out: http://wixcam.citylink.co.nz/zephyrometer/ Now the folks moaning about the wind in Wellington will have documentary proof :-). g'night all Si