Route servers on Citylink (WIX)
Hi Folks For the last couple of months I've had a linux box running the MRT route daemon as an experimental BGP route server, with links between a few organisations on Citylink who were happy to participate in a trial. The main aim of the experiment was to consider ways that Citylink connected organisations with BGP capable routers but without AS numbers might exchange routing data, so that they can send data directly between each other, without having to go through one (or several) ISP routers. As a rudimentary route server, MRT seems to work really well (it lacks some of the RIPE tools of the Merit RA, but on the other hand, it compiles under Linux :-). We ended up setting up MRT to peer with the organisations who had AS numbers, and then setup all the other routers in AS number 65502 (chosen at random) as static routes within MRT, and redistributed them into BGP. Turn off synchronisation on the routers in AS65502, and everything seemed to work as advertised (this is speaking from memory, I haven't looked at it for about 6 weeks). Citylink is now at the stage where we've got customers pestering us to incorporate them into the system, and a management itchy keen to take some money off of these people. So I guess what I'm after from the list is some opinions on whether this is a good idea, and I'm on the right track. Obviously, for this to be more than a toy I need the ISP's on Citylink to participate in such an exchange, so recommendations on making it easier for them to be involved are welcome. Cheers Si --------- To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
Hi Simon, On Thu, Oct 08, 1998 at 12:14:43AM +1300, Simon Blake wrote:
For the last couple of months I've had a linux box running the MRT route daemon as an experimental BGP route server, with links between a few organisations on Citylink who were happy to participate in a trial.
The main aim of the experiment was to consider ways that Citylink connected organisations with BGP capable routers but without AS numbers might exchange routing data, so that they can send data directly between each other, without having to go through one (or several) ISP routers.
Is it reasonable to assume that all networks reachable in this manner (a) will be numbered on globally-unique addresses, and (b) will have a WIX next hop within 202.7.0.0/23?
As a rudimentary route server, MRT seems to work really well (it lacks some of the RIPE tools of the Merit RA, but on the other hand, it compiles under Linux :-). We ended up setting up MRT to peer with the organisations who had AS numbers, and then setup all the other routers in AS number 65502 (chosen at random) as static routes within MRT, and redistributed them into BGP. Turn off synchronisation on the routers in AS65502, and everything seemed to work as advertised (this is speaking from memory, I haven't looked at it for about 6 weeks).
This is a cool idea, but I have a minor reservation based on the setup you described. I have a problem with the use of a "private use" ASN (rfc1930) for essentially the same reason that I object to the use of rfc1918 addresses on WIX - this is shared infrastructure, and we should use globally-unique numbering (for IP addresses and for ASNs). To not do so may/will cause problems - we should not have to worry about _our_ use of any private-use ASNs or IP addresses within _our_ network conflicting with anybody else's. The way to fix this is for the Citylink route servers to operate under a globally-unique ASN, and to essentially hide all the private-use ASNs to other networks when advertising the routes on to ISPs. Ciscos will do this automagically for private-use ASNs (64512 to 65535), and I assume mrt can do something similar? Since all the BGP speakers (with global or private-use ASNs) will learn routes with next hop addresses on the 202.7.0.0/23 network, they will route traffic to them directly (since this is a connected route). This scheme very closely follows that suggested in rfc2270, with Citylink taking the role of the "local provider", talking BGP to their local customers and providing transit to other "global providers". I'm pretty sure this is also how most other big exchanges function with route servers. I'm not sure how your redistributed static advertisements fit into the picture - surely only traffic in one direction would escape the ISP's router in this case? Or was that just part of the mrt testing?
Citylink is now at the stage where we've got customers pestering us to incorporate them into the system, and a management itchy keen to take some money off of these people. So I guess what I'm after from the list is some opinions on whether this is a good idea, and I'm on the right track. Obviously, for this to be more than a toy I need the ISP's on Citylink to participate in such an exchange, so recommendations on making it easier for them to be involved are welcome.
I think this is definitely a good idea, and will help everybody out
tremendously, as long as (a) the route servers operate under a globally-
unique ASN [as above], (b) everybody commits to filtering the
advertisements they receive, and (c) those filters are built from
consistent, published policy (e.g. in the IRR).
I might also add (d) as long as there are at least two route servers,
preferably located in different buildings in Wellington :)
Joe
--
Joe Abley
Hey Joe On Thu, 8 Oct 1998, Joe Abley wrote:
Hi Simon,
On Thu, Oct 08, 1998 at 12:14:43AM +1300, Simon Blake wrote:
The main aim of the experiment was to consider ways that Citylink connected organisations with BGP capable routers but without AS numbers might exchange routing data, so that they can send data directly between each other, without having to go through one (or several) ISP routers.
Is it reasonable to assume that all networks reachable in this manner (a) will be numbered on globally-unique addresses, and (b) will have a WIX next hop within 202.7.0.0/23?
Yes, on both counts.
As a rudimentary route server, MRT seems to work really well (it lacks some of the RIPE tools of the Merit RA, but on the other hand, it compiles under Linux :-). We ended up setting up MRT to peer with the organisations who had AS numbers, and then setup all the other routers in AS number 65502 (chosen at random) as static routes within MRT, and redistributed them into BGP. Turn off synchronisation on the routers in AS65502, and everything seemed to work as advertised (this is speaking from memory, I haven't looked at it for about 6 weeks).
This is a cool idea, but I have a minor reservation based on the setup you described.
I have a problem with the use of a "private use" ASN (rfc1930) for essentially the same reason that I object to the use of rfc1918 addresses on WIX - this is shared infrastructure, and we should use globally-unique numbering (for IP addresses and for ASNs). To not do so may/will cause problems - we should not have to worry about _our_ use of any private-use ASNs or IP addresses within _our_ network conflicting with anybody else's.
I don't have any real problem with using either private or public AS numbers - we used the private number because Citylink doesn't have it's own AS number, not because we're anti the idea of getting a public ASN. I'm not convinced it's such an issue, though. Every organisation is going to want to filter, prioritise and avoid rebroadcasting networks with the CL ASN, regardless of whether it's public or private, otherwise you'll end up carting national traffic for organisations that aren't your clients.
The way to fix this is for the Citylink route servers to operate under a globally-unique ASN, and to essentially hide all the private-use ASNs to other networks when advertising the routes on to ISPs. Ciscos will do this automagically for private-use ASNs (64512 to 65535), and I assume mrt can do something similar?
Possibly, MRT is something of a work in progress, and it's a while since I've read the docs. I would suspect however, that if Citylink got a global ASN, we wouldn't have much need for private-use ASNs anyway, we could just put all the non ASN'd orgnisations into the Citylink ASN.
Since all the BGP speakers (with global or private-use ASNs) will learn routes with next hop addresses on the 202.7.0.0/23 network, they will route traffic to them directly (since this is a connected route).
That's how it works.
This scheme very closely follows that suggested in rfc2270, with Citylink taking the role of the "local provider", talking BGP to their local customers and providing transit to other "global providers".
Does it? I should get out more :-).
I'm pretty sure this is also how most other big exchanges function with route servers.
The only difference being that most exchanges are between organisations with unique ASN's - that's why they're on the exchange. Citylink has this wrinkle that most of the organisations on the network have no AS number, and no hope (or need) of getting one.
I'm not sure how your redistributed static advertisements fit into the picture - surely only traffic in one direction would escape the ISP's router in this case? Or was that just part of the mrt testing?
I (or, I should say, we - much thanks goes to Dean Pemberton for spending his last hours in NZ working with me on this) was examining ways of improving reliability, and reducing reliance on each organisations router being correctly setup. I liked the idea of all the non AS numbered Citylink customers sharing the same ASN, which immediately implied an interior BGP network. The problem I had was that (theoritically) all routers in an IBGP net should be peers, which seemed to me to defeat the purpose of having a route server in first place. So rather then setup each client router to redistribute from their internal routing protocol into the iBGP net on Citylink, we just bunged static routes for those networks straight into the route servers, and redistributed the static routes into BGP on the route servers. This has several advantages - it's stable, possibly clueless customers have to email or ring up to get changes made, rather than accidentally readvertising a pile of routes they shouldn't, and it makes configuration of the client routers simpler. On the down side, it's more admin work for the route server administrator, but these small networks aren't likely to change the networks they want to advertise much, so I don't see that moves/adds/changes are going to be a big issue. Obviously, for the big (in a network sense) organisations that do have their own AS numbers, we do bog standard eBGP peering.
Citylink is now at the stage where we've got customers pestering us to incorporate them into the system, and a management itchy keen to take some money off of these people. So I guess what I'm after from the list is some opinions on whether this is a good idea, and I'm on the right track. Obviously, for this to be more than a toy I need the ISP's on Citylink to participate in such an exchange, so recommendations on making it easier for them to be involved are welcome.
I think this is definitely a good idea, and will help everybody out tremendously, as long as
(a) the route servers operate under a globally- unique ASN [as above],
I'm sure citylink can spring for an AS number.
(b) everybody commits to filtering the advertisements they receive,
That's critical.
and (c) those filters are built from consistent, published policy (e.g. in the IRR).
I don't have a good feeling for how the policy in the IRR works, so I can't comment here.
I might also add (d) as long as there are at least two route servers, preferably located in different buildings in Wellington :)
We have two identical machines, and the intention is to stash them in different buildings, near the main switches. I haven't yet worked out an elegant way of keeping them updated and consistent, but it shouldn't be a problem. Cheers Si --------- To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
Hey Guys; Email from the land with no gas =)
This is a cool idea, but I have a minor reservation based on the setup you described.
I have a problem with the use of a "private use" ASN (rfc1930) for essentially the same reason that I object to the use of rfc1918 addresses on WIX - this is shared infrastructure, and we should use globally-unique numbering (for IP addresses and for ASNs). To not do so may/will cause problems - we should not have to worry about _our_ use of any private-use ASNs or IP addresses within _our_ network conflicting with anybody else's.
I don't have any real problem with using either private or public AS numbers - we used the private number because Citylink doesn't have it's own AS number, not because we're anti the idea of getting a public ASN.
I'm not convinced it's such an issue, though. Every organisation is going to want to filter, prioritise and avoid rebroadcasting networks with the CL ASN, regardless of whether it's public or private, otherwise you'll end up carting national traffic for organisations that aren't your clients.
I'm with Joe on this one. I think that while there is no problem with using rfc1918 or rfc 1930 numbers internal to an AS that globally unique addressing should be used at the interfaces to other AS's. If we all had to keep a track of which rfc1918 or rfc1930 addresses each other were using it would just be a big pain in the butt.
The way to fix this is for the Citylink route servers to operate under a globally-unique ASN, and to essentially hide all the private-use ASNs to other networks when advertising the routes on to ISPs. Ciscos will do this automagically for private-use ASNs (64512 to 65535), and I assume mrt can do something similar?
Possibly, MRT is something of a work in progress, and it's a while since I've read the docs. I would suspect however, that if Citylink got a global ASN, we wouldn't have much need for private-use ASNs anyway, we could just put all the non ASN'd orgnisations into the Citylink ASN.
dean nods Good call Batman
This scheme very closely follows that suggested in rfc2270, with Citylink taking the role of the "local provider", talking BGP to their local customers and providing transit to other "global providers".
Does it? I should get out more :-).
Ack - I get out more to *avoid* reading rfc's =)
I'm pretty sure this is also how most other big exchanges function with route servers.
The only difference being that most exchanges are between organisations with unique ASN's - that's why they're on the exchange. Citylink has this wrinkle that most of the organisations on the network have no AS number, and no hope (or need) of getting one.
No problem. You can bundle them all into one AS and as long as none of them are doing BGP to their upstream provider then all is well and good. Even if they are, it only takes a little more clue to return everything to the aforementioned `well and good' state.
I'm not sure how your redistributed static advertisements fit into the picture - surely only traffic in one direction would escape the ISP's router in this case? Or was that just part of the mrt testing?
I (or, I should say, we - much thanks goes to Dean Pemberton for spending his last hours in NZ working with me on this) was examining ways of improving reliability, and reducing reliance on each organisations router being correctly setup.
Yeah - quite a large part of this project was to design a setup that would require minimun configuration at the client end. Alot of clients are just having boxes dropped at their site and have no idea of how to configure them to run something like BGP. And those are the ones with routers =). It would be preferable for everyone on WIX to get an AS number and run BGP, but for some this is just not an option. Atleast this way, the people who are running BGP (primarily the ISP's and organisations with clue) do not have to maintain static routes to the organisations who are not. Not having to maintain static routes on these routers is a Good Thing (tm). So for an end user who is not running BGP, the other BGP speakers on WIX will route traffic to them directly and not congest the router of the users ISP. And for end users who are running BGP with rfc1930 ASN's everything should happen just fine (because they will be hidden behind the ASN that WIX will buy, hint hint).
This has several advantages - it's stable, possibly clueless customers have to email or ring up to get changes made, rather than accidentally readvertising a pile of routes they shouldn't, and it makes configuration of the client routers simpler. On the down side, it's more admin work for the route server administrator, but these small networks aren't likely to change the networks they want to advertise much, so I don't see that moves/adds/changes are going to be a big issue. Obviously, for the big (in a network sense) organisations that do have their own AS numbers, we do bog standard eBGP peering.
We should look at some sort of route database with email support for users to email in requests in a format that the administrator can find usefull. See point below about policy registries.
I think this is definitely a good idea, and will help everybody out tremendously, as long as
(a) the route servers operate under a globally- unique ASN [as above],
I'm sure citylink can spring for an AS number.
Cool - they should definatly do this (see rant in section 1)
(b) everybody commits to filtering the advertisements they receive,
That's critical.
And should be happening anyway. Filtering should be happening at the ingress and egress points of everyones networks.
and (c) those filters are built from consistent, published policy (e.g. in the IRR).
I don't have a good feeling for how the policy in the IRR works, so I can't comment here.
I can. Being able to generate your route maps and filters from a policy registry makes things a lot easier and less prone to error. The main hurdle is getting people to understand what is required and getting them to input the correct information into the registry. It would be a good idea for the WIX route admin to update a policy registry with all the static route information that they are populating into MRT. That way filter generation by the other parties is a simple process.
I might also add (d) as long as there are at least two route servers, preferably located in different buildings in Wellington :)
Yeah we wouldn't want a POWER CRISIS to take out most of the city and get both boxes would we =). I spose I should add that making them gas powered is also a bad move.
We have two identical machines, and the intention is to stash them in different buildings, near the main switches. I haven't yet worked out an elegant way of keeping them updated and consistent, but it shouldn't be a problem.
I'm sure you'll work something out. Dean --------- To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
As the no of people with connections on citylink grows I would expect something like this would have to be done, certainly we don't want or need to be seeing traffic from our customers that is only going straight back onto citylink. Why send traffic where it doesn't need to go? Does CNHL have the resource (in Simon Blake perhaps) to manage it? What sort of charge would be levied on customers in addition to what they currently pay? Its my opinion that this should be part of the service fee that is paid to maintain a connection currently but then of course I'm not a shareholder in CNHL. If it is charged in addition to, then those people who might choose not to take part (due to it costing) in fact reduce the value of the service to those who do. Regards Shane
-----Original Message----- From: owner-nznog(a)list.waikato.ac.nz [mailto:owner-nznog(a)list.waikato.ac.nz]On Behalf Of Simon Blake Sent: Thursday, 8 October 1998 00:15 To: nznog(a)list.waikato.ac.nz Subject: Route servers on Citylink (WIX)
Hi Folks
For the last couple of months I've had a linux box running the MRT route daemon as an experimental BGP route server, with links between a few organisations on Citylink who were happy to participate in a trial.
The main aim of the experiment was to consider ways that Citylink connected organisations with BGP capable routers but without AS numbers might exchange routing data, so that they can send data directly between each other, without having to go through one (or several) ISP routers.
As a rudimentary route server, MRT seems to work really well (it lacks some of the RIPE tools of the Merit RA, but on the other hand, it compiles under Linux :-). We ended up setting up MRT to peer with the organisations who had AS numbers, and then setup all the other routers in AS number 65502 (chosen at random) as static routes within MRT, and redistributed them into BGP. Turn off synchronisation on the routers in AS65502, and everything seemed to work as advertised (this is speaking from memory, I haven't looked at it for about 6 weeks).
Citylink is now at the stage where we've got customers pestering us to incorporate them into the system, and a management itchy keen to take some money off of these people. So I guess what I'm after from the list is some opinions on whether this is a good idea, and I'm on the right track. Obviously, for this to be more than a toy I need the ISP's on Citylink to participate in such an exchange, so recommendations on making it easier for them to be involved are welcome.
Cheers Si
--------- To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
--------- To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
Hi Shane On Thu, 8 Oct 1998, Shane Cole wrote:
Why send traffic where it doesn't need to go? Does CNHL have the resource (in Simon Blake perhaps) to manage it?
I'm equal partner in Katipo Communications, for all intents and purposes selfemployed - if Citylink are happy to contract me to look after WIX, I'm really keen to do so.
What sort of charge would be levied on customers in addition to what they currently pay?
That's a matter for discussion, both within Citylink and publicly, I guess, and I can't speak for what Citylink will attempt to charge. However, I can provide some background. About a year ago, Citylink contracted a local consulting company to do a study on whether a WIX would be a viable business opportunity for CNHL. That report concluded that this was the case, and reinforced my arguement that improved network management (a WIX) was necessary for Citylink to ensure that the ethernet network scaled to match demand. Having said that, they made some assumptions that have since proved to be invalid: a) the WIX would only be of interest to ISP's, and thus the maximum number of organisations contributing to WIX would be perhaps 10 or 12. b) we'd need a couple of big Sparc servers to run the service. Since then, we've seen demand from a much wider range of organisations (and less from the ISP's, since they're using Merit), and the hardware costs have coalesced down to a bunch of PC's running open source software. So I would hope that the board pull back from the recommendations of the report, which (from memory) were $1000 to join the exchange, and $200/month, to something a bit lower.
Its my opinion that this should be part of the service fee that is paid to maintain a connection currently but then of course I'm not a shareholder in CNHL.
Nobody wants to pay more than they have to, but, OTOH, Citylink has to at least attempt to make a profit for it's shareholders.
If it is charged in addition to, then those people who might choose not to take part (due to it costing) in fact reduce the value of the service to those who do.
Sure, but to reverse that arguement, they lose the benefits of participating if they don't. I hope that Citylink don't set the price as a barrier to entry, because the more users there are the better it is for everybody, but I'm also realistic enough to realise that the current situation (each ISP running it's own IP network over Citylink) will continue into the forseeable future, and that there will be some proportion of Citylink users for whom improving their local connectivity is not an issue worthy of the extra cost (not just the WIX charges, but organising and setting up a BGP router). Ultimately, if a Citylink customer wants to setup the interface on their router such that they can only see their ISP, that's their choice. Cheers Si --------- To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
participants (4)
-
Dean Pemberton -
Joe Abley -
Shane Cole -
Simon Blake