sounds like a problem due to route filters more than anything. All your IP's or just a block? jamie On Thu, 2007-07-19 at 09:07 +1200, Nick Larsen (ZeroOne Operations) wrote:

The noticeable problem we faced, was traffic on the Telstra network destined for our IP's was being routed overseas, and back to us, then the traffic back to Telstra was going domestically.

Usually this wouldn't have been such a problem for the average user, but since port 80 traffic was timing out, we believe one leg of the traffic flow was somehow missing Telstra's caching proxy, but I've had no confirmation on this, so it's only theoretical.

Nick Larsen Network Technician

ZeroOne (NZ) Limited 0800 ZEROONE www.zeroone.co.nz

-----Original Message----- From: Joe Abley [mailto:jabley(a)ca.afilias.info] Sent: Thursday, 19 July 2007 2:46 a.m. To: Anton Smith Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Telstra contact - BGP advertisements

On 18-Jul-2007, at 06:30, Anton Smith wrote:

...

How many operators out there are using strict RPF?

With a lot of interconnection points around the place to me it seems a bit harsh to use strict RPF.

And in fact, I wonder if Telstra Clear are using it, since a lot of times I notice broken connectivity due to what appears to be asymmetric routing, and I then have to work around it.

I think two useful things to remember are:

1. All routing is asymmetric, in general.

2. "Strict-mode" RPF is inappropriate to apply to an interface if the thing at the other end has other connectivity to the Internet. Unless you are absolutely convinced that the device at the far end is not multi-homed, applying strict-mode RPF is an error (and presumably should be corrected just like any other error).

So, if $carrier has applied strict-mode RPF checks to their interface facing you, and you are multi-homed, you need to call them to report their configuration error so it can be corrected. If they doubt the existence of the error, point them at RFC 3704/BCP 84.

However, if the path from A in your network to B elsewhere is not the same as the path from B back to A, then that's normal, and the observed asymmetry (in isolation) is no reason for a change. (If one or both of the paths is congested, or has some other problem, then obviously those are still problems that might deserve escalation.)

Joe

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

If your an ISP and connecting to TelstraClear and you are multi-homed you can request a non-blended connection. Non-blended connections do not go through their proxy system. How do you tell if you have a non-blended connection? Your domestic and international traffic are delivered down separate VLANs/circuits. Blended connections have all your traffic delivered down a single connection. -----Original Message----- From: Nathan Ward [mailto:nznog(a)daork.net] Sent: Thursday, 19 July 2007 10:07 a.m. To: nznog Subject: Re: [nznog] Telstra contact - BGP advertisements Bingo, this man wins a prize, Anton does not :-P Based on some tinkering I did some time ago, TelstraClear appear to run fully transparent HTTP proxy servers. Fully transparent means that they terminate your TCP session that was originally intended for some server overseas (as per partially-transparent), and then establish a new TCP session out to the original destination from YOUR IP address. When HTTP packets come back from Out There, they redirect traffic "to you" back to the appropriate proxy server. So, Nick's TelstraClear-connected visitors' TCP sessions are going out to a TelstraClear proxy, then new sessions are being built from the proxies (ip.src == visitor) to Nick's servers, and Nick's servers are returning traffic directly to the customer, not via TelstraClear. As you can imagine, ephemeral ports and so on probably don't always match, so connections get thrown away. This sort of transparent proxying was really designed for enterprise networks, or providers who have a single border network. Personally, I prefer to deploy partially transparent proxying - sure, some really s**ty old sites that use IP as a unique identifier for your session (etc.) don't work well, but IMO it's better than this (relatively) hard to diagnose bug. The other alternative is, of course, to push all traffic to other ASes via some border routers (or more importantly, your switches that do your cache offloading). That is, however, often pretty difficult/ costly. On 19/07/2007, at 9:07 AM, Nick Larsen (ZeroOne Operations) wrote:

The noticeable problem we faced, was traffic on the Telstra network destined for our IP's was being routed overseas, and back to us, then the traffic back to Telstra was going domestically.

Usually this wouldn't have been such a problem for the average user, but since port 80 traffic was timing out, we believe one leg of the traffic flow was somehow missing Telstra's caching proxy, but I've had no confirmation on this, so it's only theoretical.

Nick Larsen Network Technician

ZeroOne (NZ) Limited 0800 ZEROONE www.zeroone.co.nz

-----Original Message----- From: Joe Abley [mailto:jabley(a)ca.afilias.info] Sent: Thursday, 19 July 2007 2:46 a.m. To: Anton Smith Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Telstra contact - BGP advertisements

On 18-Jul-2007, at 06:30, Anton Smith wrote:

...

How many operators out there are using strict RPF?

With a lot of interconnection points around the place to me it seems a bit harsh to use strict RPF.

And in fact, I wonder if Telstra Clear are using it, since a lot of times I notice broken connectivity due to what appears to be asymmetric routing, and I then have to work around it.

I think two useful things to remember are:

1. All routing is asymmetric, in general.

2. "Strict-mode" RPF is inappropriate to apply to an interface if the thing at the other end has other connectivity to the Internet. Unless you are absolutely convinced that the device at the far end is not multi-homed, applying strict-mode RPF is an error (and presumably should be corrected just like any other error).

So, if $carrier has applied strict-mode RPF checks to their interface facing you, and you are multi-homed, you need to call them to report their configuration error so it can be corrected. If they doubt the existence of the error, point them at RFC 3704/BCP 84.

However, if the path from A in your network to B elsewhere is not the same as the path from B back to A, then that's normal, and the observed asymmetry (in isolation) is no reason for a change. (If one or both of the paths is congested, or has some other problem, then obviously those are still problems that might deserve escalation.)

Joe

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

!DSPAM:22,469e8124262211160419206!

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

You're right, we're not a TelstraClear customer. I got some excellent help from a TelstraClear "router monkey", who diagnosed the problem, and it seems one of our upstream providers had to issue the good old 'clear ip bgp neighbor x.x.x.x soft in' command... Oops. So our routing is now fine (Thanks for help on this Telstra =D). Nick Larsen Network Technician ZeroOne (NZ) Limited 0800 ZEROONE www.zeroone.co.nz -----Original Message----- From: Nathan Ward [mailto:nznog(a)daork.net] Sent: Thursday, 19 July 2007 10:24 a.m. To: nznog Subject: Re: [nznog] Telstra contact - BGP advertisements Nick's routes are going overseas when coming from TelstraClear - this tells me that he's not a TelstraClear customer. (In addition to that, if he was a customer, he'd be asking his account manager for help, not TelstraClear router monkeys via NZNOG). In addition, this problem only occurs when TelstraClear thing a domestic prefix is somewhere internationally - my tinkering suggested that they didn't proxy connections to domestic HTTP servers. On 19/07/2007, at 10:18 AM, Philip D'Ath wrote:

If your an ISP and connecting to TelstraClear and you are multi-homed you can request a non-blended connection. Non-blended connections do not go through their proxy system.

How do you tell if you have a non-blended connection? Your domestic and international traffic are delivered down separate VLANs/circuits. Blended connections have all your traffic delivered down a single connection.

-----Original Message----- From: Nathan Ward [mailto:nznog(a)daork.net] Sent: Thursday, 19 July 2007 10:07 a.m. To: nznog Subject: Re: [nznog] Telstra contact - BGP advertisements

Bingo, this man wins a prize, Anton does not :-P

Based on some tinkering I did some time ago, TelstraClear appear to run fully transparent HTTP proxy servers. Fully transparent means that they terminate your TCP session that was originally intended for some server overseas (as per partially-transparent), and then establish a new TCP session out to the original destination from YOUR IP address. When HTTP packets come back from Out There, they redirect traffic "to you" back to the appropriate proxy server.

So, Nick's TelstraClear-connected visitors' TCP sessions are going out to a TelstraClear proxy, then new sessions are being built from the proxies (ip.src == visitor) to Nick's servers, and Nick's servers are returning traffic directly to the customer, not via TelstraClear. As you can imagine, ephemeral ports and so on probably don't always match, so connections get thrown away.

This sort of transparent proxying was really designed for enterprise networks, or providers who have a single border network. Personally, I prefer to deploy partially transparent proxying - sure, some really s**ty old sites that use IP as a unique identifier for your session (etc.) don't work well, but IMO it's better than this (relatively) hard to diagnose bug. The other alternative is, of course, to push all traffic to other ASes via some border routers (or more importantly, your switches that do your cache offloading). That is, however, often pretty difficult/ costly.

On 19/07/2007, at 9:07 AM, Nick Larsen (ZeroOne Operations) wrote:

...

The noticeable problem we faced, was traffic on the Telstra network destined for our IP's was being routed overseas, and back to us, then the traffic back to Telstra was going domestically.

Usually this wouldn't have been such a problem for the average user, but since port 80 traffic was timing out, we believe one leg of the traffic flow was somehow missing Telstra's caching proxy, but I've had no confirmation on this, so it's only theoretical.

Nick Larsen Network Technician

ZeroOne (NZ) Limited 0800 ZEROONE www.zeroone.co.nz

-----Original Message----- From: Joe Abley [mailto:jabley(a)ca.afilias.info] Sent: Thursday, 19 July 2007 2:46 a.m. To: Anton Smith Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Telstra contact - BGP advertisements

On 18-Jul-2007, at 06:30, Anton Smith wrote:

...

How many operators out there are using strict RPF?

With a lot of interconnection points around the place to me it seems a bit harsh to use strict RPF.

And in fact, I wonder if Telstra Clear are using it, since a lot of times I notice broken connectivity due to what appears to be asymmetric routing, and I then have to work around it.

I think two useful things to remember are:

1. All routing is asymmetric, in general.

2. "Strict-mode" RPF is inappropriate to apply to an interface if the thing at the other end has other connectivity to the Internet. Unless you are absolutely convinced that the device at the far end is not multi-homed, applying strict-mode RPF is an error (and presumably should be corrected just like any other error).

So, if $carrier has applied strict-mode RPF checks to their interface facing you, and you are multi-homed, you need to call them to report their configuration error so it can be corrected. If they doubt the existence of the error, point them at RFC 3704/BCP 84.

However, if the path from A in your network to B elsewhere is not the same as the path from B back to A, then that's normal, and the observed asymmetry (in isolation) is no reason for a change. (If one or both of the paths is congested, or has some other problem, then obviously those are still problems that might deserve escalation.)

Joe

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

!DSPAM:22,469e91b5262212093516027!

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog