Re: [nznog] jumbo frames
Jumbo frames aside from the usefulness of bigger payload for mpls/tunnels etc add some value in an environment where there's some packet loss.
I would have thought it makes it worse, assuming a physical fault resulting in a constant bit error rate combined with larger packet size means higher probability that a packet is corrupted (as there are more bits in it). The resulting packet is discarded which is a bigger loss than discarding a smaller one.
For example in an environment with say a 1Gbps link where packet loss is .1% with 1500 byte MTU you get 28Mbps througput. With Jumboframes that leaps to approx 162Mbps.
Can I see the maths behind this? If you're talking about packet loss due to congestion and not using TCP then you're probably right
On Wed, 28 Mar 2007, Jonathan Woolley wrote:
Jumbo frames aside from the usefulness of bigger payload for mpls/tunnels etc add some value in an environment where there's some packet loss.
I would have thought it makes it worse, assuming a physical fault resulting in a constant bit error rate combined with larger packet size means higher probability that a packet is corrupted (as there are more bits in it). The resulting packet is discarded which is a bigger loss than discarding a smaller one.
If you're fragmenting and drop one packet of a 4-packet-sequence you waste the other 3 packets anyway as its gotta send the whole oversize packet as multiple fragments all over again. My guess is that is his reasoning - and seems fair... Mark.
Jonathan Woolley wrote:
Jumbo frames aside from the usefulness of bigger payload for mpls/tunnels etc add some value in an environment where there's some packet loss.
I would have thought it makes it worse, assuming a physical fault resulting in a constant bit error rate combined with larger packet size means higher probability that a packet is corrupted (as there are more bits in it). The resulting packet is discarded which is a bigger loss than discarding a smaller one.
For example in an environment with say a 1Gbps link where packet loss is .1% with 1500 byte MTU you get 28Mbps througput. With Jumboframes that leaps to approx 162Mbps.
Can I see the maths behind this? If you're talking about packet loss due to congestion and not using TCP then you're probably right
I assume he's talking about my TFRC math from http://wand.net.nz/~perry/max_download.php Assuming you've saturated your link, a one bit in 'n' is going to take out one packet every so often (if it takes out more than that you're not going to get anything much worth discussing over your link in the first place). So increasing the MTU isn't going to make it more likely that a packet is lost, it just means you need to retransmit a larger packet. However, since you're using larger MTU's, it's going to take you less RTT's to get back up to full speed on your link, and therefore your link will spend more of its time going at full rate than if you had a lower MTU (since it would require more RTT's at lower throughput before it starts saturating the link again).
On Wed, 2007-03-28 at 17:07 +1200, Perry Lorier wrote:
Can I see the maths behind this? If you're talking about packet loss due to congestion and not using TCP then you're probably right
I assume he's talking about my TFRC math from http://wand.net.nz/~perry/max_download.php
That's what I'm talking about. I wouldn't normally do a post like the following but MTU and particularly TCP receive window issues need to grokked by all operators. So here goes.... Last year I was invited by the nice folks at REANNZ to present to their members practical insights learnt from operating national gigabit services. What I presented built upon the fantastic work done by Perry and the top blokes at WAND. There's been a fair bit of talk about MTU, but really the common enemy to us all is retarded default TCP window buffer settings in common operating systems. If collectively we're going to make the NZ internet blisteringly fast we've all got a fair amount of time dedicated to getting peoples heads around the LFN effect to make that happen. I can not emphasize enough how big a deal this is. The WAND guys have said it at conference before, but it needs to be said again. Small buffers affect internet performance like a cliff face and frankly I think is a major culprit in poor performance to the US and attached interweb which is 120-150ms away. We really need to make people aware of this issue. We want them to fill the tubes right? For those of you vaguely interested, check this out: https://osn.fx.net.nz/LFN/LFN/LFN.html cheers jamie
On Wed, 2007-03-28 at 23:46 +1200, Jamie Baddeley wrote:
For those of you vaguely interested, check this out: https://osn.fx.net.nz/LFN/LFN/LFN.html
Nyet! http://osn.fx.net.nz/LFN/LFN/LFN.html works. jamie
nice ad :P Jamie Baddeley wrote:
On Wed, 2007-03-28 at 23:46 +1200, Jamie Baddeley wrote:
For those of you vaguely interested, check this out: https://osn.fx.net.nz/LFN/LFN/LFN.html
Nyet!
http://osn.fx.net.nz/LFN/LFN/LFN.html
works.
jamie
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On 28/03/2007 11:46 p.m., Jamie Baddeley wrote:
On Wed, 2007-03-28 at 17:07 +1200, Perry Lorier wrote:
Can I see the maths behind this? If you're talking about packet loss due to congestion and not using TCP then you're probably right
I assume he's talking about my TFRC math from http://wand.net.nz/~perry/max_download.php
That's what I'm talking about.
Thanks Jamie - something I'm sure I've heard before but never took the time to find out more about, for no good reason other than I'm busy. For any other idiots like me: http://en.wikipedia.org/wiki/Bandwidth-delay_product Gerard -- Netspace Services Limited http://www.netspace.net.nz Phone +64 4 917 8098 Mobile +64 21 246 2266 Level One, 220 Thorndon Quay, Thorndon PO Box 12-082, Thorndon, Wellington 6004, New Zealand
There's been a fair bit of talk about MTU, but really the common enemy to us all is retarded default TCP window buffer settings in common operating systems. If collectively we're going to make the NZ internet blisteringly fast we've all got a fair amount of time dedicated to getting peoples heads around the LFN effect to make that happen.
Think it's only a few guys that are cool enough to have big enough networks to see bandwidth delay issues on Long Fast Networks? Next time you're on a XP machine on DSL, goto http://nzadsl.co.nz/speedtest/ and check what speed you're getting. You'll probably get no more than about 4.5mbit/s using a SINGLE tcp stream. (Multiple TCP streams will of course get more bandwidth) This is because XP has a RWin of about 17,520ish bytes. Latency over the DSL network to the speed test is about 30ms: * 17520 in bits (rather than bytes) is 17520*8 = 140160 bits. * 30ms in seconds is .03s. Thus bits / .03 s = 4,672,000 bits/second = 4.7Mb/s Even if your line rate is 6mbit you'll only get about 4.7mbit/s. This is why people say "turning off interleaving makes your line go faster". It doesn't; it lowers the latency, and thus increases your throughput. You could get the exact same effect by increasing rwin. But Wait! It gets WORSE. It's 170ms RTT to the US from here. So: 140160 bits / .17s = 824,470 bits/s == 824.5kbit/s. So the most bandwidth you can get out of a default XP install over a single TCP stream from the US is about 800kbit/s. What about .EU? Well, lets call .eu about 300ms... 140160 bits / .3 = 467,200 bits/s == 467kbit/s. So you can't get more than 450kbit/s over a single TCP stream from europe on a default XP install, *even if your line rate is 6mbit/s and there is no congestion anywhere*. So what should you set your rwin to? rwin (in bytes) = (bandwidth_in_bits_per_second/(latency_in_ms/1000))/8 Thus to get 6mbit over a single tcp stream from europe 300ms away you need: estimated_ws = 2,500,000 = (6,000,000/(300/1000))/8 That is about 2.5 MB of rwin. Good (large) rwin sizes are[1]: 64,240 128,480 513,920 1,027,840 2,055,680 4,111,360 8,222,720 So we pick 4,111,360 bytes as our optimum window size. I'm not a windows person, you modify your registry at your own risk. If it breaks, you get to keep all the pieces. If you're confused already, don't try this at home! I believe the keys you want to set/modify/twiddle/whatever are: HKLM/SYSTEM\CurrentControlSet\Services\Tcpip\Parameters GlobalMaxTcpWindowSize="4111360" HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters TcpWindowSize="4111360" Now try the speedtest again... ---- [1]: rwin's should be n*mss*2**m, where you want n to be as large as possible, and m as small as possible. (n*mss) can't be larger than 65535. mss is usually 1460, so n can't be larger than 44. so 64240*2**m, pick m as appropriate.
Perry Lorier wrote:
Think it's only a few guys that are cool enough to have big enough networks to see bandwidth delay issues on Long Fast Networks?
Next time you're on a XP machine on DSL, goto http://nzadsl.co.nz/speedtest/ and check what speed you're getting. You'll probably get no more than about 4.5mbit/s using a SINGLE tcp stream. (Multiple TCP streams will of course get more bandwidth)
This is because XP has a RWin of about 17,520ish bytes. Latency over the DSL network to the speed test is about 30ms:
[snip]
But Wait! It gets WORSE. It's 170ms RTT to the US from here. So:
140160 bits / .17s = 824,470 bits/s == 824.5kbit/s.
[snip lots more clever maths] So: All service providers should be looking at deploying ALGs/proxies running on hosts with much better TCP configuration, to avoid the long haul latency impact? Easier than fixing every single client host... aj.
On Thu, 2007-03-29 at 10:12 +1000, Alastair Johnson wrote:
Perry Lorier wrote:
Think it's only a few guys that are cool enough to have big enough networks to see bandwidth delay issues on Long Fast Networks?
Next time you're on a XP machine on DSL, goto http://nzadsl.co.nz/speedtest/ and check what speed you're getting. You'll probably get no more than about 4.5mbit/s using a SINGLE tcp stream. (Multiple TCP streams will of course get more bandwidth)
This is because XP has a RWin of about 17,520ish bytes. Latency over the DSL network to the speed test is about 30ms:
[snip]
But Wait! It gets WORSE. It's 170ms RTT to the US from here. So:
140160 bits / .17s = 824,470 bits/s == 824.5kbit/s.
[snip lots more clever maths]
So: All service providers should be looking at deploying ALGs/proxies running on hosts with much better TCP configuration, to avoid the long haul latency impact?
That's exactly what the satellite guys do.
Easier than fixing every single client host...
A mixture of both is the answer. Proxies would be OK for the internet but not so ok for private networking, so customer education is required regardless. jamie
On 29/03/2007, at 12:32 PM, jamie baddeley wrote:
On Thu, 2007-03-29 at 10:12 +1000, Alastair Johnson wrote:
Easier than fixing every single client host...
A mixture of both is the answer. Proxies would be OK for the internet but not so ok for private networking, so customer education is required regardless.
But what about our end to end principle that was so thoroughly defended during the IPv6/NAT/oh-noes-the-internets-be-breaking thread earlier this year? :) For satellite networks, yep, ALGs are the answer. They're built into many satellite modems, including those that IPstar use. For anything else there's no point in introducing yet another piece of serious complexity into the network. As you say, user education is the way to go. SCP runs like a dog over satellite links. I'd be interested to know if the TCP acceleration built into the IPstar terminals has any effect on SCP (the stock version with inbuilt windowing). Anyone with an IPstar connection able to shed some light? Cheers, Jonny.
Alastair Johnson wrote:
Perry Lorier wrote:
Think it's only a few guys that are cool enough to have big enough networks to see bandwidth delay issues on Long Fast Networks?
Next time you're on a XP machine on DSL, goto http://nzadsl.co.nz/speedtest/ and check what speed you're getting. You'll probably get no more than about 4.5mbit/s using a SINGLE tcp stream. (Multiple TCP streams will of course get more bandwidth)
This is because XP has a RWin of about 17,520ish bytes. Latency over the DSL network to the speed test is about 30ms:
[snip]
But Wait! It gets WORSE. It's 170ms RTT to the US from here. So:
140160 bits / .17s = 824,470 bits/s == 824.5kbit/s.
[snip lots more clever maths]
So: All service providers should be looking at deploying ALGs/proxies running on hosts with much better TCP configuration, to avoid the long haul latency impact?
Putting a web proxy in New Zealand that parents of a web proxy in the US and putting a wpad.<domainname> server in your network pointing at them would probably produce a pretty significant performance boost for people doing HTTP without the nastiness of transparent proxies or other middleboxes causing issues. WAND did on this in late 90's about how to use whatever bandwidth NZ had IIRC before the southern cross cable was finished: https://secure.wand.net.nz/pubDetail.php?id=23 https://secure.wand.net.nz/pubDetail.php?id=13 https://secure.wand.net.nz/pubDetail.php?id=24 https://secure.wand.net.nz/pubDetail.php?id=25
Some of you may have spotted Perry's "deliberate" mistake Perry Lorier wrote:
So what should you set your rwin to? rwin (in bytes) = (bandwidth_in_bits_per_second/(latency_in_ms/1000))/8
Thus to get 6mbit over a single tcp stream from europe 300ms away you need: estimated_ws = 2500,000 = (6,000,000/(300/1000))/8 That is about 2.5 MB of rwin.
That should be bandwidth delay _product_ so: estimated_ws = 225,000 = (6,000,000*(300/1000))/8 That is about 220kB of rwin which is not quite so bad as burning 4MB on every TCP connection. Note also that the TCP send window needs to be set the same so if your US or Euro server isn't similarly configured then you're screwed anyway. Richard.
From a (crazed jumbo frame fanatic) network engineer at US cluster computing facility after 5 years of jumbo frame research and 3 years jumbo frame enabled success...
The broad scope LFN "math" for jumbo frames with TCP like protocols is: MaxRate = MSS/RTT * 0.7/sqrt(p) where MSS is maximum segment size, RTT is the round trip time, and p is the probability of packet loss. (Mathis et.al 1997) Since RTT is usually not under your control the equation is optimised by increasing MSS or reducing packet loss. Quick notes: -We see a throughput increases of 2-5 times in the building with 9K jumbo frames on GigE and 10GbE links. (Massive data sets and image files) -The latest Linux (e.g. 2.6.18) has autotuned TCP, all you need to do is increase the net.core and net.ipv4 maximum buffer values to match the expected maximum RTT. (We use 16Mbytes) -The latest Linux also has the new draft ieee pathmtu. -You need to watch out for applications. For example ssh has an internal TCP window and on an LFN you want the psc.edu patch. -Although most network gear is now jumbo clean (i.e. 9000 bytes and above) there are a number of firewalls that aren't. Anyone willing to run bwctld/iperf that wants a remote throughput test point please drop me an email. (There are a number of public bwctl servers on Internet2 if you prefer a different or more distant site.) http://www.psc.edu/~mathis/MTU/ {details of most of the above on this link} http://e2epi.internet2.edu/bwctl/ http://www.psc.edu/~rapier/hpn-ssh/ Paul Hyder NOAA Earth System Research Laboratory, Global Systems Division, High Performance Computing Boulder, Colorado The NZNOG list?... Because, I spend as much time as possible in Twizel. And yes, I got the gray beard the slow way. Jonathan Woolley wrote:
Jumbo frames aside from the usefulness of bigger payload for mpls/tunnels etc add some value in an environment where there's some packet loss.
I would have thought it makes it worse, assuming a physical fault resulting in a constant bit error rate combined with larger packet size means higher probability that a packet is corrupted (as there are more bits in it). The resulting packet is discarded which is a bigger loss than discarding a smaller one.
For example in an environment with say a 1Gbps link where packet loss is .1% with 1500 byte MTU you get 28Mbps througput. With Jumboframes that leaps to approx 162Mbps.
Can I see the maths behind this? If you're talking about packet loss due to congestion and not using TCP then you're probably right
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
participants (11)
-
Alastair Johnson -
Gerard Creamer -
jamie baddeley -
Jamie Baddeley
-
Jonathan Woolley -
Jonny Martin -
Mark Foster -
Paul Hyder -
Perry Lorier -
Richard Nelson -
Tom