bufferbloat talk at nznog friday

Dave Taht

25 Jan 2015 25 Jan '15

6:45 p.m.

I am curious as to what aspects of bufferbloat I should cover more deeply in my upcoming talk at nznog this friday? http://www.nznog.org/home/2015-papers Most of my own personal focus has been on speeding up the edge of the internet, and now wifi, while I have a feeling I should touch upon more on island-to-world connectivity and satellite connectivity issues for this audience. We've given plenty of talks in the past. See talks by van jacobson, jim gettys, jesper dangaurd-bauer, Stephen hemminger, Eric Dumazet, Toke Hoiland Jorgensen, here: http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos And the manifesto for FQ+AQM here: https://gettys.wordpress.com/2013/07/10/low-latency-requires-smart-queuing-t... So... To what extent do NZNOG folk already grok the causes and cures for bufferbloat? I'm perfectly willing to drop my usual detailed explanation of how tcp bandwidth probing works and talk abut new stuff going on, and wifi, if desired... talk more to bandwidth management and traffic engineering tools, etc. ... there is someone here today doing mikrotik training, and as yet only mikrotik only supports SFQ, not fq_codel. http://forum.mikrotik.com/viewtopic.php?f=1&t=89221 http://forum.mikrotik.com/viewtopic.php?f=2&t=63594 I have a pair of slides on the differences between SFQ and fq_codel. ... And so on. I'm here all week... I've borrowed a NZ vm, run a few experiments worldwide, am fiddling with some parameters... -- Dave Täht http://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Show replies by date

Ewen McNeill

25 Jan 25 Jan

7:46 p.m.

Hi Dave, On 26/01/15 7:45, Dave Taht wrote:

...

I am curious as to what aspects of bufferbloat I should cover more deeply in my upcoming talk at nznog this friday? [... and] [t]alk abut new stuff going on, and wifi, if desired... talk more to bandwidth management and traffic engineering tools,

I think at this point there's pretty reasonable understanding of how bufferbloat happens -- and less understanding of how to practically deal with it, especially within an operator network which faces a variety of latency challenges, eg, cross-town, within-NZ, various distances outside US, satellite. Clearly the transfer speeds for some of those are improved by using non-trivial buffers, and the transfer speeds of others are made drastically worse by having too much in flight. I suspect many operators have a mix of all of them. (Obviously if the customer has their own buffer bloat issues the operator can't magically fix things completely -- but "do no harm", ie don't make things worse, seems a reasonable aim.) If you had any insights on, eg, how bufferbloat interoperated with upstream provider policing policies and how to deal with that that'd likely be of interest too. I'm guessing is "shape within the upstream policing envelope" -- but that's sometimes easier said than done if the upstream providers policing is very sensitive to instantaneous bursts of packets. And any advice on defining such operator shaping/policing policies to make them less likely to cause adverse behaviour (eg, being less sensitive to packet arrival time of the "last packet of the burst arrived 1ms too soon, so we threw it away, have a nice day" variety).

...

there is someone here today doing mikrotik training, and as yet only mikrotik only supports SFQ, not fq_codel. [...] I have a pair of slides on the differences between SFQ and fq_codel.

I think it would be great to talk about this too. And also if there's any thing that can be done with the Mikrotik (and hence SFQ) to mitigate the disadvantages. There are lots of Mikrotiks deployed in New Zealand, including a whole bunch on not-entirely-predictable-bandwidth (unlicensed) multi-mile radio links.

...

And so on. I'm here all week... I've borrowed a NZ vm, run a few experiments worldwide, am fiddling with some parameters...

I think many would be interested in any parameters that worked well for in-NZ, to Australia (relatively low latency, several CDNs/Amazon/etc there), to USA (much of content there) and to Europe. So any results you find would be great. Plus of course if there are specific things that, eg, operator helpdesks should be recommending to customers that complain of transfer speed issues that seem like bufferbloat related that'd also be useful. But could possibly just be some slides pointing to useful resources for end users. Ewen

Jonathon Exley

26 Jan 26 Jan

4:13 a.m.

On a related note, I have been poking around in ns2 trying to see if I can simulate TCP flows over a UFB service to see the effect of different BWP burst sizes. However the diffserv model in ns2 doesn't drop packets marked red, it only assigns a higher RED drop probability - not quite the behaviour I was looking for. Has anyone on the list played with ns2 or ns3 and looked at modelling QoS stuff? Jonathon. -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Ewen McNeill Sent: Monday, 26 January 2015 8:47 a.m. To: Dave Taht; NZNOG(a)list.waikato.ac.nz Subject: Re: [nznog] bufferbloat talk at nznog friday ---8<--- And any advice on defining such operator shaping/policing policies to make them less likely to cause adverse behaviour (eg, being less sensitive to packet arrival time of the "last packet of the burst arrived 1ms too soon, so we threw it away, have a nice day" variety). ---8<--- This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

Dave Taht

27 Jan 27 Jan

3:27 a.m.

On Mon, Jan 26, 2015 at 5:13 PM, Jonathon Exley wrote:

...

On a related note, I have been poking around in ns2 trying to see if I can simulate TCP flows over a UFB service to see the effect of different BWP burst sizes.

I don't get the latter two acronyms.

...

However the diffserv model in ns2 doesn't drop packets marked red, it only assigns a higher RED drop probability - not quite the behaviour I was looking for.

I think my not understanding the first part of the question leads to me not understanding this part.

...

Has anyone on the list played with ns2 or ns3 and looked at modelling QoS stuff?

The release candidate ns2 code containing pie, codel, fq_codel, red and variants: http://nsnam.isi.edu/nsnam/index.php/Roadmap The release candiate ns3 code incorporating codel and fq_codel: http://www.nsnam.org/wiki/AQM_enhancements Both are nearly done and could use more testers. I personally avoid ns2 (tcl. ugh.) for ns3 (python, yea!), and in neither case will a ns2 or ns3 model give you results that even remotely match the real (bufferbloated) world, unless you configure your sim to match it. (there's a pretty good test in the ns3 codebase now). How I think about "QoS" is pretty different, In terms of importance: fixing the device driver queues with techniques like BQL, then fair queueing, followed by aqm, fixing the actual device driver queues, and then and only then attempting packet prioritization of any kind. By just applying "QoS" you are trying to direct the end of the firehose whilst holding onto it next to the fire-engine.

...

Jonathon.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Ewen McNeill Sent: Monday, 26 January 2015 8:47 a.m. To: Dave Taht; NZNOG(a)list.waikato.ac.nz Subject: Re: [nznog] bufferbloat talk at nznog friday

---8<---

And any advice on defining such operator shaping/policing policies to make them less likely to cause adverse behaviour (eg, being less sensitive to packet arrival time of the "last packet of the burst arrived 1ms too soon, so we threw it away, have a nice day" variety).

---8<---

This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Dave Täht thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Ewen McNeill

3:54 a.m.

On 27/01/15 16:27, Dave Taht wrote:

...

On Mon, Jan 26, 2015 at 5:13 PM, Jonathon Exley wrote:

...
On a related note, I have been poking around in ns2 trying to see if I can simulate TCP flows over a UFB service to see the effect of different BWP burst sizes.

I don't get the latter two acronyms.

UFB == Ultra Fast Broadband, the marketing term for (basically) fibre to the home/premises in New Zealand (see, eg, http://ufb.org.nz/). BWP I'd assume was BandWidth Product from context. Crossing the streams from another Dave Taht reply to the NZNOG list:

...

Lastly, I couldn't help but want this big islam firmware upgrade for ipv6 support to include bufferbloat related fixes, but I know I'm dreaming....

Let's dream big. It sounds like Chorus (who run a large chunk of the DSL headends in New Zealand) are already talking to their vendor at present about wanting a firmware release with a specific combination of features. And contemplating a country-wide rollout when they find one they can put into production. That's definitely not going to be an upgrade that happens every day! (Nor is it likely to be happening tomorrow, so there's still time to tweak the plan.) If there are specific ISAM features/versions/bug fixes/whatever that would make the bufferbloat situation better, it'd be worth talking to some of the Chorus folks (two of whom I see active on the NZNOG list this week, and some of whom I assume will be at the NZNOG conference later this week) and giving them specific things to add to their "and we want these features too" list. Because if it's not this country-wide rollout, it might be Many Months before there's another one to try to get it included in. Ewen

Jonathon Exley

3:57 a.m.

Oops - acronym soup. UFB = Ultrafast Broadband, delivered on fibre to the premises. BWP = bandwidth profile, as per MEF standards. The issue we face is that the rollout of the UFB network is subject to contracted SLAs which have very tight frame delay variation. In order to achieve the SLA target, Chorus have implemented small burst sizes on the ingress policers. If the traffic coming from the service provider or the end user is not shaped to match the burst size then TCP throughput is reduced. Unfortunately some equipment struggles to shape the traffic with sufficiently small egress bursts to get good throughput. It would be interesting to see if active queue management such as codel & variants improves throughput in this situation - which is sort of the reverse of the conditions that prompted its development (not enough delay rather than too much). Jonathon. -----Original Message----- From: Dave Taht [mailto:dave.taht(a)gmail.com] Sent: Tuesday, 27 January 2015 4:27 p.m. To: Jonathon Exley Cc: NZNOG(a)list.waikato.ac.nz Subject: Re: [nznog] bufferbloat talk at nznog friday On Mon, Jan 26, 2015 at 5:13 PM, Jonathon Exley wrote:

...

On a related note, I have been poking around in ns2 trying to see if I can simulate TCP flows over a UFB service to see the effect of different BWP burst sizes.

I don't get the latter two acronyms.

...

However the diffserv model in ns2 doesn't drop packets marked red, it only assigns a higher RED drop probability - not quite the behaviour I was looking for.

I think my not understanding the first part of the question leads to me not understanding this part.

...

Has anyone on the list played with ns2 or ns3 and looked at modelling QoS stuff?

...

Jonathon.

-----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Ewen McNeill Sent: Monday, 26 January 2015 8:47 a.m. To: Dave Taht; NZNOG(a)list.waikato.ac.nz Subject: Re: [nznog] bufferbloat talk at nznog friday

---8<---

And any advice on defining such operator shaping/policing policies to make them less likely to cause adverse behaviour (eg, being less sensitive to packet arrival time of the "last packet of the burst arrived 1ms too soon, so we threw it away, have a nice day" variety).

---8<---

This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Dave Täht thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Dave Taht

3:45 a.m.

On Mon, Jan 26, 2015 at 8:46 AM, Ewen McNeill wrote:

...

Hi Dave,

On 26/01/15 7:45, Dave Taht wrote:

...
I am curious as to what aspects of bufferbloat I should cover more deeply in my upcoming talk at nznog this friday? [... and] [t]alk abut new stuff going on, and wifi, if desired... talk more to bandwidth management and traffic engineering tools,

I think at this point there's pretty reasonable understanding of how bufferbloat happens -- and less understanding of how to practically deal with it, especially within an operator network which faces a variety of latency challenges, eg, cross-town, within-NZ, various distances outside US, satellite. Clearly the transfer speeds for some of those are improved by using non-trivial buffers, and the transfer speeds of others are made drastically worse by having too much in flight. I suspect many operators have a mix of all of them. (Obviously if the customer has their own buffer bloat issues the operator can't magically fix things completely -- but "do no harm", ie don't make things worse, seems a reasonable aim.)

If you had any insights on, eg, how bufferbloat interoperated with upstream provider policing policies and how to deal with that that'd likely be of interest too.

Well, I would love it if I could get more data from folk willing to install netperf-wrapper and run a 5 minute test that would be good. In particular, I'd like to know how the 100mbit fiber links in the cities are behaving, a typical slow DSL connection, a faster VDSL connection, and I see there are tons and tons of wifi links here, so if I could get some folk here to install netperf-wrapper and run it's test suite and supply some results run across those technologies, I'd be able to get a bit of a grip on what is going on here, and talk to those results over beer. Speedtest results don't cut it. - they don't test for latency under load. get netperf-wrapper from: https://github.com/tohojo/netperf-wrapper I note that bandwidth at the hotel here is quite poor, so if you can install the needed dependencies before arriving at the conference, I'm sure everyone would be grateful. on debian derived linux those are: apt-get install python-matplotlib python-qt4 fping apt-get build-dep netperf git clone https://github.com/dtaht/netperf.git git clone https://github.com/tohojo/netperf-wrapper.git cd netperf; ./configure --enable-demo; make; sudo make install cd ../netperf-wrapper; sudo python setup.py install An example script is here: http://snapon.lab.bufferbloat.net/~d/nz/ - note I dont have a decent server setup in country... It is also possible to get this to build on a mac, that's more involved, requires macports, time, and beer.

...

I'm guessing is "shape within the upstream policing envelope"

I would rather like folk to shape rather than police wherever possible.

...

-- but that's sometimes easier said than done if the upstream providers policing is very sensitive to instantaneous bursts of packets.

And they are. Policing is based on a radically incorrect assumption on how flows behave. It *IS* possible to build a kinder, gentler, policer, but I'll get to it in my talk.

...

And any advice on defining such operator shaping/policing policies to make them less likely to cause adverse behaviour (eg, being less sensitive to packet arrival time of the "last packet of the burst arrived 1ms too soon, so we threw it away, have a nice day" variety).

Demand shaped, rather than policed connections in your SLAs?

...

...
there is someone here today doing mikrotik training, and as yet only mikrotik only supports SFQ, not fq_codel. [...] I have a pair of slides on the differences between SFQ and fq_codel.

I think it would be great to talk about this too. And also if there's any thing that can be done with the Mikrotik (and hence SFQ) to mitigate the disadvantages. There are lots of Mikrotiks deployed in New Zealand, including a whole bunch on not-entirely-predictable-bandwidth (unlicensed) multi-mile radio links.

Got it.

...

...
And so on. I'm here all week... I've borrowed a NZ vm, run a few experiments worldwide, am fiddling with some parameters...

I think many would be interested in any parameters that worked well for in-NZ, to Australia (relatively low latency, several CDNs/Amazon/etc there), to USA (much of content there) and to Europe. So any results you find would be great.

It turns out very few parameter changes were required for any of these destinations (all sub 250ms) with fq_codel. I saw only a modest improvement if I changed the defaults to fully account for 250ms transit. It was pretty hopeless with everything else I tried! Of what was left SFQ was best, things like RED, hopeless on wildly varying RTTs. In my present location, however, I am seeing a *450ms* RTT to the eu, which I sure hope isn't normal for NZ, just this hotel. ( ping netperf-eu.bufferbloat.net ). dair-833:~ d$ ping netperf-eu.bufferbloat.net PING kau.toke.dk (130.243.26.64): 56 data bytes 64 bytes from 130.243.26.64: icmp_seq=0 ttl=42 time=409.153 ms 64 bytes from 130.243.26.64: icmp_seq=1 ttl=42 time=432.350 ms 64 bytes from 130.243.26.64: icmp_seq=2 ttl=42 time=455.123 ms If anyone has a power supply suitable for a wndr3800 and NZ power standards, I brought one, might be able to fix matters here.....

...

Plus of course if there are specific things that, eg, operator helpdesks should be recommending to customers that complain of transfer speed issues that seem like bufferbloat related that'd also be useful. But could possibly just be some slides pointing to useful resources for end users.

Sort of why I just asked if openwrt derived gear was a problem here. That stuff is the farthest along by far, for home cpe.

...

Ewen

-- Dave Täht thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Ewen McNeill

5:08 a.m.

Hi Dave, On 27/01/15 16:45, Dave Taht wrote:

...

Well, I would love it if I could get more data from folk willing to install netperf-wrapper and run a 5 minute test that would be good. [...]

on debian derived linux those are:

In case it helps someone else trying to test this week... For "Debian derived Linux" read "Ubuntu" AFAICT (netperf is not packaged in Debian, up through Debian Experimental, so "apt-get build-dep netperf" won't work there; I can send a packages-my-Ubuntu-system-wanted-to-install list in case someone wants to try on Debian). It's also apparently _not_ Ubuntu 14.04 LTS, because that appears to install netperf-wrapper as a Python Egg with the given install instructions, and netperf-wrapper appears to be completely unable to find its tests within the Python Egg, resulting in it complaining: -=- cut here -=- Fatal error: Hostname lookup failed for host rrul: [Errno -2] Name or service not known -=- cut here -=- You can tell you have this problem if "netperf-wrapper --list-tests" fails with a Python stack trace, rather than returning a list of tests. My kludgy workaround (which seems to have worked) was: -=- cut here -=- cd /usr/local/share && sudo ln -s ~/src/netperf-wrapper . -=- cut here -=- which then lets "nztest.sh" (http://snapon.lab.bufferbloat.net/~d/nz/nztest.sh) run after some editing to point it at something other than the non-existent NZ server (I chose the US west coast). Serious suggestion: perhaps it would help to bundle this with Docker or similar? It'd then be easier for people to install a known-to-work version with just a "needs modern Linux kernel" dependency.

...

In my present location, however, I am seeing a *450ms* RTT to the eu, which I sure hope isn't normal for NZ, just this hotel. ( ping netperf-eu.bufferbloat.net ).

It seems about 75-100ms too high. From Vodafone cable in Wellington: -=- cut here -=- ewen(a)ashram:~$ ping netperf-eu.bufferbloat.net PING kau.toke.dk (130.243.26.64): 56 data bytes 64 bytes from 130.243.26.64: icmp_seq=0 ttl=39 time=341.222 ms 64 bytes from 130.243.26.64: icmp_seq=1 ttl=39 time=342.191 ms [...] -=- cut here -=- and it's about 15ms lower (ie, around 325ms) from my colo box in central Wellington (which is roughly the cable Internet overhead, so an expected difference). 320-350ms is roughly the RTT I'd expect to see to Europe out of New Zealand, mostly due to speed-of-light not being infinite. (I do work on a few systems based in Europe so have fairly consistently seen this for years.)

...

If anyone has a power supply suitable for a wndr3800 and NZ power standards, I brought one, might be able to fix matters here.....

A quick online search suggests this is a 12V DC, 2.5A power supply (eg, http://www.myaccount.charter.com/customers/support.aspx?supportarticleid=329...). If so, they seem likely to be fairly common.

...

Sort of why I just asked if openwrt derived gear was a problem here. That stuff is the farthest along by far, for home cpe.

The main issue here for DSL is that there is an approval process ("Telepermit") for legally connecting equipment to NZ copper lines, so only models that someone has put through the approval process can be legally used (and IIRC each importer has to have their own import approved). For Vodafone Cable the CPE device is usually supplied by Vodafone (and acts as a bridge), and it's standard-Ethernet from there in, so there may be more choice. I've not gone looking for specific models that CoroWRT has focused on in NZ (except for looking for WNDR3800 just now, and not immediately finding any obviously for sale -- but IIRC the model is getting harder to find everywhere now). Ewen

Joel Wirāmu Pauling

5:45 a.m.

WNDR3800 is Telepermit/Sold in NZ I have dozens of them around the country... many running cerowrt builds. The default PSU is DC 110/240 Switchmode, a physical adaptor is all that is required. -Joel On 26 January 2015 at 21:08, Ewen McNeill wrote:

...

Hi Dave,

On 27/01/15 16:45, Dave Taht wrote:

...
Well, I would love it if I could get more data from folk willing to install netperf-wrapper and run a 5 minute test that would be good. [...]

on debian derived linux those are:

In case it helps someone else trying to test this week...

For "Debian derived Linux" read "Ubuntu" AFAICT (netperf is not packaged in Debian, up through Debian Experimental, so "apt-get build-dep netperf" won't work there; I can send a packages-my-Ubuntu-system-wanted-to-install list in case someone wants to try on Debian).

It's also apparently _not_ Ubuntu 14.04 LTS, because that appears to install netperf-wrapper as a Python Egg with the given install instructions, and netperf-wrapper appears to be completely unable to find its tests within the Python Egg, resulting in it complaining:

-=- cut here -=- Fatal error: Hostname lookup failed for host rrul: [Errno -2] Name or service not known -=- cut here -=-

You can tell you have this problem if "netperf-wrapper --list-tests" fails with a Python stack trace, rather than returning a list of tests.

My kludgy workaround (which seems to have worked) was:

-=- cut here -=- cd /usr/local/share && sudo ln -s ~/src/netperf-wrapper . -=- cut here -=-

which then lets "nztest.sh" (http://snapon.lab. bufferbloat.net/~d/nz/nztest.sh) run after some editing to point it at something other than the non-existent NZ server (I chose the US west coast).

Serious suggestion: perhaps it would help to bundle this with Docker or similar? It'd then be easier for people to install a known-to-work version with just a "needs modern Linux kernel" dependency.

In my present location, however, I am seeing a *450ms* RTT to the eu,

...
which I sure hope isn't normal for NZ, just this hotel. ( ping netperf-eu.bufferbloat.net ).

It seems about 75-100ms too high. From Vodafone cable in Wellington:

-=- cut here -=- ewen(a)ashram:~$ ping netperf-eu.bufferbloat.net PING kau.toke.dk (130.243.26.64): 56 data bytes 64 bytes from 130.243.26.64: icmp_seq=0 ttl=39 time=341.222 ms 64 bytes from 130.243.26.64: icmp_seq=1 ttl=39 time=342.191 ms [...] -=- cut here -=-

and it's about 15ms lower (ie, around 325ms) from my colo box in central Wellington (which is roughly the cable Internet overhead, so an expected difference).

320-350ms is roughly the RTT I'd expect to see to Europe out of New Zealand, mostly due to speed-of-light not being infinite. (I do work on a few systems based in Europe so have fairly consistently seen this for years.)

If anyone has a power supply suitable for a wndr3800 and NZ power

...
standards, I brought one, might be able to fix matters here.....

A quick online search suggests this is a 12V DC, 2.5A power supply (eg, http://www.myaccount.charter.com/customers/support.aspx? supportarticleid=3294). If so, they seem likely to be fairly common.

Sort of why I just asked if openwrt derived gear was a problem here.

...
That stuff is the farthest along by far, for home cpe.

The main issue here for DSL is that there is an approval process ("Telepermit") for legally connecting equipment to NZ copper lines, so only models that someone has put through the approval process can be legally used (and IIRC each importer has to have their own import approved).

For Vodafone Cable the CPE device is usually supplied by Vodafone (and acts as a bridge), and it's standard-Ethernet from there in, so there may be more choice.

I've not gone looking for specific models that CoroWRT has focused on in NZ (except for looking for WNDR3800 just now, and not immediately finding any obviously for sale -- but IIRC the model is getting harder to find everywhere now).

Ewen

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Dave Taht

6:17 a.m.

On Tue, Jan 27, 2015 at 6:45 PM, Joel Wirāmu Pauling wrote:

...

WNDR3800 is Telepermit/Sold in NZ I have dozens of them around the country... many running cerowrt builds.

Heh. I didn't know. Can you share some of your typical SQM settings for various services and bandwidths? In particular the DSL compensation code was always kind of hairy. I'd also like to note that ubnt's gear was a frequent target for the cerowrt effort, notably the pico and nanostations are extensively in play in the largest testbed. I think highly of their default firmware, but it lacks ipv6, routing support, and bufferbloat fixes. Their AirOS qos system is quite good, being fq-based, but didn't have aqm-ish facilities, and so far as I know (in their AirFiber product) they just poured the fq portion of all that into the FPGA, and didn't do much to manage queue length. A netperf-wrapper rrul (latency with load) result on one of the airfiber boxes in rain and outside of it would be interesting.

...

The default PSU is DC 110/240 Switchmode, a physical adaptor is all that is required.

Have no brick here at all. oops.

...

-Joel

On 26 January 2015 at 21:08, Ewen McNeill wrote:

...
Hi Dave,

On 27/01/15 16:45, Dave Taht wrote:

...
Well, I would love it if I could get more data from folk willing to install netperf-wrapper and run a 5 minute test that would be good. [...]

on debian derived linux those are:

In case it helps someone else trying to test this week...

For "Debian derived Linux" read "Ubuntu" AFAICT (netperf is not packaged in Debian, up through Debian Experimental, so "apt-get build-dep netperf" won't work there; I can send a packages-my-Ubuntu-system-wanted-to-install list in case someone wants to try on Debian).

It's also apparently _not_ Ubuntu 14.04 LTS, because that appears to install netperf-wrapper as a Python Egg with the given install instructions, and netperf-wrapper appears to be completely unable to find its tests within the Python Egg, resulting in it complaining:

-=- cut here -=- Fatal error: Hostname lookup failed for host rrul: [Errno -2] Name or service not known -=- cut here -=-

You can tell you have this problem if "netperf-wrapper --list-tests" fails with a Python stack trace, rather than returning a list of tests.

My kludgy workaround (which seems to have worked) was:

-=- cut here -=- cd /usr/local/share && sudo ln -s ~/src/netperf-wrapper . -=- cut here -=-

which then lets "nztest.sh" (http://snapon.lab.bufferbloat.net/~d/nz/nztest.sh) run after some editing to point it at something other than the non-existent NZ server (I chose the US west coast).

Serious suggestion: perhaps it would help to bundle this with Docker or similar? It'd then be easier for people to install a known-to-work version with just a "needs modern Linux kernel" dependency.

...
In my present location, however, I am seeing a *450ms* RTT to the eu, which I sure hope isn't normal for NZ, just this hotel. ( ping netperf-eu.bufferbloat.net ).

It seems about 75-100ms too high. From Vodafone cable in Wellington:

-=- cut here -=- ewen(a)ashram:~$ ping netperf-eu.bufferbloat.net PING kau.toke.dk (130.243.26.64): 56 data bytes 64 bytes from 130.243.26.64: icmp_seq=0 ttl=39 time=341.222 ms 64 bytes from 130.243.26.64: icmp_seq=1 ttl=39 time=342.191 ms [...] -=- cut here -=-

and it's about 15ms lower (ie, around 325ms) from my colo box in central Wellington (which is roughly the cable Internet overhead, so an expected difference).

320-350ms is roughly the RTT I'd expect to see to Europe out of New Zealand, mostly due to speed-of-light not being infinite. (I do work on a few systems based in Europe so have fairly consistently seen this for years.)

...
If anyone has a power supply suitable for a wndr3800 and NZ power standards, I brought one, might be able to fix matters here.....

A quick online search suggests this is a 12V DC, 2.5A power supply (eg, http://www.myaccount.charter.com/customers/support.aspx?supportarticleid=329...). If so, they seem likely to be fairly common.

...
Sort of why I just asked if openwrt derived gear was a problem here. That stuff is the farthest along by far, for home cpe.

The main issue here for DSL is that there is an approval process ("Telepermit") for legally connecting equipment to NZ copper lines, so only models that someone has put through the approval process can be legally used (and IIRC each importer has to have their own import approved).

For Vodafone Cable the CPE device is usually supplied by Vodafone (and acts as a bridge), and it's standard-Ethernet from there in, so there may be more choice.

I've not gone looking for specific models that CoroWRT has focused on in NZ (except for looking for WNDR3800 just now, and not immediately finding any obviously for sale -- but IIRC the model is getting harder to find everywhere now).

Ewen

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Dave Täht thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Joel Wirāmu Pauling

6:29 a.m.

Most of them are on Fibre fed ethernet switched networks, so fq_codel without SQM is generally what I keep them on without tweaking SQM as it doesn't help on 100/100mbit+ I have a few behind 130/25mbit Cable connections and generally just use Simple QOS with around 2-5% bellow observed peak rates out of the wan interface after loading the connection up to NZ servers.Generally Telstra/Voda's Cable scheduling (Docsis3 Cisco's) is I think single Q buffer. Tuning SQM again here isn't a big help. fq_codel by itself is enough to keep latency spikes lower than default pfifo generally found on most Linux based CPE's out there and give measurable user happiness gains. One area I want to have a play with is multiqueue SQM/fq_codel performance at 100/1000mbit + on Intel Atom. As per discussions on the cerowrt list the Mip's stuff really is limited once you start looking at Gig uplinks which are carrying VXLAN traffic to the edge (SDN CPE) - as I have a work related interest in this at the moment. -Joel On 26 January 2015 at 22:17, Dave Taht wrote:

...

On Tue, Jan 27, 2015 at 6:45 PM, Joel Wirāmu Pauling wrote:

...
WNDR3800 is Telepermit/Sold in NZ I have dozens of them around the country... many running cerowrt builds.

Heh. I didn't know. Can you share some of your typical SQM settings for various services and bandwidths? In particular the DSL compensation code was always kind of hairy.

I'd also like to note that ubnt's gear was a frequent target for the cerowrt effort, notably the pico and nanostations are extensively in play in the largest testbed.

I think highly of their default firmware, but it lacks ipv6, routing support, and bufferbloat fixes. Their AirOS qos system is quite good, being fq-based, but didn't have aqm-ish facilities, and so far as I know (in their AirFiber product) they just poured the fq portion of all that into the FPGA, and didn't do much to manage queue length.

A netperf-wrapper rrul (latency with load) result on one of the airfiber boxes in rain and outside of it would be interesting.

...
The default PSU is DC 110/240 Switchmode, a physical adaptor is all that

is

...
required.

Have no brick here at all. oops.

...
-Joel

On 26 January 2015 at 21:08, Ewen McNeill

...
...
Hi Dave,

On 27/01/15 16:45, Dave Taht wrote:

...
Well, I would love it if I could get more data from folk willing to install netperf-wrapper and run a 5 minute test that would be good. [...]

on debian derived linux those are:

In case it helps someone else trying to test this week...

For "Debian derived Linux" read "Ubuntu" AFAICT (netperf is not packaged in Debian, up through Debian Experimental, so "apt-get build-dep

netperf"

...
won't work there; I can send a

...
...
list in case someone wants to try on Debian).

It's also apparently _not_ Ubuntu 14.04 LTS, because that appears to install netperf-wrapper as a Python Egg with the given install instructions, and netperf-wrapper appears to be completely unable to find its tests within the Python Egg, resulting in it complaining:

-=- cut here -=- Fatal error: Hostname lookup failed for host rrul: [Errno -2] Name or service not known -=- cut here -=-

You can tell you have this problem if "netperf-wrapper --list-tests" fails with a Python stack trace, rather than returning a list of tests.

My kludgy workaround (which seems to have worked) was:

-=- cut here -=- cd /usr/local/share && sudo ln -s ~/src/netperf-wrapper . -=- cut here -=-

which then lets "nztest.sh" (http://snapon.lab.bufferbloat.net/~d/nz/nztest.sh) run after some editing to point it at something other than the non-existent NZ server (I chose

wrote: packages-my-Ubuntu-system-wanted-to-install the

...
...
US west coast).

Serious suggestion: perhaps it would help to bundle this with Docker or similar? It'd then be easier for people to install a known-to-work version with just a "needs modern Linux kernel" dependency.

...
In my present location, however, I am seeing a *450ms* RTT to the eu, which I sure hope isn't normal for NZ, just this hotel. ( ping netperf-eu.bufferbloat.net ).

It seems about 75-100ms too high. From Vodafone cable in Wellington:

-=- cut here -=- ewen(a)ashram:~$ ping netperf-eu.bufferbloat.net PING kau.toke.dk (130.243.26.64): 56 data bytes 64 bytes from 130.243.26.64: icmp_seq=0 ttl=39 time=341.222 ms 64 bytes from 130.243.26.64: icmp_seq=1 ttl=39 time=342.191 ms [...] -=- cut here -=-

and it's about 15ms lower (ie, around 325ms) from my colo box in central Wellington (which is roughly the cable Internet overhead, so an expected difference).

320-350ms is roughly the RTT I'd expect to see to Europe out of New Zealand, mostly due to speed-of-light not being infinite. (I do work on a few systems based in Europe so have fairly consistently seen this for years.)

...
If anyone has a power supply suitable for a wndr3800 and NZ power standards, I brought one, might be able to fix matters here.....

A quick online search suggests this is a 12V DC, 2.5A power supply (eg,

http://www.myaccount.charter.com/customers/support.aspx?supportarticleid=329... ).

...
If so, they seem likely to be fairly common.

...
Sort of why I just asked if openwrt derived gear was a problem here. That stuff is the farthest along by far, for home cpe.

The main issue here for DSL is that there is an approval process ("Telepermit") for legally connecting equipment to NZ copper lines, so only models that someone has put through the approval process can be legally used (and IIRC each importer has to have their own import approved).

For Vodafone Cable the CPE device is usually supplied by Vodafone (and acts as a bridge), and it's standard-Ethernet from there in, so there may be more choice.

I've not gone looking for specific models that CoroWRT has focused on in NZ (except for looking for WNDR3800 just now, and not immediately finding any obviously for sale -- but IIRC the model is getting harder to find everywhere now).

Ewen

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Dave Taht

6:01 a.m.

On Tue, Jan 27, 2015 at 6:08 PM, Ewen McNeill wrote:

...

Hi Dave,

On 27/01/15 16:45, Dave Taht wrote:

...
Well, I would love it if I could get more data from folk willing to install netperf-wrapper and run a 5 minute test that would be good. [...]

on debian derived linux those are:

In case it helps someone else trying to test this week...

Given the high level of enthusiasm displayed I wish I'd made this request last week....

...

For "Debian derived Linux" read "Ubuntu" AFAICT (netperf is not packaged in Debian, up through Debian Experimental, so "apt-get build-dep netperf" won't work there; I can send a packages-my-Ubuntu-system-wanted-to-install list in case someone wants to try on Debian).

It's also apparently _not_ Ubuntu 14.04 LTS, because that appears to install netperf-wrapper as a Python Egg with the given install instructions, and netperf-wrapper appears to be completely unable to find its tests within the Python Egg, resulting in it complaining:

There is also a bug with that version's matplotlib - not a problem for the test itself, but for generating the charts.

...

-=- cut here -=- Fatal error: Hostname lookup failed for host rrul: [Errno -2] Name or service not known -=- cut here -=-

You can tell you have this problem if "netperf-wrapper --list-tests" fails with a Python stack trace, rather than returning a list of tests.

My kludgy workaround (which seems to have worked) was:

-=- cut here -=- cd /usr/local/share && sudo ln -s ~/src/netperf-wrapper . -=- cut here -=-

which then lets "nztest.sh" (http://snapon.lab.bufferbloat.net/~d/nz/nztest.sh) run after some editing to point it at something other than the non-existent NZ server (I chose the US west coast).

I imagine the python libs are stored in /usr/local/share....

...

Serious suggestion: perhaps it would help to bundle this with Docker or similar? It'd then be easier for people to install a known-to-work version with just a "needs modern Linux kernel" dependency.

When we started this project, we couldn't trust the devices, the device drivers, the qdiscs, the tcps, or the oses, and adding a vm on top of that tended to compound things. It took a really long time to get to the roots of the problems and while all that is mostly fixed on modern linuxes.... To this day, many co-location facilities are still running ancient linuxes on their bare hardware, or hypervisors with behavior that is not well understood, with invisible workloads elsewhere in the system. It's really hard to trust those results, except when the devices you are testing are running at very low rates relative to the host you are testing from. Docker hadn't appeared as a force, when we started, either. Certainly containerization is saner than vms, but when you are testing for a problem that goes all the way down to the raw hardware, having less abstractions is of a help. :/ That said, a docker version of the tool would be fabulous to have around and we'd welcome to contribution of an image easier for more people to install! Before it is asked, we didn't do a web bandwidth measurement tool (like speedof.me or speedtest) because we found universally that they were inaccurate above about 20Mbits, as were java based things like netalyzer. Instead, we did netperf-wrapper, which has been now proven to scale to 40GigE, and scales to a gbit even on fairly low end hardware. Netperf results are trusted by the kernel developers also, unlike iperf. That said, one day, we'll try to do up a tool that works on every platform, runs fast, and can do the kind of analytics you can do that the netperf-wrapper --gui can do - the analytical and graphical portions of the engine can deal with any form of periodic sampling of a flow, from any tool, and there are wrappers now for things like d-itg, and iperf in it, so it's increasingly misnamed as a tool. I agree strongly that we need to make the core tests easier to install! But it was more important to get back accurate data at least, at first. In the interim, the pain of installing it is incurred once, and the benefits last. One of my favorite things to do is compare a speedtest result to a netperf-wrapper one, in front of someone dubious about bufferbloat....

...

...
In my present location, however, I am seeing a *450ms* RTT to the eu, which I sure hope isn't normal for NZ, just this hotel. ( ping netperf-eu.bufferbloat.net ).

It seems about 75-100ms too high. From Vodafone cable in Wellington:

-=- cut here -=- ewen(a)ashram:~$ ping netperf-eu.bufferbloat.net PING kau.toke.dk (130.243.26.64): 56 data bytes 64 bytes from 130.243.26.64: icmp_seq=0 ttl=39 time=341.222 ms 64 bytes from 130.243.26.64: icmp_seq=1 ttl=39 time=342.191 ms [...] -=- cut here -=-

and it's about 15ms lower (ie, around 325ms) from my colo box in central Wellington (which is roughly the cable Internet overhead, so an expected difference).

320-350ms is roughly the RTT I'd expect to see to Europe out of New Zealand, mostly due to speed-of-light not being infinite. (I do work on a few systems based in Europe so have fairly consistently seen this for years.)

...
If anyone has a power supply suitable for a wndr3800 and NZ power standards, I brought one, might be able to fix matters here.....

A quick online search suggests this is a 12V DC, 2.5A power supply (eg, http://www.myaccount.charter.com/customers/support.aspx?supportarticleid=329...). If so, they seem likely to be fairly common.

common as in a radio shack or equivalent in roturua?

...

...
Sort of why I just asked if openwrt derived gear was a problem here. That stuff is the farthest along by far, for home cpe.

The main issue here for DSL is that there is an approval process ("Telepermit") for legally connecting equipment to NZ copper lines, so only models that someone has put through the approval process can be legally used (and IIRC each importer has to have their own import approved).

The only fully BQL'd and fq_codeled DSL device I have used at is the transcend ( can't quite remember the name?) dsl box made in australia. It was built around a geode, and dave woodhouse fixed their driver, but good. There are undoubtably others at this point. Free.fr uses their own custom DSL firmware for their revolution V6 boxes.

...

For Vodafone Cable the CPE device is usually supplied by Vodafone (and acts as a bridge), and it's standard-Ethernet from there in, so there may be more choice.

What are the typical rates sold? This is the typical result of (comcast) cable in the USA - see image 2 for the "normal" state of cable there: http://snapon.lab.bufferbloat.net/~cero2/jimreisert/results.html and this is a more extensive set of tests and tunings showing how to address up and down bandwidth issues, using the rrul test as a guide. http://burntchrome.blogspot.co.nz/2014_05_01_archive.html

...

I've not gone looking for specific models that CoroWRT has focused on in NZ (except for looking for WNDR3800 just now, and not immediately finding any obviously for sale -- but IIRC the model is getting harder to find everywhere now).

Well, I had planned to donate mine to someone here that needs it... except that I didn't bring a power supply suitable.... And everything important that was in cerowrt is in openwrt chaos calmer, and any of the thousands of platforms openwrt supports could be used here. Also, the latest edgerouter software has fq_codel, as do several high end home routers like the netgear X4 and everything with streamboost. We have been trying to coax mikrotik to pay attention... Now, having given all this info out, can I just skip the talk and drink beer? :)

...

Ewen

-- Dave Täht thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Joel Wirāmu Pauling

5:48 a.m.

On 26 January 2015 at 19:45, Dave Taht wrote:

...

On Mon, Jan 26, 2015 at 8:46 AM, Ewen McNeill wrote:

...
Hi Dave,

On 26/01/15 7:45, Dave Taht wrote:

In my present location, however, I am seeing a *450ms* RTT to the eu, which I sure hope isn't normal for NZ, just this hotel. ( ping netperf-eu.bufferbloat.net ).

dair-833:~ d$ ping netperf-eu.bufferbloat.net

PING kau.toke.dk (130.243.26.64): 56 data bytes

64 bytes from 130.243.26.64: icmp_seq=0 ttl=42 time=409.153 ms 64 bytes from 130.243.26.64: icmp_seq=1 ttl=42 time=432.350 ms 64 bytes from 130.243.26.64: icmp_seq=2 ttl=42 time=455.123 ms

Unfortunately those numbers look pretty standard for a Hotel connection. I am happy if I get ~330ms to anything in Europe from NZ even on fat Carrier/ISP pipes. -Joel

3721

Age (days ago)

3722

Last active (days ago)

List overview

Download

12 comments

4 participants

participants (4)

Dave Taht
Ewen McNeill
Joel Wirāmu Pauling
Jonathon Exley