On Sun, May 24, 2015 at 01:55:09PM +1200, Nathan Ward wrote:
Hi, Ia**m still researching exactly how it works, and of course there doesna**t seem to be much documentation publicly available and it sounds like ita**s difficult to get information by asking directly, but,
I asked a few questions back when it was new.
empirical evidence and reading their website suggests that theya**re judging how a**gooda** an Internet connection is by downloading a bunch of files from a few servers under different conditions (i.e. some files avoid caches, some dona**t, etc.).
Benchmarking web performance is complex. I used to be of the belief that testing mostly cached performance is a good test. But now the majority of "slow sites" are using https, and a lot of slow sites actually to be a problem with the cdn or end destination.
This is bad, because there is little in common between downloading a file from a server, and actual user experience of say browsing the web, sending email, or playing games. Despite this, many networks in NZ have large
Right. I did some of my own testing years back, and from my own testing it seemed to be that there two central isuses. One was most web sites require doing a lot of if-modified-since requests, and with high latency, as there are too many to do in one round trip time, and loss can make a /huge/ difference to performance, as not only does it delay that request, but subsequent requests. (also if there's loss you could have to wait 3 seconds for retransmit.. Linux has changed to 1 second as default now, which should make things better - normally it's sped up by acking subseqent packets but IMS requests don't trigger that) Not only do they seem to not test to enough international destinations, but they don't parallelise requests like a browser does. And that is a bit complicated to implement.
amounts of engineering work driven by TrueNet, because thata**s the current standard that ISPs judge themselves by against the competition.
It does seem to be good at testing handover congestion.
While Ia**m not aware of any network having specific config in place for TrueNet, its results drive investment and change in areas that are perhaps not really that important.
Handover congestion impacts all traffic.
One particular area of concern that I have, is that the servers they download these test files from are sometimes not in a particularly good network position. Theya**re not testing a**Interneta** performance, theya**re testing performance downloading some files from a particular network location. They used to have some files on TradeMe for this, and
Getting a good representive spread of network locations is hard.
that was perhaps a little better, but that is, I understand, no longer the case. I should say, TradeMe was a slightly better test of user experience because it was a reasonable thing for networks to be optimised for. It is no better in terms of testing against a wider range of targets, and it doesna**t actually test real user experience.
I think Amazon and Ebay would be better tests myself. :)
Another area of concern is the whole a**cacheda** thing, which furthers the outdated notion that web caching is a always good idea. Ia**m not saying that it isna**t, every network is different, but it should be up to the network operator to decide whether caching improves their customer experience/costs/etc. and in which parts of their network. Because TrueNet
I hate to say it, but I've always been really keen on caching of resources, but the vast majority of bulky content is moving towards cdn's etc, and forcing to not cache. Even things like Netflix which uses http disables caching.
dona**t test customer experience, rather they test the performance of a**cacheda** data, providers who have decided that caching isna**t for them are penalised.
Whatever you do you are going to favour some setups, and disfavour others. The biggest concern I've had is that they seem to favour burstable single threaded connections. When the majority of slowness is often on more active connections with more simultaneous access where AQM can really help.
They also *seem* to only provide data about providers who have paid them money, though obviously I cana**t back that up. If it is true, it is strange given that ComCom pay them as well. This is not technical though so isna**t really appropriate for this list, but I figured Ia**d mention it. If anyone can back that up, email me.
I think it's actually just providers that they have enough probes in the location of. I wouldn't say Truenet is malicious, I just think it's a bit simplistic.
I have more thoughts on this, but theya**re not yet well formed.. I intend to write up something along these lines in the next couple of weeks. If other people have data/thoughts that would be useful, email me.
In some ways i think it's best to keep on list? A bit of a ramble.. Ok my concerns with Truenet started when they were comparing "advertised speed" with cable and "maximum attained speed" with ADSL. And so Cable by giving burst over their advertised speed often gave results above 100%. In my own experience with cable, there often seemed to be a lot of packet loss on active connections. And my experience of using cable was that it was often slower than ADSL for normal web browsing, downloads, etc. But that it went fine with threaded downloads. I think there were two factors with Cable performance being bad. One was that the TCP/IP stack used on the transparent proxies gave lower performance than direct TCP connections from Linux to Linux, which seemed to hurt Europe performance quite noticably. As well as that normal TCP/IP connections seemed to get loss where UDP was fine. This may have been to do with the spikiness of TCP/IP where UDP is constant rate. But the end result was that TCP/IP would struggle to do even 20 megabit when UDP would do 20 megabit with 0% loss. The problem with web sites is that usually there isn't enough time to rapidly increase window sizes, even with TCP cubic fast enough to give 20 megabit/sec+ on small files. And on medium files any loss can cut out the ramping up, and force retransmits, giving a noticable hit. That is somewhere that a transparent proxy can really help with, as if the retransmits can be cut out, you can get 25%+ speed boost on medium sized files, even with no caching. The second was there seemed to be often poor DNS performance. Which is really hard to test, as you if you benchmark the same destinations it will give HIT performance, when MISS is what matters. It may have changed now, but it seemed very common in the past to use 203.96.152.4, and 203.96.152.12, hard coded, and for some reason or other, performance seemed to be significantly worse than ADSL ISP's. Which brings me to, what are you trying to test? Are you more interested in peak performance or worst case performance or median performance; testing popular sites, testing close sites, testing a specific representation, or a representation that covers multiple regions or what. There are a few solutions out there to test existing connections. I used to use this Chrome benchmarking plugin, which requires manually running, and has stopped working. There's namebench to test DNS that can go through your browser history. But as far as real-world like testing, there isn't a lot of automated stuff that can easily be setup. A good test to my mind would be to emulate browsers, run all the time regardless of other traffic on the link. Measure "pass/fail" rather than performance - say if a web site loads in 4 seconds that's fail, if it loads in 2 seconds pass. Measure availability. If a connection can't play Youtube videos 5% of the time, just call that 5% of time a fail, as some users will shift their viewing times, others will let videos buffer, and otherws will change providers. And then do you want to test large files or only medium sized files? If I'm downloading a few GB then I I am unlikely to watch it download. But for a 10mb file then I'd be waiting for it to download, so if it took 2 seconds, 5 seconds or 10 seconds it would matter to me. At the same time, if I upload a file of a few GB and I can't use the connection while it uploads then it's much more inconvenient. I realise the reason that Truenet want to wait for an idle connection is that they want reproducability - but do you really care if your internet is fast or slow while you're not using it? And I think that since Truenet has begun handover congestion has become less of an issue. And in general international performance isn't usually an issue. And if a web site is going slow, now days, I think it's often the destination, or a congestion issue internationally. So instead of "all international is slow" things have shifted to issues with Comcast, Verizon, AT&T, etc.. And a lot of these issues can even be region based in the US... like issues to Comcast in San Jose, and it can be the other end, or the connection within the US to them. And it's pretty well known now that there are a lot of peering issues in the US. Some places are worse than others, and for instance Kansas can be kind of hit and miss, but hardly any web sites are hosted there that New Zealanders would go to. Also there are things like a lot of CDN's being hosted in Asia - and connections to Japan can be routed via Australia or the US. Singapore is often routed via the US. Countries like China and Singapore the latency can get pretty bad routing via the US, but Japan often doesn't matter nearly as much. And if these CDN's are pull caches and pull from the US, and you go NZ -> US -> Singapore, then the CDN pulls from the US, there can be a significant decrease in performance compared to hitting a US CDN. So yeah, Truenet measures handover congestion fine, but to do "better" testing would require a lot of concerted effort. And I really think it's the kind of thing that has a global audience, and really would need test nodes in other countries as well, that can do things like VOIP emulation between nodes. Ben.