After the discussion a few weeks back about DNS performance, I asked one of my colleagues, Brendon Jones to add DNS performance to the gTLD/Root servers to our Active Measurement Platform (AMP) which is already monitoring the .nz nameservers. These have now had a while to collect some data and show a fairly interesting (and IMHO pretty visualisation of New Zealands DNS performance) * http://erg.cs.waikato.ac.nz/amp/matrix.php/latency/NZ/NZ+DNS For starters, we've in the past measured performance to the .nz ccTLD name servers to track their performance within New Zealand. This shows a pretty healthy coverage for .nz. Full marks to all the people who have done the hard work to make this happen. * http://erg.cs.waikato.ac.nz/amp/matrix.php/hops/NZ/NZ+DNS This in comparison shows how many hops we see in a traceroute to the .nz ccTLD servers. All the New Zealand name servers are firewalled in such a way we can't get an accurate count, but this at least provides a lower bound. You can see people who don't peer at WIX don't see the near instance of ns7. * http://erg.cs.waikato.ac.nz/amp/matrix.php/latency/NZ/root+DNS * http://erg.cs.waikato.ac.nz/amp/matrix.php/hops/NZ/root+DNS Second up, we added a test to all of our measurement points to the Root Servers. This shows quite distinctively that there are several places in New Zealand whose peering policy means that they don't see some, or in the case of Otago Uni's CS Dept, any, New Zealand based instances. vuw interestingly doesn't appear to be able to contact any f.root instance at all. New Zealand seems to be fairly well covered with F, I, J and even a fairly close K root. * http://erg.cs.waikato.ac.nz/amp/matrix.php/latency/NZ/gtld+DNS * http://erg.cs.waikato.ac.nz/amp/matrix.php/hops/NZ/gtld+DNS This shows the same visualisation to all of the gTLD servers. This shows a much more unhappy view of New Zealand. Our monitoring points are quite biased towards universities which generally prefer KAREN, which has poor coverage (which appears to be due to KARENs policies) and thus show very poor numbers. However it doesn't paint a particularly rosy picture for much of the rest of New Zealand either, with Maxnet and TheLoop also failing to find any instances anywhere near New Zealand at all. * http://erg.cs.waikato.ac.nz/amp/matrix.php/latency/NZ/afilias+DNS * http://erg.cs.waikato.ac.nz/amp/matrix.php/hops/NZ/afilias+DNS Afilias provide nameserving for several zones including .org/.mobi and so on. Right this instant TelstraClear doesn't appear to be able to get to b0.org.afilias-nst.org http://erg.cs.waikato.ac.nz/amp/graph.php?src=NZ&dst=b0.org.afilias-nst.org at all, so again many of the universities show failures, although this time it doesn't appear to be routing issues with KAREN. Also, just as we were setting up collecting some test data (but unfortunately not traceroute data), KAREN coincidentally had a major outage in Hamilton which impacted the University of Waikato. This let us see what happens when KAREN's routes aren't available: (See? Unscheduled outages /can/ have an upside!) http://erg.cs.waikato.ac.nz/amp/graph.php?src=ampz-waikato&dst=b.root-servers.net&rge=1-day&date=2010-05-25 http://erg.cs.waikato.ac.nz/amp/graph.php?src=ampz-waikato&dst=e.root-servers.net&rge=1-day&date=2010-05-25 http://erg.cs.waikato.ac.nz/amp/graph.php?src=ampz-waikato&dst=j.root-servers.net&rge=1-day&date=2010-05-25 http://erg.cs.waikato.ac.nz/amp/graph.php?src=ampz-waikato&dst=k.root-servers.net&rge=1-day&date=2010-05-25 This shows that if we don't have KAREN routes available, then our performance to b, e, j and k root *improves*, Sigh. Also our performance to F root degrades as our commodity internet connection suddenly has to handle the additional load: http://erg.cs.waikato.ac.nz/amp/graph.php?src=ampz-waikato&dst=f.root-servers.net&rge=1-day&date=2010-05-25 So, all in all, New Zealand's DNS Performance is better than I had seen (my two measurement points inside Waikato University and Rurallink were two of the worst to choose from, Rurallink doesn't yet host an AMP node so doesn't appear here). Hopefully KAREN will in the future consider hosting/peering directly with at least a root server, and NZ ccTLD server so if an Universities commidity connection falls over then you can still resolve (and therefore create new connections to) other research institutions. KAREN could either start not accepting "scenic" routes from other R&E networks for other anycast instances of Root/gTLD/ccTLD servers, or provide access to them via less amusing routes by increasing their peering. People who don't peer at WIX miss out on the instances hosted there. If you're not peering, some of your customers are getting slower results for DNS lookups than necessary making web pages take longer, to load, and thus your service appear to be slower. Yet another reason to improve your peering. Ideas and comments welcomed!
Perry, On Sun, 2010-06-13 at 14:21 +1200, Perry Lorier wrote:
After the discussion a few weeks back about DNS performance, I asked one of my colleagues, Brendon Jones to add DNS performance to the gTLD/Root servers to our Active Measurement Platform (AMP) which is already monitoring the .nz nameservers. These have now had a while to collect some data and show a fairly interesting (and IMHO pretty visualisation of New Zealands DNS performance)
I note that TelstraClear and Telecom aren't listed in these statistics as sources. Given that these are major ISPs (and they don't seem to peer at the NZIX exchanges :), have you any way of gathering statistics for their connectivity to either the .nz nameservers or the root/gTLD nameservers? What about other ISPS which don't have affiliation to either the NZIX exchanges or to one of the universities (and who aren't peering with them)? [commentary about AMP results deleted]
Ideas and comments welcomed!
Tim
--
Tim Frost
Tim Frost wrote:
Perry,
On Sun, 2010-06-13 at 14:21 +1200, Perry Lorier wrote:
After the discussion a few weeks back about DNS performance, I asked one of my colleagues, Brendon Jones to add DNS performance to the gTLD/Root servers to our Active Measurement Platform (AMP) which is already monitoring the .nz nameservers. These have now had a while to collect some data and show a fairly interesting (and IMHO pretty visualisation of New Zealands DNS performance)
I note that TelstraClear and Telecom aren't listed in these statistics as sources. Given that these are major ISPs (and they don't seem to peer at the NZIX exchanges :), have you any way of gathering statistics for their connectivity to either the .nz nameservers or the root/gTLD nameservers?
If you look closely at the traceroutes from many of the Universities (eg Waikato) you'll see that they use Telstra Clear as their upstream, although this is obviously imperfect as they also generally have KAREN connectivity. Unfortunately I don't think we have any visibility into Telecom's network.
What about other ISPS which don't have affiliation to either the NZIX exchanges or to one of the universities (and who aren't peering with them)?
Many of the "nsX" amp nodes are hosted at various ISPs (if you hover over them they give you a description that might be more useful), for example the AMP node called ns4a is hosted by Orcon. We're always interested in hosting more AMP boxes around the Internet (and particularly in New Zealand) to broaden what problems can be diagnosed by the AMP pages. For a variety of reasons, we provide the hardware and maintain the software build on the machines. Unfortunately we're funded by various grants, none of which at this moment cover buying new AMP machines. (But if people are willing to help with the costs, I'm sure we can arrange something). We generally try and put AMP boxes with cooperating sponsors, usually at points in the network where high performance use is expected, so AMP currently deliberately doesn't measure what end user DSL customers are likely to see. If anyone at Telecom or TelstraClear wants to host an AMP box to help diagnose routing issues between ISPs then get in touch and we'll see what we can organise.
Hi Perry, What is involved and required to host an AMP node? If you could send me the details I can see what we can do on our end to locate on in a suitable part of the network. Regards Paul Tinson Senior Specialist Telecom NZ -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Perry Lorier Sent: Monday, 21 June 2010 4:55 p.m. To: Tim Frost Cc: NZNOG List Subject: Re: [nznog] New Zealand DNS Performance Tim Frost wrote:
Perry,
On Sun, 2010-06-13 at 14:21 +1200, Perry Lorier wrote:
After the discussion a few weeks back about DNS performance, I asked one of my colleagues, Brendon Jones to add DNS performance to the gTLD/Root servers to our Active Measurement Platform (AMP) which is already monitoring the .nz nameservers. These have now had a while to collect some data and show a fairly interesting (and IMHO pretty visualisation of New Zealands DNS performance)
I note that TelstraClear and Telecom aren't listed in these statistics as sources. Given that these are major ISPs (and they don't seem to peer at the NZIX exchanges :), have you any way of gathering statistics for their connectivity to either the .nz nameservers or the root/gTLD nameservers?
If you look closely at the traceroutes from many of the Universities (eg Waikato) you'll see that they use Telstra Clear as their upstream, although this is obviously imperfect as they also generally have KAREN connectivity. Unfortunately I don't think we have any visibility into Telecom's network.
What about other ISPS which don't have affiliation to either the NZIX exchanges or to one of the universities (and who aren't peering with them)?
Many of the "nsX" amp nodes are hosted at various ISPs (if you hover over them they give you a description that might be more useful), for example the AMP node called ns4a is hosted by Orcon. We're always interested in hosting more AMP boxes around the Internet (and particularly in New Zealand) to broaden what problems can be diagnosed by the AMP pages. For a variety of reasons, we provide the hardware and maintain the software build on the machines. Unfortunately we're funded by various grants, none of which at this moment cover buying new AMP machines. (But if people are willing to help with the costs, I'm sure we can arrange something). We generally try and put AMP boxes with cooperating sponsors, usually at points in the network where high performance use is expected, so AMP currently deliberately doesn't measure what end user DSL customers are likely to see. If anyone at Telecom or TelstraClear wants to host an AMP box to help diagnose routing issues between ISPs then get in touch and we'll see what we can organise. _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
Hi Paul, Unfortunately Perry is shortly leaving us and NZ so it would be best if you dealt with me regarding AMP. I'm replying on list just in case anyone else is vaguely interested. We use a standardised hardware platform for AMP monitors. They are supermicro 1U rackmount servers running debian linux. The AMP project is not currently funded and so we ask if potential hosts are able to pay the purchase cost of a machine. We require a static IP address (preferably v6 as well as v4) and a single ethernet connection (preferably gigabit - for MTU and latency reasons, not bandwidth). The connection is best if it is not behind any firewall, or at least a minimally intrusive firewall. The basic information we require is that in the form at http://wand.net.nz/amp/questions.php Last time we measured, most AMP monitors were had combined in and out traffic of between 15 and 25 MB per day although this will have gone up a little with an increase in the number of deployed monitors and the DNS testing. Apart from some of the DNS queries all the traffic should be to NZ destinations. The destinations are those listed on the AMP website - other AMP monitors, DNS servers, some high profile NZ websites and the management system here at Waikato. We manage and secure the machines from Waikato. Logins for local hosts are available. If there is anything else you need to know, just ask. Richard Nelson. On 22/06/10 09:23, Paul Tinson wrote:
Hi Perry,
What is involved and required to host an AMP node? If you could send me the details I can see what we can do on our end to locate on in a suitable part of the network.
Regards
Paul Tinson Senior Specialist
Telecom NZ
Covering the cost of a machine might be achievable depending on cost etc; does it have to be the same spec node? If not then I am sure we could provide a machine from a pool of available ones. I will start the conversation with those further up the chain than me that would give the yes/no on this as well. On list again incase we did end up running a different spec node so there is some traceable history of that as a deviation in a public forum:). Regards Paul Tinson Senior Specialist Telecom NZ ________________________________ This communication, including any attachments, is confidential. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002. -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Richard Nelson Sent: Tuesday, 22 June 2010 1:54 p.m. To: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] New Zealand DNS Performance Hi Paul, Unfortunately Perry is shortly leaving us and NZ so it would be best if you dealt with me regarding AMP. I'm replying on list just in case anyone else is vaguely interested. We use a standardised hardware platform for AMP monitors. They are supermicro 1U rackmount servers running debian linux. The AMP project is not currently funded and so we ask if potential hosts are able to pay the purchase cost of a machine. We require a static IP address (preferably v6 as well as v4) and a single ethernet connection (preferably gigabit - for MTU and latency reasons, not bandwidth). The connection is best if it is not behind any firewall, or at least a minimally intrusive firewall. The basic information we require is that in the form at http://wand.net.nz/amp/questions.php Last time we measured, most AMP monitors were had combined in and out traffic of between 15 and 25 MB per day although this will have gone up a little with an increase in the number of deployed monitors and the DNS testing. Apart from some of the DNS queries all the traffic should be to NZ destinations. The destinations are those listed on the AMP website - other AMP monitors, DNS servers, some high profile NZ websites and the management system here at Waikato. We manage and secure the machines from Waikato. Logins for local hosts are available. If there is anything else you need to know, just ask. Richard Nelson. On 22/06/10 09:23, Paul Tinson wrote:
Hi Perry,
What is involved and required to host an AMP node? If you could send me the details I can see what we can do on our end to locate on in a suitable part of the network.
Regards
Paul Tinson Senior Specialist
Telecom NZ
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
Bit of an off-topic request. Does anyone has any stats on Recursive DNS appliances (infoblox etc) vs Bind on a server? Has anyone actually seen real life improvements? From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Perry Lorier Sent: Sunday, 13 June 2010 2:22 p.m. To: NZNOG List Subject: [nznog] New Zealand DNS Performance After the discussion a few weeks back about DNS performance, I asked one of my colleagues, Brendon Jones to add DNS performance to the gTLD/Root servers to our Active Measurement Platform (AMP) which is already monitoring the .nz nameservers. These have now had a while to collect some data and show a fairly interesting (and IMHO pretty visualisation of New Zealands DNS performance) * http://erg.cs.waikato.ac.nz/amp/matrix.php/latency/NZ/NZ+DNS For starters, we've in the past measured performance to the .nz ccTLD name servers to track their performance within New Zealand. This shows a pretty healthy coverage for .nz. Full marks to all the people who have done the hard work to make this happen. * http://erg.cs.waikato.ac.nz/amp/matrix.php/hops/NZ/NZ+DNS This in comparison shows how many hops we see in a traceroute to the .nz ccTLD servers. All the New Zealand name servers are firewalled in such a way we can't get an accurate count, but this at least provides a lower bound. You can see people who don't peer at WIX don't see the near instance of ns7.. * http://erg.cs.waikato.ac.nz/amp/matrix.php/latency/NZ/root+DNS * http://erg.cs.waikato.ac.nz/amp/matrix.php/hops/NZ/root+DNS Second up, we added a test to all of our measurement points to the Root Servers. This shows quite distinctively that there are several places in New Zealand whose peering policy means that they don't see some, or in the case of Otago Uni's CS Dept, any, New Zealand based instances. vuw interestingly doesn't appear to be able to contact any f.root instance at all. New Zealand seems to be fairly well covered with F, I, J and even a fairly close K root. * http://erg.cs.waikato.ac.nz/amp/matrix.php/latency/NZ/gtld+DNS * http://erg.cs.waikato.ac.nz/amp/matrix.php/hops/NZ/gtld+DNS This shows the same visualisation to all of the gTLD servers. This shows a much more unhappy view of New Zealand. Our monitoring points are quite biased towards universities which generally prefer KAREN, which has poor coverage (which appears to be due to KARENs policies) and thus show very poor numbers. However it doesn't paint a particularly rosy picture for much of the rest of New Zealand either, with Maxnet and TheLoop also failing to find any instances anywhere near New Zealand at all. * http://erg.cs.waikato.ac.nz/amp/matrix.php/latency/NZ/afilias+DNS * http://erg.cs.waikato.ac.nz/amp/matrix.php/hops/NZ/afilias+DNS Afilias provide nameserving for several zones including .org/.mobi and so on. Right this instant TelstraClear doesn't appear to be able to get to b0.org.afilias-nst.orghttp://erg.cs.waikato.ac.nz/amp/graph.php?src=NZ&dst=b0.org.afilias-nst.org at all, so again many of the universities show failures, although this time it doesn't appear to be routing issues with KAREN. Also, just as we were setting up collecting some test data (but unfortunately not traceroute data), KAREN coincidentally had a major outage in Hamilton which impacted the University of Waikato. This let us see what happens when KAREN's routes aren't available: (See? Unscheduled outages /can/ have an upside!) http://erg.cs.waikato.ac.nz/amp/graph.php?src=ampz-waikato&dst=b.root-servers.net&rge=1-day&date=2010-05-25 http://erg.cs.waikato.ac.nz/amp/graph.php?src=ampz-waikato&dst=e.root-servers.net&rge=1-day&date=2010-05-25 http://erg.cs.waikato.ac.nz/amp/graph.php?src=ampz-waikato&dst=j.root-servers.net&rge=1-day&date=2010-05-25 http://erg.cs.waikato.ac.nz/amp/graph.php?src=ampz-waikato&dst=k.root-servers.net&rge=1-day&date=2010-05-25 This shows that if we don't have KAREN routes available, then our performance to b, e, j and k root *improves*, Sigh. Also our performance to F root degrades as our commodity internet connection suddenly has to handle the additional load: http://erg.cs.waikato.ac.nz/amp/graph.php?src=ampz-waikato&dst=f.root-servers.net&rge=1-day&date=2010-05-25 So, all in all, New Zealand's DNS Performance is better than I had seen (my two measurement points inside Waikato University and Rurallink were two of the worst to choose from, Rurallink doesn't yet host an AMP node so doesn't appear here). Hopefully KAREN will in the future consider hosting/peering directly with at least a root server, and NZ ccTLD server so if an Universities commidity connection falls over then you can still resolve (and therefore create new connections to) other research institutions. KAREN could either start not accepting "scenic" routes from other R&E networks for other anycast instances of Root/gTLD/ccTLD servers, or provide access to them via less amusing routes by increasing their peering. People who don't peer at WIX miss out on the instances hosted there. If you're not peering, some of your customers are getting slower results for DNS lookups than necessary making web pages take longer, to load, and thus your service appear to be slower. Yet another reason to improve your peering. Ideas and comments welcomed!
Simon Allard wrote:
Bit of an off-topic request.
Does anyone has any stats on Recursive DNS appliances (infoblox etc) vs Bind on a server?
Has anyone actually seen real life improvements?
I spent some time a while back looking at various (open source) recursive DNS servers. Unbound was nice mostly because it was easily tunable to get decent performance out of it. Google's Namebench[1] is quite good at showing how your DNS servers perform. The basic thing appears to be that you need /really/ warm caches to get decent performance, especially in New Zealand. After replacing/tweaking/tuning some nameservers, we saw it take well over a week before it was getting reasonable perf results again. I suspect (although I've not done any work into this) that the one thing that will improve your nameserver performance is to have a huge cache and prefetch popular cache entries to avoid users ever seeing the cache miss latency. This I suspect will dwarf any other speed differences between any name server implementation. [1]: http://code.google.com/p/namebench/
We (well, Brendon) has fixed some bugs, tidied things up and added some more AMP graphs if people are interested: * http://erg.wand.net.nz/amp/matrix.php/latency/NZ/RIR+DNS * http://erg.wand.net.nz/amp/matrix.php/hops/NZ/RIR+DNS We've added the nameservers that serve the APNIC reverse zones (as far as we can tell). These normally aren't hit when you fetch a webpage (being reverse zones), but some services (eg ssh) may look up your hostname before allowing you to connect for logging/hostname-authentication purposes. Compared to the root/forward zone name servers, this is quite a different looking picture. Also, we're starting to do MTU tests to some places (mostly on KAREN): * http://erg.wand.net.nz/amp/matrix.php/mtu/Any/NZ Which is quite interesting. The symbol beside the number showing that pMTU discovery failed is probably important for people to investigate and get fixed. Again, if anyone has any thoughts, comments or suggestions we're keen to hear from you.
participants (5)
-
Paul Tinson -
Perry Lorier -
Richard Nelson -
Simon Allard -
Tim Frost