I'm doing a bit of DNS testing and I start noticing some weird results for a few domains. The two below appear to be specific problems since they are popular and I noticed them but I suspect there are others. The problem is that www.anz.co.nz and www.anz.com have TTLs of zero seconds. This means that *every* DNS look up for them takes either around 40 or 140ms (one server is closer/quicker than the other) since it is never cached. Similarly the TTL for www.trademe.co.nz (and www.oldfriends.co.nz) is just ten seconds so this must be constantly rechecked. Not as bad since their servers are in NZ but still there is a delay. Some companies (like google or yahoo) have very sophisticated systems that constantly check user performance and switch them from datacenter to datacenter in seconds if things start going slow. However unless your company has such a system (and very reliable and nearly DNS servers) then a TTL of a few minutes is good enough for manual updates to quickly propagate. Lower values than that will result in decreased performance for your customers. Even google and yahoo have TTLs of a minute or two. The scary thing is that both sites probably put a lot of time into making the actual pages load as fast as possible. -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
I've recently seen something like this myself, using DNS for DR, so if one server goes down the A record is modified right away so there is minimal downtime, to me this is a bad way of setting up DR. The problem I've seen is that some DNS servers don't read the 0 value and stick a default TTL of about an hr onto the zone, is this common? Regards Barry
I'm doing a bit of DNS testing and I start noticing some weird results for a few domains. The two below appear to be specific problems since they are popular and I noticed them but I suspect there are others.
The problem is that www.anz.co.nz and www.anz.com have TTLs of zero seconds. This means that *every* DNS look up for them takes either around 40 or 140ms (one server is closer/quicker than the other) since it is never cached.
Similarly the TTL for www.trademe.co.nz (and www.oldfriends.co.nz) is just ten seconds so this must be constantly rechecked. Not as bad since their servers are in NZ but still there is a delay.
Some companies (like google or yahoo) have very sophisticated systems that constantly check user performance and switch them from datacenter to datacenter in seconds if things start going slow.
However unless your company has such a system (and very reliable and nearly DNS servers) then a TTL of a few minutes is good enough for manual updates to quickly propagate. Lower values than that will result in decreased performance for your customers. Even google and yahoo have TTLs of a minute or two.
The scary thing is that both sites probably put a lot of time into making the actual pages load as fast as possible.
-- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On 13/03/2007 11:54 a.m., Barry Murphy wrote:
The problem I've seen is that some DNS servers don't read the 0 value and stick a default TTL of about an hr onto the zone, is this common?
Possibly not as common as Windows ignoring TTL altogether and opting for 30 minutes. Wouldn't some form of route changing via BGP or whatever be preferable? Gerard
Gerard Creamer wrote:
Wouldn't some form of route changing via BGP or whatever be preferable? Yep - but...
Some people are still scared of BGP. A few people arn't actually running BGP (or any dynamic protocol) to their upstreams... Different people want to fail over to different upstreams, both of which are statically announcing their netblocks... Other people want to protect against a single host failure rather than an entire routed block failure... And you can see why we get broken DNS implementations. Please people, think of the children. BGP is like the big friendly giant. Learn to use it to solve the task that it was designed to do, and load balancing for the tasks that it wasn't =) Dean
On Tue, 2007-03-13 at 14:02 +1300, Dean Pemberton wrote:
Gerard Creamer wrote:
Wouldn't some form of route changing via BGP or whatever be preferable? Yep - but...
Some people are still scared of BGP. A few people arn't actually running BGP (or any dynamic protocol) to their upstreams... Different people want to fail over to different upstreams, both of which are statically announcing their netblocks... Other people want to protect against a single host failure rather than an entire routed block failure...
And you can see why we get broken DNS implementations.
Please people, think of the children. BGP is like the big friendly giant.
Ah, thanks Dean. The Big Giant Protocol. Ho ho ho. jamie
<chooses words carefully> I can remember a time when a similar question was raised about zero time TTLs on records. It was with a bank. When they were approached as to why they did this, they replied with (paraphrasing here) "We want to make sure that each and every request for an IP address comes to our DNS server. We don't want it to be cached at all as this can lead to someone hijacking the cache. We like it this way" Now I'm not going to get into defending this point of view, I just thought that as I'd heard the reply before, I'd mention it here. Dean Simon Lyall wrote:
I'm doing a bit of DNS testing and I start noticing some weird results for a few domains. The two below appear to be specific problems since they are popular and I noticed them but I suspect there are others.
The problem is that www.anz.co.nz and www.anz.com have TTLs of zero seconds. This means that *every* DNS look up for them takes either around 40 or 140ms (one server is closer/quicker than the other) since it is never cached.
Similarly the TTL for www.trademe.co.nz (and www.oldfriends.co.nz) is just ten seconds so this must be constantly rechecked. Not as bad since their servers are in NZ but still there is a delay.
Some companies (like google or yahoo) have very sophisticated systems that constantly check user performance and switch them from datacenter to datacenter in seconds if things start going slow.
However unless your company has such a system (and very reliable and nearly DNS servers) then a TTL of a few minutes is good enough for manual updates to quickly propagate. Lower values than that will result in decreased performance for your customers. Even google and yahoo have TTLs of a minute or two.
The scary thing is that both sites probably put a lot of time into making the actual pages load as fast as possible.
On 13/03/2007, at 11:01 AM, Simon Lyall wrote:
I'm doing a bit of DNS testing and I start noticing some weird results for a few domains. The two below appear to be specific problems since they are popular and I noticed them but I suspect there are others.
The problem is that www.anz.co.nz and www.anz.com have TTLs of zero seconds. This means that *every* DNS look up for them takes either around 40 or 140ms (one server is closer/quicker than the other) since it is never cached.
Similarly the TTL for www.trademe.co.nz (and www.oldfriends.co.nz) is just ten seconds so this must be constantly rechecked. Not as bad since their servers are in NZ but still there is a delay.
Some companies (like google or yahoo) have very sophisticated systems that constantly check user performance and switch them from datacenter to datacenter in seconds if things start going slow.
However unless your company has such a system (and very reliable and nearly DNS servers) then a TTL of a few minutes is good enough for manual updates to quickly propagate. Lower values than that will result in decreased performance for your customers. Even google and yahoo have TTLs of a minute or two.
That's not uncommon these days. Things like Foundry GSLB (Global Server Load Balancing) and Cisco DistributedDirector sit in front of DNS servers (or are DNS servers in some cases), and return records with low (or zero) TTLs. This is configurable with Foundry (default 10s), I'm not sure about Cisco and others. I believe TradeMe have some Foundry kit, so it's not inconceivable that they have GSLB turned on. I did read somewhere recently that zero-second TTLs have a bad effect on some clients resolvers, something like they instead use their 'default' which is 15 minutes in many cases. Google doesn't love me today though, so I can't give you references to it. -- Nathan Ward
On 3/13/07, Nathan Ward
That's not uncommon these days. Things like Foundry GSLB (Global Server Load Balancing) and Cisco DistributedDirector sit in front of DNS servers (or are DNS servers in some cases), and return records with low (or zero) TTLs. This is configurable with Foundry (default 10s), I'm not sure about Cisco and others.
I believe TradeMe have some Foundry kit, so it's not inconceivable that they have GSLB turned on.
I did read somewhere recently that zero-second TTLs have a bad effect on some clients resolvers, something like they instead use their 'default' which is 15 minutes in many cases. Google doesn't love me today though, so I can't give you references to it.
DNS looks like a nice thing to use in DR or load balancing situations, however, in most cases it just doesn't work. I've seen a lot of ISPs that cache all DNS entries, no matter what the TTL value says. Using a TTL of 10 seconds, like what Trademe has, doesn't guarantee that, if you change the IP address, it will take only 10 seconds for clients. For example, Internet Explorer will cache the DNS entry for 30 minutes, no matter what was in the TTL (previous versions, <3.0 cached it for 24 hours!). You can change this with a registry key but I doubt anyone does that. Bojan
On 13/03/2007, at 12:22 PM, Bojan Zdrnja wrote:
I've seen a lot of ISPs that cache all DNS entries, no matter what the TTL value says.
Can you back that up? We've seen numerous claims like that over the years, with little/no backing evidence.
Using a TTL of 10 seconds, like what Trademe has, doesn't guarantee that, if you change the IP address, it will take only 10 seconds for clients. For example, Internet Explorer will cache the DNS entry for 30 minutes, no matter what was in the TTL (previous versions, <3.0 cached it for 24 hours!). You can change this with a registry key but I doubt anyone does that.
http://support.microsoft.com/kb/263558 confirms that. Do you have any details on IE 7? This document was written 27/01/07, but doesn't mention IE7. Though, using a lower TTL here helps a bit, because the client's ISP's recursive resolver won't cache it for too long, meaning switchover is 30 minutes + 10s max, not 30 minutes + $higher_ttl. -- Nathan Ward
Using a TTL of 10 seconds, like what Trademe has, doesn't guarantee that, if you change the IP address, it will take only 10 seconds for clients. For example, Internet Explorer will cache the DNS entry for 30 minutes, no matter what was in the TTL (previous versions, <3.0 cached it for 24 hours!). You can change this with a registry key but I doubt anyone does that.
http://support.microsoft.com/kb/263558 confirms that. Do you have any details on IE 7? This document was written 27/01/07, but doesn't mention IE7.
Though, using a lower TTL here helps a bit, because the client's ISP's recursive resolver won't cache it for too long, meaning switchover is 30 minutes + 10s max, not 30 minutes + $higher_ttl.
code based on the mozilla codebase (firefox, thunderbird, seamonkey etc) all appear to by default cache items for a minimum of 1 minute. This means that they can at least load all the files associated with a single page load with one DNS query rather than performing one lookup for each stylesheet, image, etc... Using 0 as a TTL has a bug in some resolvers[1] that the entry will be expired before its used thus generating an infinite stream of DNS queries. Other implementations use 0 or -1 for "never expire" for entries, which may cause the exact wrong behavior. This is perhaps not very widespread in webbrowsers, or recursive nameservers so it may not be an issue for www sites. Some Bind versions will use the minimum ttl listed in the SOA if the TTL of a record is given as 0. [rfc1536 section 5]. Presumably it only knows the SOA when it's the primary or the secondary nameserver. [1] mumbleoldircmumble.
On 3/13/07, Nathan Ward
On 13/03/2007, at 12:22 PM, Bojan Zdrnja wrote:
I've seen a lot of ISPs that cache all DNS entries, no matter what the TTL value says.
Can you back that up? We've seen numerous claims like that over the years, with little/no backing evidence.
I worked with some broken DNS resolves at my old University that did the same thing. My current ISP(s) do the things properly. There was a discussion about this somewhere else recently, I'll see what I can dig out.
http://support.microsoft.com/kb/263558 confirms that. Do you have any details on IE 7? This document was written 27/01/07, but doesn't mention IE7.
Yeah, I'm not sure what IE7 does but I would expect a similar thing. One other thing - I haven't tested this, but I saw on various places where people claimed that Internet Explorer doesn't cache A records (at all), but only CNAME records. This can be tested easily if someone has a bit of free time.
Though, using a lower TTL here helps a bit, because the client's ISP's recursive resolver won't cache it for too long, meaning switchover is 30 minutes + 10s max, not 30 minutes + $higher_ttl.
Definitely. Cheers, Bojan
On Tue, 13 Mar 2007, Simon Lyall wrote:
Similarly the TTL for www.trademe.co.nz (and www.oldfriends.co.nz) is just ten seconds so this must be constantly rechecked. Not as bad since their servers are in NZ but still there is a delay.
Thanks very much to the trademe guys for increasing their TTL. As for the ANZ guys, if you are going to keep your TTL's at zero, it would really help if approx 10% of my queries to your nameservers didn't fail.. -- Simon J. Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
On Mon, 19 Mar 2007, Simon Lyall wrote:
As for the ANZ guys, if you are going to keep your TTL's at zero, it would really help if approx 10% of my queries to your nameservers didn't fail..
I gave up on ANZ's DNS resolution and put a static entry into my hosts file. Bad, yes, but as you say the instability of their name servers is so abysmal as to render them nearly useless. -- Matthew Poole "Don't use force. Get a bigger hammer."
On Mon, 2007-03-19 at 18:25 +1200, Simon Lyall wrote:
On Tue, 13 Mar 2007, Simon Lyall wrote:
Similarly the TTL for www.trademe.co.nz (and www.oldfriends.co.nz) is just ten seconds so this must be constantly rechecked. Not as bad since their servers are in NZ but still there is a delay.
Thanks very much to the trademe guys for increasing their TTL.
As for the ANZ guys, if you are going to keep your TTL's at zero, it would really help if approx 10% of my queries to your nameservers didn't fail..
It'd make a lot of sense to locate anz.co.nz's nameservers in NZ instead of Melbourne. Right now am getting to them via Tokyo, and this is on transit that's inside the gang of four inside .au. Your mileage may vary. So, the formula seems to be: 1. Make TTL's sensible 2. Locate .nz nameservers inside .nz 3. Add a dash of anycast if you're feeling frisky 4. ??? 5. Customers experiencing an improvement in online service (aka Profit!) jamie
participants (10)
-
Barry Murphy -
Bojan Zdrnja -
Dean Pemberton -
Gerard Creamer -
jamie baddeley -
Jamie Baddeley
-
Matthew Poole -
Nathan Ward -
Perry Lorier -
Simon Lyall