On 28/03/2012, at 10:45 AM, Mauricio Freitas wrote:

I know of cases where people used to host DNS with a large ISP and after moving NS to other providers have to contact said ISP to “reset” DNS because their servers kept serving the old records for days… Even though people go on record saying “our servers respect TTLs” it seems some don’t…

I'd love to see some examples. I don't see why people would change TTLs - it's most likely to break things and increase support costs. I still think this maybe a misunderstanding about how DNS works. MMC

Cheers

Mauricio Freitas www.geekzone.co.nz www.freitasm.com www.twitter.com/freitasm

From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Craig Whitmore Sent: Wednesday, 28 March 2012 1:11 p.m. To: Cameron Bradley; NZNOG Mailing-List Subject: Re: [nznog] DNS TTL Mangling

...

It has come to my attention in the course of moving the DNS for a number of domains that several of the ISPs in this country are mangling the TTLs on records queried by their recursive DNS servers. This behaviour seems to me to be undesirable in > situations where someone may have set a record to a shorter TTL to facilitate smoother movement between hosting providers. In the cases I’m seeing, records with TTLs of 14400 are being handed out with TTLs of 86400 by the service provider’s >servers.

If an ISP (or anyone) is breaking/changing TTL's (and maybe other stuff in DNS) on purpose I would think IMHO this is bad. Think would make DNSSEC signed zones fail + other stuff you have said as the ISP is playing around with it.

Maybe you don't want to name who you think is doing it but maybe if anyone is doing this they may want to comment on the reasoning behind it.

Thanks Craig

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

I guess I forgot to note the following point. The DNS for the domains in question are not being moved either to or from any of the providers in question, but from a US-based webhost to an NZ-based one. The records (including SOA) all have a TTL of 14400, old records are still being served by some providers more than 24 hours later. Regards, Cameron Bradley From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Mark Foster Sent: Wednesday, 28 March 2012 13:21 To: Mauricio Freitas; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] DNS TTL Mangling Mauricio, What you're describing sounds to me more like 'legacy zone files & config left behind' and is relatively common (as the removal of zone files is a manual process as part of the deprovisioning work, and is often overlooked, if not by the customer than certainly by the DNS host losing the business. A very old problem. As opposed to the OP's point; ignoring the TTL specified by the manager of a domain name and imposing your own TTL strikes me as just plain 'bad'; some TTL's are low on purpose (DNS load balancing) and some are manually lowered prior to changes (such as moving ISPs and relocating mail servers, etc) and ISPs ignoring this are causing service problems for their customers, rather than the (presumed) rationale of load reduction. I'd love to see an ISP come up with justification for ignoring the zone-specified TTL. I don't see how anyone can justify being so selective in what DNS results they pass on direct, and which ones they fudge; what's next, full on DNS hijacking because ISP-knows-best? Mark. On 28/03/12 13:15, Mauricio Freitas wrote: I know of cases where people used to host DNS with a large ISP and after moving NS to other providers have to contact said ISP to "reset" DNS because their servers kept serving the old records for days... Even though people go on record saying "our servers respect TTLs" it seems some don't... Cheers Mauricio Freitas www.geekzone.co.nzhttp://www.geekzone.co.nz/ www.freitasm.comhttp://www.freitasm.com www.twitter.com/freitasmhttp://www.twitter.com/freitasm From: nznog-bounces(a)list.waikato.ac.nzmailto:nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Craig Whitmore Sent: Wednesday, 28 March 2012 1:11 p.m. To: Cameron Bradley; NZNOG Mailing-List Subject: Re: [nznog] DNS TTL Mangling

It has come to my attention in the course of moving the DNS for a number of domains that several of the ISPs in this country are mangling the TTLs on records queried by their recursive DNS servers. This behaviour seems to me to be undesirable in > situations where someone may have set a record to a shorter TTL to facilitate smoother movement between hosting providers. In the cases I'm seeing, records with TTLs of 14400 are being handed out with TTLs of 86400 by the service provider's >servers.

If an ISP (or anyone) is breaking/changing TTL's (and maybe other stuff in DNS) on purpose I would think IMHO this is bad. Think would make DNSSEC signed zones fail + other stuff you have said as the ISP is playing around with it. Maybe you don't want to name who you think is doing it but maybe if anyone is doing this they may want to comment on the reasoning behind it. Thanks Craig _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nzmailto:NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

On Wed, Mar 28, 2012 at 12:15:41AM +0000, Mauricio Freitas wrote:

I know of cases where people used to host DNS with a large ISP and after moving NS to other providers have to contact said ISP to "reset" DNS because their servers kept serving the old records for days... Even though people go on record saying "our servers respect TTLs" it seems some don't...

You mean XTRA right? That was years ago with alien.xtra.co.nz/terminator.xtra.co.nz. They took a long time to split their DNS. I'm not aware of any ISP's in NZ still mixing authorative/recursive. That said, with PPP assigning DNS thankfully it's pretty easy to people to shift to recursive for most customers if they haven't already. That said, I wonder if Paradise customer records are using the non-split DNS. 203.96.152.4 and 203.96.152.12 were popular on cable and it seems that it's very common for people to have these nameservers statically defined. % host -t ns paradise.net.nz paradise.net.nz name server rachel.paradise.net.nz. paradise.net.nz name server kirsty.paradise.net.nz. % host rachel.paradise.net.nz rachel.paradise.net.nz has address 203.96.152.4 Ben.

On Wed, 28 Mar 2012, Don Stokes wrote:

Are there still ISPs putting recursive and authortative name service in the same processes? If there are, can we name them please, so we can all point and laugh and tell everyone we know that they're clueless fracking idiots who don't deserve to have customers?

I'm sure Xtra would be happy to turn off resursion on alien and terminator if you would be happy to answer any support calls that came in.. -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.

Argueably it'd be easier to shift the DNS hosting to alternative authorative nameservers that aren't recursive than update legacy configurations. Given time it'll become just a trickle. But then there'll be isolated problems of a domain not working for one or two people. The thing is - it only hurts customers who are leaving - not a lot of motivation there. On Wed, Mar 28, 2012 at 12:54:43AM +0000, Paul Tinson wrote:

Xtra yes, Telecom probably not. I tried quite a lot no one was willing?

On 28/03/12 1:52 PM, "Simon Lyall" wrote:

...

On Wed, 28 Mar 2012, Don Stokes wrote:

...

Are there still ISPs putting recursive and authortative name service in the same processes? If there are, can we name them please, so we can all point and laugh and tell everyone we know that they're clueless fracking idiots who don't deserve to have customers?

I'm sure Xtra would be happy to turn off resursion on alien and terminator if you would be happy to answer any support calls that came in..

-- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

On 28/03/12 13:52, Simon Lyall wrote:

On Wed, 28 Mar 2012, Don Stokes wrote:

...

Are there still ISPs putting recursive and authortative name service in the same processes? If there are, can we name them please, so we can all point and laugh and tell everyone we know that they're clueless fracking idiots who don't deserve to have customers?

I'm sure Xtra would be happy to turn off resursion on alien and terminator if you would be happy to answer any support calls that came in..

So they should stop serving up authoritative DNS off them. FFS. -- don

On Wed, Mar 28, 2012 at 02:09:59PM +1300, Simon Lyall wrote:

On Wed, 28 Mar 2012, Don Stokes wrote:

...

On 28/03/12 13:52, Simon Lyall wrote:

...

On Wed, 28 Mar 2012, Don Stokes wrote:

...

Are there still ISPs putting recursive and authortative name service in the same processes? If there are, can we name them please, so we can all point and laugh and tell everyone we know that they're clueless fracking idiots who don't deserve to have customers?

I'm sure Xtra would be happy to turn off resursion on alien and terminator if you would be happy to answer any support calls that came in..

So they should stop serving up authoritative DNS off them. FFS.

I remember looking at that too (about 6 years ago when I was working there). I think Xtra was the registrar for 98% of the domains so those could be done as a big batch job through NZRS.

However that still left several hundred which would involve manually contacting people. Plus whatever programmes, scripts training were involved.

Either option would have involved significant expense ( ballpark $100,000 ) and customer inconvenience for no real gain.

Well if even doing the 98% problems would become less frequent.

However Xtra has been giving out completely different IPs as DNS servers when people connect for many ( 6 or 7 ) years now and AFAIK all the documentation tells me to use dynamic or those IPS. So the only people left are those with hardcoded entries.

So in theory the problem gets smaller since people refresh their machines regularly and put in the "learn automatically" settings. Of course in practise the "Local tech guy" manually hard-codes 202.27.184.2 & 4.

Yeah- What's 202.27.184.2? I thought Xtra's primary DNS was 202.27.158.40? There are three sets of nameservers for recursive DNS?

But I guess one day the number of people using the old IPs will be small enough it'll be worth the trouble.

Well there'll always be more new people having the same problems if they don't start handing out new IPs to people not using Xtra as a registrar. Even if at first they went to the same place. Ben.

On Wed, Mar 28, 2012 at 03:15:53PM +1300, Simon Lyall wrote:

On Wed, 28 Mar 2012, Ben Aitchison wrote:

...

On Wed, Mar 28, 2012 at 02:09:59PM +1300, Simon Lyall wrote:

...

So in theory the problem gets smaller since people refresh their machines regularly and put in the "learn automatically" settings. Of course in practise the "Local tech guy" manually hard-codes 202.27.184.2 & 4.

What's 202.27.184.2? I thought Xtra's primary DNS was 202.27.158.40? There are three sets of nameservers for recursive DNS?

Oops. Should have been 202.27.184.3 and 202.27.184.5 . Typed from memory instead of checking.

Just like the Local tech guy does! Mark.

On 28/03/12 16:07, Don Stokes wrote:

On 28/03/12 14:09, Simon Lyall wrote:

...

I'm still pointing and laughing.

DNS registrations are done by name, DNS resolvers are referenced by IP. So all they need to do is change the IP addresses that "alien.xtra.co.nz" and "terminator.xtra.co.nz" point to. Turn off recursive service at those new addresses, and stop making excuses for failing to meet industry best practice for well over a decade.

The only complication is where domain owners used their own NS records for registrations and publish their own glue records into their upstream registries/zones. I'm pretty sure Xtra didn't normally allow that, so I wouldn't imagine there would be very many.

I note Don that your own domain name has IP's published into the registry (viewed via whois), not just DNS hostnames. If your user-interface for customers registering new domain names or modifying their domain NS records makes it clear that IP's are optional, then you're better than many I've dealt with over the years. In practise the domain name registration forms have (forever that I recall) asked for IP addresses for registrations when carried out, and in most circumstances, these have been provided. Indeed I expect many registrants have set these fields as compulsory in the past? And even when not, isn't it good practice to specify the IP addresses when you figure that those will work, even if xtra.co.nz's zone get's busy/unresponsive or is accidentally left to expire or broken in some way? (insulating you from the problems this causes, in the knowledge that the IP's are very unlikely to change in the near future?) Noting that many domain names held with Xtra are there for legacy reasons (eg, back-in-the-day that's who they used, and noone's bothered moving to a cheaper/better registrar), I don't hold out much hope that many customers have alien/terminator specified without also specifying IP addresses. Back in the days when I provided technical support for domain name registrations at Xtra, i'm pretty sure we specified both IP's and Hostnames for everything we registered[1], as it was considered normal, good practise, at the time - it's only in recent years that i've started leaving the IP's out and relying on DNS. But only where the DNS zone i'm relying on is one that I have some faith in, and/or one where I can take responsibility for any muckups that may occur (eg one I manage myself). Cheers Mark. [1] Pretty sure this applied when we were registering domains ourselves on behalf of customers using the Xtra UI; Doing the same via third party registrar UI's based on customer requests (notably Domainz, a legacy from when they monopolised this space) and also when advising customers what details to use if they were doing it themselves (obviously we'd have to provision zones for them first) - I don't recall ever advising customers to only use the DNS names for the servers. This is in the early-to-mid-2000's.

On 28/03/12 17:30, Mark Foster wrote:

Noting that many domain names held with Xtra are there for legacy reasons (eg, back-in-the-day that's who they used, and noone's bothered moving to a cheaper/better registrar), I don't hold out much hope that many customers have alien/terminator specified without also specifying IP addresses.

I should further point out that I would expect many domain names with NS=alien/terminator are also registered via third party registrars, so Xtra can't even automatically make the appropriate changes in the NZRS. Heck, does Xtra even provide a UI to directly manage domain names registered through them these days? Or do changes to both the .nz Registry and to DNS zones still require manual intervention by Xtra Tech Support? Mark.

On 03/28/2012 05:30 PM, Mark Foster wrote:

I note Don that your own domain name has IP's published into the registry (viewed via whois), not just DNS hostnames.

Yep, I certainly do. But that's because I control the name servers that serve up my names, so I ought to know if they're going to change... It's marginally preferable to do it that way, as it means that the addresses of the name servers don't need an extra query to obtain them. I recall a long while ago, a major ISP had its email hosted in domain A, domain A had its NS records in domain B, which in turn had its NS records in domain C, which finally had some actual A records. Note that B was in a different TLD to A & C. Since BIND 8's forwarding strategy goes like this: On receipt of query from client * If query can be answered from known data (authoritative or cached), answer the query * Else, send query to best next hop (again from known data) * Stop. On receipt of query answer: * Cache it. * Stop. It took several retries /by the client/ before it would get through that loop with enough cached answers to get a response. Enough that the client resolver would give up and claim the name couldn't be resolved. A few manual queries could prime the cache enough to get answers, but of course the next time you wanted to send them mail the cache entries had expired and it was off around that loop again... BIND 9 is better at this, actually continuing the recursion on receipt of query answers, so it will eventually give a positive answer (or a SERVFAIL) to a client. But the fact remains that if you have the NS records in the same zone, with glue, no other name servers need to be queried between the registry NSes and the zone's NSes. However, it is a very bad idea to enshrine IP addresses of servers that you don't control into glue records held in registry records the name server operator can't change, as the name server may at some point need to change its IP address. If the NS is in a domain the operator controls, they can just change it (and possibly upstream glue for the NS's name) and the domains carried on it keep working as if nothing happened. A competent name server operator will refuse to accept a zone with glue they don't control. The second best approach, and the best when the registrant and name server operator aren't the same entity, is to use the /same name servers/ as the name server operator's domain, that is: /isp-domain. /NS ns1./isp-domain. / NS ns2./isp-domain. /ns1/.isp-domain/ A /ns1-ip-address/ ns/2.isp/-domain A /ns2-ip-address/ /customer-domain/. NS ns1./isp-domain./ NS ns2./isp-domain./ and not: /isp-domain. /NS ns1./isp-domain. / NS ns2./isp-domain. /ns1/.isp-domain/ A /ns1-ip-address/ ns/2.isp/-domain A /ns2-ip-address/ /customer-domain/. NS cust-ns1./isp-domain./ NS cust-ns2./isp-domain./ as in the latter config, cust-ns1./isp-domain/ won't have a glue record, and ns1./isp-domain/ will need to be consulted to get it before /customer-domain/ can be resolved. I believe Xtra always used alien/terminator as the name servers back in the day; they didn't accept in-domain NS records referring to those IP addresses ... or at least if they ever did, they were a very small minority.

Back in the days when I provided technical support for domain name registrations at Xtra, i'm pretty sure we specified both IP's and Hostnames for everything we registered[1], as it was considered normal, good practise, at the time - it's only in recent years that i've started leaving the IP's out and relying on DNS. But only where the DNS zone i'm relying on is one that I have some faith in, and/or one where I can take responsibility for any muckups that may occur (eg one I manage myself).

NZ has always (since the Waikato University days) only pushed glue records for name servers that are within the domain they are NS for. It's a strategy that has worked extremely well. (Various gTLD registries have had some fairly bad ideas about insisting on glue IP addresses when they don't actually need them.) The "addresses" submitted to the NZ registry for alien/terminator (and other NSes) were often crap (wrong, syntactically incorrect, names instead of numbers, stories about what the registrant did in their holidays et c.); it didn't matter since the data was discarded long before they ever got anywhere near a name server. The only place where the glue for alien/terminator.xtra.co.nz has ever been published in the DNS at any time was alongside the xtra.co.nz delegation. It actually threw me when working on the DRS implementation in early 2000 that the glue data in the Waikato records was such complete garbage. It turned out that they had been capturing it unconditionally, but if the NSes weren't within the domain being registered, it wasn't checked for any kind of validity since it would never be published. A major (and slightly controversial!) part of the DRS deployment was to clean up all the Waikato data before importing it, and in that process the unnecessary glue was removed. The NZRS SRS used to discard unnecessary glue; I believe it now actually rejects it. Other TLD name servers similarly won't publish a glue record for an NS that isn't in the TLD zone. Doing so is disallowed by the DNS RFCs. They may push extraneous glue that's within the same TLD ... they used to, but I'm not sure what current practice is. -- don

On 03/28/2012 09:55 PM, Mark Foster wrote:

... So how does this relate to the .nz root zone and glue records for external DNS Server A records provided during registration, then? [see below]

Not sure what you mean. If you provide a glue IP address to a registrar that isn't for a name inside your domain, they may capture and store it, bit it's not stored in the NZ SRS and not published to the DNS. It's just useless bits cluttering up the registrar's database.

Well I just modified one of my .nz domains (using Discount Domains) ... it previously only had NS address records for it's DNS servers. I populated the IP of my first DNS server and their web management tool automatically resolved the other two and has populated the nameserver fields with both hostnames and IP addresses. The Address records are indeed outside of the zone in question. (Bonus points, the IP address field was indeed described as optional.)

As an exercise, I then modified my Primary DNS server for the domain in question, to be the correct address record, but an incorrect Ip address (8.8.8.8, ohai Google). It accepted this too, no error, so there's obviously no checking that the A record matches the IP involved...

In this case, it's not going into the DNS, so there's nothing really to check. But note that even for in-domain glue records, there's no special checking (beyond syntax and IP range) of glue IP addresses. Registrars may provide some checking, but as long as the IP address is valid (and the NS in the domain), the glue record will be pushed by the SRS without further checking. There used to be (in the Waikato system, and I think in the DRS) a requirement that name servers be up and answering with the zone before the registry would accept an update. But the SRS has never done that. Mainly this is because automated tools need to be able to commit to registering a domain in the registry before they deploy the new zone to their name servers. Probably DD just send a DNS request to the specified IP and blithely populate the fields from the response. Never mind that it's not authoritative ... -- don

On 2012-03-27, at 20:29, Don Stokes wrote:

Craig Whitmore wrote:

...

If an ISP (or anyone) is breaking/changing TTL's (and maybe other stuff in DNS) on purpose I would think IMHO this is bad. Think would make DNSSEC signed zones fail + other stuff you have said as the ISP is playing around with it. Um, no, messing with the TTL doesn't break DNSSEC (the TTL isn't signed), although potentially a vastly extended TTL could push caching of a DNSSEC records beyond the expiry of their keys.

Keys in DNSSEC do not have inception or expiry times. Keys don't expire. Signatures expire. The TTL on a DNSKEY is only really relevant in this discussion when it comes to key rollover, but recommended practice for rollovers involves significant delays before keys are withdrawn, long after the last signatures with those keys have been withdrawn. Any TTL mangling of that magnitude is likely to have noticeable impact on customers in general, never mind the (likely tiny) population that validate. And if a cache is broken that badly, either the operator of the cache cares and will fix it, or they don't care and their customers will go somewhere else. Either way, problem solved.

(If a < 24 hour expiry did this, whoever is maintaining the DNSSEC keys is Doing It Wrong.) But in general, every recursive name server updates the TTL of its cached records every time it issues a cached answer to a query.

The required behaviour in the case where RRSIGs expire before their TTL is well-specified in the DNSSEC protocol. I have heard no credible suggestion that common/widespread implementations get this wrong. Modifying TTLs in caches should have zero practical impact on DNSSEC operation. I have limited sympathy for complaints about how caches are operated. If you don't get good results from the cache you're using, use a different one or run your own. Joe

On Wed, Mar 28, 2012 at 01:22:56PM +1300, Sebastian Castro wrote:

On 28/03/12 13:01, Cameron Bradley wrote:

...

providers. In the cases I?m seeing, records with TTLs of 14400 are being handed out with TTLs of 86400 by the service provider?s servers. For example, if a record has a short TTL (300 seconds), then is "stored" in the cache with a minimum TTL (1 hour) set by the operator. The same applies to records with large TTLs (few days) are put into the cache with 1-day TTL.

I think pushing 5 second ttls up to 60 seconds. Or maybe even 60 to 300 could be ok. Especially in a home setting, where a transparent proxy may even be catching your traffic anyway. But he's complaining about 14400 being forced up to 86400 which is completely different.

I'm not aware of how common is this practice, but the argument I heard is to ease the load in cache in the case of low TTLs.

...

How common is this practice, and what are the benefits to the SP from doing it? From my perspective there is also the concern that this, for all intents and purposes appears to be bad practice, and serves to ?break? DNS in itself.

Effectively a good DNS administrator would like to control their TTL at their will (we do!) based on a rational process. For example, CDN operators use low TTL to quickly react to outages. But there is also lots of breakage out there caused by non-rational decisions. If you put a low TTL plus a nameserver with the wrong config, you can easily get a query storm in your cache.

Argueably anycast is a much better solution for reacting quickly, and DNS is often used for load balancing to seperate regions. Google having a TTL of 60 seconds seems to encourage a lot of queries. And Facebook with 120 seconds is at least slightly better. If anycast was more widely deployed for CDNs then we wouldn't have this uproar about CDNs not being linked to the source IP address, and going long distances when using shared DNS caches. And realistically if there is anycast and many seperate locations for recursive DNS then there's going to be a lower hit count without shared caching: which hurts smaller sites more than more popular sites and creates even more of a digital divide. Ben.