DNS problems yesterday?
Hey There, I was wondering if anyone noticed an odd external DNS problem happening yesterday, the National Library's main website was affected for several hours. We could resolve internally but nothing from outside would get resolved. Nothing was changed on our DNS servers. Vodafone seemed to be affected also. Default Server: planet.natlib.govt.nz Address: 192.122.171.130
server 210.55.131.76 Default Server: jupiter.natlib.govt.nz Address: 210.55.131.76
www Server: jupiter.natlib.govt.nz Address: 210.55.131.76
Name: slbweb.natlib.govt.nz Address: 210.55.131.96 Aliases: www.natlib.govt.nz
server defiant.netgate.net.nz Default Server: defiant.netgate.net.nz Address: 202.37.245.17
www.natlib.govt.nz Server: defiant.netgate.net.nz Address: 202.37.245.17
Name: slbweb.natlib.govt.nz Address: 210.55.131.96 Aliases: www.natlib.govt.nz
server reliant.net.nz *** Can't find address for server reliant.net.nz: Non-existent host/domain server reliant.netgate.net.nz Default Server: reliant.netgate.net.nz Address: 202.37.245.20
www.natlib.govt.nz Server: reliant.netgate.net.nz Address: 202.37.245.20
Name: slbweb.natlib.govt.nz Address: 210.55.131.96 Aliases: www.natlib.govt.nz
server 203.96.152.4 Default Server: rachel.paradise.net.nz Address: 203.96.152.4
www.natlib.govt.nz Server: rachel.paradise.net.nz Address: 203.96.152.4
*** rachel.paradise.net.nz can't find www.natlib.govt.nz: Non-existent host/domain
server 202.73.198.15 Default Server: isdn1ldv.vodafone.co.nz Address: 202.73.198.15
www.natlib.govt.nz Server: isdn1ldv.vodafone.co.nz Address: 202.73.198.15
*** isdn1ldv.vodafone.co.nz can't find www.natlib.govt.nz: Non-existent host/domain Cheers Patrick McHale National Library of New Zealand.
natlib.govt.nz SOA dns1.natlib.govt.nz. networks.natlib.govt.nz. 2007011817 3600 1200 1728000 86400 The serial suggests (of course, it doesn't prove, as the serial is a freeform number), that changes were done lots yesterday. It is my assumption that someone was fiddling. On 19/01/2007, at 9:26 AM, Patrick Mchale wrote:
Hey There,
I was wondering if anyone noticed an odd external DNS problem happening yesterday, the National Library's main website was affected for several hours. We could resolve internally but nothing from outside would get resolved. Nothing was changed on our DNS servers. Vodafone seemed to be affected also.
Default Server: planet.natlib.govt.nz Address: 192.122.171.130
server 210.55.131.76 Default Server: jupiter.natlib.govt.nz Address: 210.55.131.76
www Server: jupiter.natlib.govt.nz Address: 210.55.131.76
Name: slbweb.natlib.govt.nz Address: 210.55.131.96 Aliases: www.natlib.govt.nz
server defiant.netgate.net.nz Default Server: defiant.netgate.net.nz Address: 202.37.245.17
www.natlib.govt.nz Server: defiant.netgate.net.nz Address: 202.37.245.17
Name: slbweb.natlib.govt.nz Address: 210.55.131.96 Aliases: www.natlib.govt.nz
server reliant.net.nz *** Can't find address for server reliant.net.nz: Non-existent host/ domain server reliant.netgate.net.nz Default Server: reliant.netgate.net.nz Address: 202.37.245.20
www.natlib.govt.nz Server: reliant.netgate.net.nz Address: 202.37.245.20
Name: slbweb.natlib.govt.nz Address: 210.55.131.96 Aliases: www.natlib.govt.nz
server 203.96.152.4 Default Server: rachel.paradise.net.nz Address: 203.96.152.4
www.natlib.govt.nz Server: rachel.paradise.net.nz Address: 203.96.152.4
*** rachel.paradise.net.nz can't find www.natlib.govt.nz: Non- existent host/domain
server 202.73.198.15 Default Server: isdn1ldv.vodafone.co.nz Address: 202.73.198.15
www.natlib.govt.nz Server: isdn1ldv.vodafone.co.nz Address: 202.73.198.15
*** isdn1ldv.vodafone.co.nz can't find www.natlib.govt.nz: Non- existent host/domain
Cheers
Patrick McHale National Library of New Zealand.
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
!DSPAM:22,45afd7fe128422051574865!
I was wondering if anyone noticed an odd external DNS problem happening yesterday
Yes we had some issues as well.
We kept getting the following messages:
named[4498]: client 127.0.0.1#42127: no more recursive clients: quota reached
(We have a limit of 1000 and an average of 25)
And we could not get to some of the root servers, ie: a.ROOT-SERVERS.NET
So I am assuming there was a DOS attack somewhere which affected DNS traffic.
Assumption: The name server could not get to some root servers, the queries
kept pilling up and we hit the quota.
It looks like it was an issue on the
- 14th ( 422333 quota reached)
- 17th ( 170472 quota reached)
- 18th ( 547309 quota reached)
To put it into perspective here are the numbers for the previous 5 weeks:
4228
28
16
856
801
--
------------------------------------------------------------------------
Jean-Francois Pirus
On 18-Jan-2007, at 18:21, Jean-Francois Pirus wrote:
I was wondering if anyone noticed an odd external DNS problem happening yesterday
Yes we had some issues as well.
We kept getting the following messages: named[4498]: client 127.0.0.1#42127: no more recursive clients: quota reached (We have a limit of 1000 and an average of 25)
Do you run open resolvers, or do you restrict use of your recursive servers (by source address) to your customers only? Almost every case I've seen where bind9 suffers query spikes like you're describing (and are not just being hammered by an enormous throng of customers) it has been because the server was being used by someone far away as a packet amplifier. Throw on an ACL to restrict recursive lookups (and to deny queries, if the servers aren't also authority servers) and the problem frequently goes away.
And we could not get to some of the root servers, ie: a.ROOT- SERVERS.NET
In case it's useful to know for future testing, F and I are the servers that you have the greatest chance of reaching locally. Joe
Joe Abley wrote:
Throw on an ACL to restrict recursive lookups (and to deny queries, if the servers aren't also authority servers) and the problem frequently goes away.
I'd be interested to see a working BIND 9 ACL to restrict recursion to certain clients only. -- Juha Saarinen www.geekzone.co.nz/juha | Skype: juha_saarinen blogs.pcworld.co.nz/pcworld/techsploder www.computerworld.co.nz | MSN: juha_saarinen(a)msn.com Voice: +64 9 950 3023 Subtle recursive jokes in .sigs are not funny.
Joe Abley wrote:
Throw on an ACL to restrict recursive lookups (and to deny queries, if the servers aren't also authority servers) and the problem frequently goes away.
I'd be interested to see a working BIND 9 ACL to restrict recursion to certain clients only.
in named.conf: acl "localonly" { 192.168.1.0/24; ... 192.168.250.0/24; }; options { .... allow-recursion { "localonly"; }; .... }; see the BIND admin reference manual for more info (or one of the many howtos available on teh intarwebs) /joshua -- A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. - Douglas Adams -
On Fri, 2007-01-19 at 13:02 +1300, joshua sahala wrote:
Joe Abley wrote:
Throw on an ACL to restrict recursive lookups (and to deny queries, if the servers aren't also authority servers) and the problem frequently goes away.
I'd be interested to see a working BIND 9 ACL to restrict recursion to certain clients only.
in named.conf:
acl "localonly" { 192.168.1.0/24; ... 192.168.250.0/24; };
options {
....
allow-recursion { "localonly"; };
....
};
see the BIND admin reference manual for more info (or one of the many howtos available on teh intarwebs)
/joshua
Is there a significant difference between doing this and setting up two different BIND "views"? I'm currently using two view, one for our internal networks, and one for external networks, with an ACL to decide which view applies and recursion disabled for the external view. I've noticed that with a "views" configuration, the external view is very slow to update (the servers are run as slaves) when the master is updated. The internal view updates almost immediately, but it can be up to an hour or so before queries hitting the external view get the up-to-date records. Would I be losing anything important if I switched to just using the allow-recursion ACL? I suspect views might have been designed for a different configuration scenario... -- --Michael Fincham Unleash Technology Solutions
I'd be interested to see a working BIND 9 ACL to restrict recursion to
certain clients only.
http://www.cymru.com/Documents/secure-bind-template.html Very good base point for securing your bind setup.
On 18-Jan-2007, at 18:48, Juha Saarinen wrote:
Joe Abley wrote:
Throw on an ACL to restrict recursive lookups (and to deny queries, if the servers aren't also authority servers) and the problem frequently goes away.
I'd be interested to see a working BIND 9 ACL to restrict recursion to certain clients only.
Ask Team Cymru and ye shall receive. All that, and more! http://www.cymru.com/Documents/secure-bind-template.html Joe
Do you run open resolvers, Noooo, we've had ACL's on bind for a long time. Only our subnets can do recursive queries.
This server does a lot of recursive queries as it's a mail server and has to check all those #%$#$% spam connections.
In case it's useful to know for future testing, F and I are the servers that you have the greatest chance of reaching locally.
Thanks, I'll keep that in mind for next time.
--
------------------------------------------------------------------------
Jean-Francois Pirus
We kept getting the following messages: named[4498]: client 127.0.0.1#42127: no more recursive clients: quota reached (We have a limit of 1000 and an average of 25)
from the BIND 9.2 ARM: recursive-clients The maximum number of simultaneous recursive lookups the server will perform on behalf of clients. The default is 1000. Because each recursing client uses a fair bit of memory, on the order of 20 kilobytes, the value of the recursive-clients option may have to be decreased on hosts with limited memory.
And we could not get to some of the root servers, ie: a.ROOT-SERVERS.NET
how were you testing this? did you try any of the other root servers? If any of the root server were inaccessible it would have caused a whole lot more noise on teh intarwebs.
So I am assuming there was a DOS attack somewhere which affected DNS traffic.
where somewhere == your server
Assumption: The name server could not get to some root servers, the queries kept pilling up and we hit the quota.
I think you need to look closer to home, so to speak... Is this DNS server of your publicly accessible? If so, does it allow recursion from anywhere/anyone? If so, then you become a free DNS server for others, like spammers. Try restricting recursion to your network only. If it is already restricted, then check all your network hosts for signs of malware/virii/backdoors/etc as it seems possible that an internal host or two was spewing spam and making a lot of bogus recursive queries hth /joshua -- A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools. - Douglas Adams -
from the BIND 9.2 ARM:
Yes, I have read the documentation.
how were you testing this? did you try any of the other root servers? If any of the root server were inaccessible it would have caused a whole lot more noise on teh intarwebs.
It was an intermittent problem. ie: there was some packet loss probably restricted to DNS traffic.
Is this DNS server of your publicly accessible? If so, does it allow recursion from anywhere/anyone?
Nope.
--
------------------------------------------------------------------------
Jean-Francois Pirus
participants (8)
-
Jean-Francois Pirus
-
Joe Abley
-
joshua sahala
-
Juha Saarinen
-
Michael Fincham
-
Nathan Ward
-
Patrick Mchale
-
Simon Allard