Hi All, One of the sessions this morning was from Geoff Huston on the recent NTP server DDoS attacks. The presentation is here https://conference.apnic.net/data/37/2014-02-25-ntpddos_1392941955.pdf In short if you appear on the http://openntpproject.org/ list then you should be looking to clean this up before you contribute to global DDoS attacks the size of which we have not seen in the past. I've run the WIX and APE peering ASNs against the page and here are the number of open NTP servers. You can do this yourself by substituting your AS (instead of 9483) into the following URL http://openntpproject.org/searchby-asn.cgi?search_asn=9483 If you want it as a csv then use http://openntpproject.org/searchby-asn.cgi?search_asn=9483&csv=1 Here is the WIX & APE list. Bear in mind that this only covers the NTP servers which are in the AS that peers on the exchange. Not the customers of that AS. You should use the page above to check all your customer ASNs as well and work with them to patch any servers. WIX Open NTP servers 4823 total 709 AS4770 618 AS2687 603 AS9503 368 AS23838 304 AS23655 272 AS17746 256 AS18119 205 AS24183 187 AS45280 187 AS24466 158 AS18400 122 AS10022 118 AS18015 117 AS55454 91 AS9872 73 AS132040 63 AS24324 58 AS38477 53 AS18353 51 AS45177 39 AS17412 27 AS17435 25 AS42 23 AS9433 19 AS17705 17 AS24347 12 AS23905 10 AS8674 8 AS9834 4 AS24074 4 AS24005 4 AS17649 4 AS17484 3 AS38329 3 AS132003 3 AS12076 2 AS24006 1 AS7696 1 AS18021 1 AS17792 APE Open NTP servers 11877 total 2472 AS4739 1773 AS9889 1236 AS9738 709 AS4770 618 AS2687 603 AS9503 460 AS4826 368 AS23838 304 AS23655 285 AS45459 272 AS17746 256 AS18119 229 AS38333 205 AS24183 187 AS45280 187 AS24466 185 AS24130 158 AS18400 125 AS9790 125 AS9245 122 AS10022 118 AS18015 97 AS56030 91 AS9872 73 AS132040 63 AS24324 58 AS38477 53 AS18353 51 AS45177 44 AS58666 39 AS17412 37 AS55561 27 AS17435 26 AS9303 25 AS42 23 AS9898 23 AS9433 19 AS55752 19 AS17705 17 AS9431 16 AS18199 10 AS24398 10 AS23710 8 AS9834 7 AS17492 6 AS55785 5 AS9559 5 AS55872 4 AS24074 4 AS24005 4 AS17649 3 AS45946 3 AS12076 2 AS9897 2 AS9896 2 AS37974 1 AS38451 1 AS132347 1 AS132304 1 AS10200
What are other’s doing at a network level, if anything? I’ve been mulling over the idea of using L4 Redirect on Cisco ISG profiles to transparently redirect all customer NTP requests towards our NTP servers, which will then allow blocking NTP traffic upstream except to our NTP servers. It’s possibly a little controversial to transparently redirect NTP, but we’ve had DNS redirection in place for over a year now and not a single complaint. We put that in place to tackle people using the likes of OpenDNS/Google DNS then complaining when CDN performance sucked. Prefer to avoid L4 redirection if possible though given how intrusive it is but for most of the customers I’ve looked into on our list of open NTP addresses we don’t control their CPE and trying to get them to sort it out has proven difficult. -Scott ________________________________ The content of this message and any attachments may be privileged, confidential or sensitive. Any unauthorised used is prohibited. Views expressed in this message are those of the individual sender, except where stated otherwise with appropriate authority. All pricing provided is valid at the time of writing only and due to factors such as the exchange rate, may change without notice. Sales are made subject to our Terms & Conditions, available on our website or on request. ________________________________
Scott Pettit
What are other’s doing at a network level, if anything?
If you've the ability to match on length, ntp timesync requests & replies are 76 bytes in length (minus layer-2 framing).
You can allow that & drop anything else from/to UDP/123 for targets under attack or ntpds being abused.
Transparently redirecting DNS or ntp is hostile & unethical, if not actionable. Setting up policies so that customers must by default use your recursive DNS & ntp setups makes perfect sense, as long as those policies are made clear & as long as 'advanced' customers can opt out.
With regard to DNS, also allowing Google DNS & OpenDNS makes sense; for ntp, pool.ntp.org, time.apple.com, etc. Most customers will be happy with your infrastructure, plus the popular 3rd-party ones.
---------------------------------------
Roland Dobbins
On 25/02/2014, at 17:55, Roland Dobbins
Transparently redirecting DNS or ntp is hostile & unethical, if not actionable. Setting up policies so that customers must by default use your recursive DNS & ntp setups makes perfect sense, as long as those policies are made clear & as long as 'advanced' customers can opt out.
I assume you mean non-notified transparent redirection. Hostile and unethical are interesting terms to use if customers are informed of the behaviour. Depending on one's customer base, the vast majority of users are likely far more interested in "Can I get a correct answer to DNS questions?" and "Can I sync my clocks to something that looks like the correct time?" than "Can I get an answer from DNS/NTP servers of my choice?" An opt-out policy of "You must use my recursive DNS and NTP infrastructure" (presumably enforced by packet filters) will almost certainly result in more support calls from such a customer base than transparently redirecting the same traffic to (supposedly) known-good servers. That is to say, a filter-enforced policy combined with transparent redirection may make more sense than a filter-enforced policy alone. Doing either without the opt-out component is not something I'd consider a good idea, but as they say, your network, your rules. I did overhear someone mention that transparent snaffling of packets on a network run by an company called End2End was somewhat amusing, though :p Cheers -Mike
On Feb 25, 2014, at 1:53 PM, Mike Jager
I assume you mean non-notified transparent redirection.
Correct - I should've made that clear, thanks for pointing it out.
That being said, how many customers understand enough to know what they're agreeing to have performed on their traffic?
Also, there could be very serious consequences for dorking around with ntp, especially - far too many critical systems (incorrectly) utilize the public Internet for this sort of thing.
-----------------------------------------------------------------------
Roland Dobbins
Forgive me for missing the obvious here, but isn't the answer to drop packets emitting from customers on UDP/123 above _a certain rate limit_? Transparent UDP/123 redirection is going to break a lot of assumptions people have about how their current systems work, and would certainly get me, if i were a customer, very hot under the collar indeed. Debugging the subtle problems this would cause would mean a lot of wasted hours for many expensive people. Regards, Joel van Velden Cloud Scale Ltd NZ Cloud Storage API-compatible with Amazon S3. On 25/02/2014 11:00 p.m., Dobbins wrote:
On Feb 25, 2014, at 1:53 PM, Mike Jager
wrote: I assume you mean non-notified transparent redirection. Correct - I should've made that clear, thanks for pointing it out.
That being said, how many customers understand enough to know what they're agreeing to have performed on their traffic?
Also, there could be very serious consequences for dorking around with ntp, especially - far too many critical systems (incorrectly) utilize the public Internet for this sort of thing.
----------------------------------------------------------------------- Roland Dobbins
// http://www.arbornetworks.com Luck is the residue of opportunity and design.
-- John Milton
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On Feb 25, 2014, at 6:19 PM, Joel van Velden
Forgive me for missing the obvious here, but isn't the answer to drop packets emitting from customers on UDP/123 above a certain rate limit?
IMHO, the obvious solution is to block ntp packets which aren't 76 bytes in length towards either attack targets or to/from ntpds being abused, because source-based QoS isn't that commonplace, plus per-source numbers can be relatively (*relatively*) low compared to the attack aggregate.
This approach has been used with considerable success over the last week-and-a-half, FWIW.
-----------------------------------------------------------------------
Roland Dobbins
On 25 Feb 2014, at 5:32, Dobbins, Roland
On Feb 25, 2014, at 6:19 PM, Joel van Velden
wrote: Forgive me for missing the obvious here, but isn't the answer to drop packets emitting from customers on UDP/123 above a certain rate limit?
IMHO, the obvious solution is to block ntp packets which aren't 76 bytes in length towards either attack targets or to/from ntpds being abused, because source-based QoS isn't that commonplace, plus per-source numbers can be relatively (*relatively*) low compared to the attack aggregate.
This approach has been used with considerable success over the last week-and-a-half, FWIW.
It doesn’t do great things for the long-term development of NTP as a protocol, though. I understand the need to put out flames during a big fire, but short-term fixes that don’t cause immediate damage have a habit of sticking around. Joe
On Wed, 26 Feb 2014, Joe Abley wrote:
I understand the need to put out flames during a big fire, but short-term fixes that don’t cause immediate damage have a habit of sticking around.
Especially when they turn into "Security best practice" http://support.microsoft.com/kb/828263 -- Simon Lyall | Very Busy | Web: http://www.simonlyall.com/ "To stay awake all night adds a day to your life" - Stilgar
That's why we use NAT for security, right?
On Thu, Feb 27, 2014 at 9:45 AM, Simon Lyall
On Wed, 26 Feb 2014, Joe Abley wrote:
I understand the need to put out flames during a big fire, but short-term fixes that don't cause immediate damage have a habit of sticking around.
Especially when they turn into "Security best practice"
http://support.microsoft.com/kb/828263
-- Simon Lyall | Very Busy | Web: http://www.simonlyall.com/ "To stay awake all night adds a day to your life" - Stilgar
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On 27/02/14 07:57, Sam Russell wrote:
That's why we use NAT for security, right?
While you’re at it, it’s probably a good idea to block all ICMP, as you don’t want hackers to be able to ping things (after all, that’ll only encourage them to break in), and ICMP isn’t actually used for anything other than pings.
On 27/02/2014, at 1:55 pm, Jeremy Visser
On 27/02/14 07:57, Sam Russell wrote:
That's why we use NAT for security, right?
While you’re at it, it’s probably a good idea to block all ICMP, as you don’t want hackers to be able to ping things (after all, that’ll only encourage them to break in), and ICMP isn’t actually used for anything other than pings.
Please warn people before you use sarcasm. I’ll be fine, I just need some pictures of kittens and dandelions.
I would have thought that the NCSC have something to say about this all. I remember a while ago they contacted lots of ISPs telling them about their open DNS Servers but as far as I can tell I have heard nothing from them for a while (Or maybe I am wrong and didn¹t notice) As these NTP Servers have caused quite a lot of issues/outages with quite a number of NZ ISPs lately you¹d think the NCSC would have something to say but all I can see from their press releases etc they talk about Firefox and Mobile Devices. - Craig
On Feb 27, 2014, at 12:18 AM, Joe Abley
I understand the need to put out flames during a big fire, but short-term fixes that don’t cause immediate damage have a habit of sticking around.
Concur 100%. And I advocate it being done selectively where possibly, in mitigation centers through which traffic destined for attack targets is diverted during an attack.
-----------------------------------------------------------------------
Roland Dobbins
participants (12)
-
Craig Whitmore
-
Dean Pemberton
-
Dobbins, Roland
-
Jeremy Visser
-
Joe Abley
-
Joel van Velden
-
Lloyd Parkes
-
Mike Jager
-
Roland Dobbins
-
Sam Russell
-
Scott Pettit
-
Simon Lyall