There is also a discussion going on over at the pool list; http://lists.ntp.org/pipermail/pool/2016-December/007996.html I've seen the same across a number of my (personal) pool servers. Alex Smith Infrastructure Engineer | Trade Me E. alex(a)trademe.co.nz M. 022 0599 037 -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Tony Wicks Sent: Friday, 16 December 2016 11:57 AM To: Lincoln Reid Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] pool.ntp.org traffic gone wild Someone on NANOG has just asked the same thing.

On 16/12/2016, at 11:42 AM, Lincoln Reid wrote:

Hi,

Is anyone else in NZ seeing their pool.ntp.org traffic volumes up by an order of magnitude over the last few days?

20mbps+ seems like quite a lot of NTP.

Cheers,

-- Lincoln Reid Head of Networks ACSData - AS18119 lincoln(a)acsdata.co.nz Phone: +64 4 939 2200 Fax: +64 4 939 2201

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

As another local datapoint, We are also seeing the same sharp rise in NTP connections. Cheers Shane Geddes | Systems Engineer | Solarix Networks ddi. +64 9 951 5092 | shane.geddes(a)solarix.co.nz 5 Omega Street, Rosedale, Auckland, New Zealand www.solarix.co.nz ________________________________________ From: nznog-bounces(a)list.waikato.ac.nz [nznog-bounces(a)list.waikato.ac.nz] on behalf of Lincoln Reid [lincoln(a)acsdata.co.nz] Sent: Friday, 16 December 2016 2:20 p.m. To: Alex Smith (Platform); Tony Wicks Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] pool.ntp.org traffic gone wild On Fri, 2016-12-16 at 00:16 +0000, Alex Smith (Platform) wrote:

There is also a discussion going on over at the pool list;

http://lists.ntp.org/pipermail/pool/2016-December/007996.html

There doesn't seem to be any consensus there on what has caused it yet. iptables on my personal pool server had a default maximum of ~65k connections which was way too low. I disabled connection tracking yesterday for NTP as one of the folks in the list.ntp.org thread also did and all looked good from then on.

I've seen the same across a number of my (personal) pool servers.

Thanks, good to get another local datapoint. Cheers, -- Lincoln Reid Head of Networks ACSData - AS18119 lincoln(a)acsdata.co.nz Phone: +64 4 939 2200 Fax: +64 4 939 2201 _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

One thing that pops to my mind is that we've seen a lot of TR-069/064 'SetNTPServers' exploits in the wild the past few days. Many of these try to download and execute a script, though some actually set the NTP servers first. Perhaps a huge number of devices that previously had no NTP configured, suddenly do? Completely wild theory, but who knows. Cam -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Roland Dobbins Sent: Friday, 16 December 2016 4:18 PM To: nznog Subject: Re: [nznog] pool.ntp.org traffic gone wild On 16 Dec 2016, at 8:31, Shane Geddes wrote:

We are also seeing the same sharp rise in NTP connections.

Maybe something like this? http://pages.cs.wisc.edu/~plonka/netgear-sntp/ ----------------------------------- Roland Dobbins _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

I do indeed. -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Roland Dobbins Sent: Friday, 16 December 2016 5:06 PM To: nznog Subject: Re: [nznog] pool.ntp.org traffic gone wild On 16 Dec 2016, at 10:38, Cameron Bradley wrote:

Many of these try to download and execute a script

Do you mean Mirai compromises using the TR-069/-064 propagation vector? ----------------------------------- Roland Dobbins _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

The chatter in #ntp on IRC infers that it was through a change made by a IoT vendor (though that's all the info that's been given, so take that with as much salt as you wish). It does however, seem to have had the unfortunate side effect of knocking a number of servers out of the pool, dropping it down to either remaining. I've added a couple of new ones myself (personally and day job has allowed me to add two) to try and bring the numbers up, but if your able and willing it would be awesome if some others could donate a little bit of resource to the pool. It's quite widely used and quite under-resourced here. Regards, Alex Smith E: alex(a)smith.is P: 022 0521 257 On Fri, Dec 16, 2016 at 5:24 PM, Roland Dobbins wrote:

On 16 Dec 2016, at 11:07, Cameron Bradley wrote:

I do indeed.

...

This makes sense - I'll do some digging and see if any of the Mirai variants we've captured is setting the timeservers on compromised devices to pool.ntp.org addresses.

----------------------------------- Roland Dobbins _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

On Fri, Dec 16, 2016 at 6:17 PM, Joseph B wrote:

Quite surprising to see so many queries from outside of our region, I wonder if the DNS geolocating for the NTP pool isn't quite working as it should. Then again I haven't looked that closely recently.

This is an interesting point, and something that seems to be reflected in data here too. There is a good chunk of the data inbound on my personal ones that is EU sourced. Other people in #ntp are seeing a large increase from the US. My point regarding the low numbers in pool is still valid though, even with ntp.net.nz (which is not in pool) the pool is so widely used it could do with a bit of TLC to avoid it impacting pool operators in the future (and thus the end users). I'll have to dig out the stats but I'm sure there are some folks who use far more of it than they should who have the ability to contribute - or maybe I need to name/shame ;) Have a great weekend all, Alex Smith E: alex(a)smith.is P: 022 0521 257

This is an interesting point, and something that seems to be reflected in data here too. There is a good chunk of the data inbound on my personal ones that is EU sourced. Other people in #ntp are seeing a large increase from the US. 

So another data point re US increase. One of my boxes in the US pool is at Linode, I don't run any other graphing on it however Linode provide some of their own graphs. Inbound IPv4 traffic for last month. https://www.dropbox.com/s/bhrutx8if1zr39l/Screenshot%202016-12-16%2016.07.22... CPU for the last month. https://www.dropbox.com/s/93zjxzn6dxl86qz/Screenshot%202016-12-16%2016.07.10... The traffic graphs for IPv6 show no jump like the IPv4 graph, it's doing 4x as much traffic as compared with a few days back, previously the traffic pattern has been fairly stable. Cheers, Joseph

Looks like it may possibly be the snapchat app causing all of this http://mailman.nanog.org/pipermail/nanog/2016-December/089590.html ________________________________________ From: nznog-bounces(a)list.waikato.ac.nz on behalf of Roland Dobbins Sent: Saturday, 17 December 2016 2:31 a.m. To: nznog Subject: Re: [nznog] pool.ntp.org traffic gone wild On 16 Dec 2016, at 19:38, Joseph B wrote:

it's doing 4x as much traffic as compared with a few days back, previously the traffic pattern has been fairly stable.

Any info on the source distribution? Is the delta of sntp queries originating from what appear to be broadband access networks, or . . . ? ----------------------------------- Roland Dobbins _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

Can you get a couple of hydrogen maser's while your at it 😄 Sent from my iPhone

On 16/12/2016, at 1:17 PM, Daniel Griggs wrote:

Hi Everybody,

It seems like a salient time to let everyone know we recently commissioned our fourth public rubidium backed, GPS sync'd NTP server.

Our NTP servers are for there for NZ. So check them out on http://www.ntp.net.nz

If you have any suggests for improvements or expansions let us know

Daniel Griggs NZRS

...

On 16/12/2016, at 11:56 AM, Tony Wicks wrote:

Someone on NANOG has just asked the same thing.

...

On 16/12/2016, at 11:42 AM, Lincoln Reid wrote:

Hi,

Is anyone else in NZ seeing their pool.ntp.org traffic volumes up by an order of magnitude over the last few days?

20mbps+ seems like quite a lot of NTP.

Cheers,

-- Lincoln Reid Head of Networks ACSData - AS18119 lincoln(a)acsdata.co.nz Phone: +64 4 939 2200 Fax: +64 4 939 2201

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog