On 11/02/2014, at 5:31 pm, Tim Price wrote:

Was just wondering how this effects .nz’s public NTP servers, i know that i’ve had to firewall off a number of NTP servers that were in the oceanic pool up until a day ago. How long before we lose ntp[1-3].ntp.net.nz?

They are mostly protected against this but there are some issues with the Symmetricom software that makes this a 'mostly' rather than a 'fully'. Not serious and we can explain off-list if you wish. We won't be buying Symmetricom again or recommending them, unless the new owners do something miraculous. Jay

On 11/02/2014, at 4:43 pm, Juha Saarinen wrote:

...

http://www.itnews.com.au/News/372033,worlds-largest-ddos-strikes-us-europe.a...

Amplification factor: 58.5.

-- Juha Saarinen twitter: juhasaarinen

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Jay Daley Chief Executive .nz Registry Services (New Zealand Domain Name Registry Limited) desk: +64 4 931 6977 mobile: +64 21 678840 linkedin: www.linkedin.com/in/jaydaley

On Feb 11, 2014, at 6:14 PM, Joel van Velden wrote:

So even older ntp servers should be immune by default.

Many aren't - and it's primarily level-6/level-7 commands which are abused by attackers, AFAIK. http://www.openntpproject.org/ ----------------------------------------------------------------------- Roland Dobbins // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton

On Wed, Feb 12, 2014 at 12:49:39PM +1300, Don Stokes wrote:

It might also be a useful occasion to plug nz.pool.ntp.org. If you have Right after major attacks on open NTP servers doesn't seems like the best time to plug opening your NTP servers ... 8-)

Seriously though, NTP is a service ISPs should be providing to customers. Doing local anycast of NTP service to well-run stratum 2 servers, (in turn talking to well-run stratum 1 servers, such as the NZRS ntp.net.nz servers and the MSL servers) is a far better idea than having punters querying random NTP pools of unknown quality, or opening up NTP servers to the whole world.

I think the biggest problem with this idea is that it's recommended to have at least 3 NTP servers. So not only do you have to encourage ISP's to have a NTP server, but multiple NTP servers.. Ben.

On 12/02/2014, at 1:07 pm, Nathan Ward wrote:

For resiliency (so that one can be offline or unreachable for an extended period, and lets face it, this is the Internet) you need 4.

I agree, four is best in case one goes and the remaining two disagree. Paranoid but not unknown. It's a weakness of ntp.net.nz that we only have three and something we would like to address at some point this year. Jay

Others disagree and consider a bad ticker to be a fault that should be resolved just like poor connectivity, and only engineer for one of these situations at a time. You can choose :-)

-- Nathan Ward _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Jay Daley Chief Executive .nz Registry Services (New Zealand Domain Name Registry Limited) desk: +64 4 931 6977 mobile: +64 21 678840 linkedin: www.linkedin.com/in/jaydaley

On 12/02/14 12:53, Ben wrote:

I think the biggest problem with this idea is that it's recommended to have at least 3 NTP servers. So not only do you have to encourage ISP's to have a NTP server, but multiple NTP servers.. Ben.

That you need three NTP servers is largely a myth. It's true in certain configurations, and it's good to have plenty of sources for your stratum 2 servers (I use the three NZRS ones and the two MSL ones) but these are generally specialised configurations where where millisecond resolution timing is required - and seriously, if timing is that important to your application, run a local GPS clock and leave the Internet out of it. For "Internet" applications, or just having your system clock line up with the time pips, one NTP server is fine. Two if your device's internal clock is completely broken and you can't survive losing sync for a few hours of downtime. That said, there's no great difficulty in running multiple NTP servers in a provider network. Put one each in your three largest POPs and anycast a local address for those who only need one server (so they hit the nearest one). It's also good to have plenty of sources for your stratum 2 servers. I use the three NZRS ones and the two MSL ones. I won't go near the NTP pools for anything important. I've actually seen NTP servers lose sync when relying on three NTP pool addresses, because all three had failed or started handing out bogus answers. ntpd only looks up the DNS once, so the IP addresses it gets on startup are the IPs it runs with until the next restart. IME, NTP pool participants are often less stable than that. Nathan Ward wrote:

At least 3 - you need 3 to NTP to do its job and eliminate bad tickers. If you only have 2, you end up with an average between them, even if one is an hour out.

Not my experience. A bad ticker usually just gets de-selected. I've only seen one bad ticker ever - one of the MSL stratum 1 machines lost track of time after a restart, and since the caesium clock was only giving it a clock sync, not an absolute time, the NTP server started handing out answers that were very accurately a couple of minutes out. If your stratum 2 servers are doing the right thing and using multiple good sources, this shouldn't cause a problem downstream. I've never seen an even moderately well configured stratum 2 or more server hand out bad answers. (Lose all sync and give stratum 16 answers, yes. Bad stratum 2+ answers, no.) -- don

On 12/02/14 14:35, Nathan Ward wrote: (stuff) I think what I'm trying to get to is: (1) Time synchronisation to at least second resolution (and preferably much better) is Important. (2) Service providers should provide a robust stratum 2 NTP service to customers. (3) They don't need to do stratum 1. There are reliable, professionally managed NTP services out there available for use. (4) If you operate a local stratum 1 server, you should run a customer facing stratum 2 service, feeding from both local and remote stratum 1 servers. (5) Using NTP pools to synchronise customer facing NTP services is a Very Bad Idea. (6) Using one's service provider's NTP service should be vastly more robust than using the NTP pools. If it's not, that represents a failure on the part of the provider. As I think about this, I'm starting to regard NTP pools as an attractive nuisance - the simplicity of using the pools (and the increasing use of them in preconfigured devices) means that an important service is being provided on unstable systems, often run by amateur operators on a grace and favour basis. That does not bode well for the general stability of applications that require good time. -- don

On Wed, 12 Feb 2014, Don Stokes wrote:

As I think about this, I'm starting to regard NTP pools as an attractive nuisance - the simplicity of using the pools (and the increasing use of them in preconfigured devices) means that an important service is being provided on unstable systems, often run by amateur operators on a grace and favour basis. That does not bode well for the general stability of applications that require good time.

Compared to 10 years ago before the pool.ntp.prg got setup the situation is a lot better. OS vendors provide their own time servers or use the pool while end user devices often get a local server via DHCP. Before then people either didn't have time sync'd or manually configured it with all sorts of problems. Whatever you provide is going to have to be zero-configuration for 99% of end users, which effectively means getting the settings to the DSL modems which use DHCP to get it to network devices. If ISPs can do this reliably then fair enough we will see a drop in usage of pool.ntp.org , but it has to be reliable or we'll end up like the DNS with people manually configuring 8.8.8.8 to get around "ISP X's dodgy DNS servers" . -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.

What are people doing about this? Is anyone planning to blanket whitelist UDP/123 like most residential ISPs currently do for SMTP? Port scanning their customers pre-emptively to see who's vulnerable? Keeping an eye on rate shapers to see odd bursts in upload speed? On Wed, Feb 12, 2014 at 4:18 PM, Simon Lyall wrote:

On Wed, 12 Feb 2014, Don Stokes wrote:

...

As I think about this, I'm starting to regard NTP pools as an attractive nuisance - the simplicity of using the pools (and the increasing use of them in preconfigured devices) means that an important service is being provided on unstable systems, often run by amateur operators on a grace and favour basis. That does not bode well for the general stability of applications that require good time.

Compared to 10 years ago before the pool.ntp.prg got setup the situation is a lot better. OS vendors provide their own time servers or use the pool while end user devices often get a local server via DHCP. Before then people either didn't have time sync'd or manually configured it with all sorts of problems.

Whatever you provide is going to have to be zero-configuration for 99% of end users, which effectively means getting the settings to the DSL modems which use DHCP to get it to network devices.

If ISPs can do this reliably then fair enough we will see a drop in usage of pool.ntp.org , but it has to be reliable or we'll end up like the DNS with people manually configuring 8.8.8.8 to get around "ISP X's dodgy DNS servers" .

-- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Something to do with an old Soekris 4501 https://www.febo.com/pages/soekris/ Dean built one of these http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html On Thu, Feb 13, 2014 at 9:14 AM, Tony Wicks wrote:

On this note, does anyone have recommendations on an affordable GPS clock source ?

*From:* Sam Russell *Sent:* Wednesday, February 12, 2014 4:22 PM *To:* Simon Lyall *Cc:* nznog *Subject:* Re: [nznog] NTP reflection used for world's largest DDoS

What are people doing about this? Is anyone planning to blanket whitelist UDP/123 like most residential ISPs currently do for SMTP? Port scanning their customers pre-emptively to see who's vulnerable? Keeping an eye on rate shapers to see odd bursts in upload speed?

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Tekron is a local Wellington Company that specialises in GPS clocks and NTP servers, http://tekron.com/ The hardware is made in NZ. On Thu, Feb 13, 2014 at 9:14 AM, Tony Wicks wrote:

On this note, does anyone have recommendations on an affordable GPS clock source ?

*From:* Sam Russell *Sent:* Wednesday, February 12, 2014 4:22 PM *To:* Simon Lyall *Cc:* nznog *Subject:* Re: [nznog] NTP reflection used for world's largest DDoS

What are people doing about this? Is anyone planning to blanket whitelist UDP/123 like most residential ISPs currently do for SMTP? Port scanning their customers pre-emptively to see who's vulnerable? Keeping an eye on rate shapers to see odd bursts in upload speed?

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

On Thu, 13 Feb 2014 11:54:19 +1300, Richard Hulse wrote:

On 13/02/2014, at 9:14 am, "Tony Wicks" wrote:

...

On this note, does anyone have recommendations on an affordable GPS clock source ?

http://tekron.com/nts-01

Disclosure: I know one of the directors.

"Request a quote" doesn't usually say "affordable" to me 8^) Do we have any idea what these actually cost? They look pretty nifty (though I can't imagine it'd turn out cheaper than a leftover Soekris and a GPS w/ PPS for my purposes, anyway) -- Michael Fincham System Administrator, Unleash Office: 0800 750 250 DDI: 03 978 1223 Mobile: 027 666 4482

On 13/02/14 09:14, Tony Wicks wrote:

On this note, does anyone have recommendations on an affordable GPS clock source ?

Affordable GPS is easy (depending on your exact definition of "affordable"), as noted in other postings. IME, the hard part is getting the GPS antenna somewhere usable. That basically means outside, and usually (especially in heavily built up areas), on the roof. -- don

On Thu, 13 Feb 2014 12:19:01 +1300, Dean Pemberton wrote:

Here is the little module which connects to the GPIO ports and gives you the NMEA and PPS signals http://www.adafruit.com/products/746

Local (very nice to deal with) supplier: http://nicegear.co.nz/gps/ultimate-gps-breakout-66-channel-w10-hz-updates/ -- Michael

On Sat, 2014-02-15 at 17:47 +1300, Jonathan Brewer wrote:

On Thu, Feb 13, 2014 at 9:14 AM, Tony Wicks wrote:

On this note, does anyone have recommendations on an affordable GPS clock source ?

David Zanetti has an NTP server he designed that will be in the $100 range when produced in small quantities.

[snip]

Perhaps if we all give him a windup he'll get it finished and selling via NiceGear.

I'm sure that if some people decided to assist financially with the development it might go up on his priority queue. ;)

On Feb 12, 2014, at 6:49 AM, Don Stokes wrote:

so I'm starting to wonder if redirecting customer NTP traffic to local NTP servers, and dropping all unauthorised inbound NTP queries at the perimeter isn't completely bad idea.

The problem is that while it can make sense for access and IDC operators to run their own recursive DNS service and by default (with an opt-out proviso) to restrict recursive DNS for access & IDC customers to their own recursive DNS service, ntp is a different kettle of fish. Access to a high number of diverse ntp sources is desirable. The fact that many OSes come with ntp set to symmetric mode (i.e., source & dest UDP/123) is certainly annoying. The good news is that it's pretty easy to identify misconfigured ntpds which are abused for ntp reflection/amplification DDoS attacks: http://www.openntpproject.org/ It's pretty easy to scan your own netblocks, and those of your customers, to identify misconfigured ntpds which an be abused. Those individual ntpds can be S/RTBHed or ACLed until they're remediated, which is much more preferable than wholesale ntp blockage. ----------------------------------------------------------------------- Roland Dobbins // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton

On Feb 11, 2014, at 11:34 AM, Michael Fincham wrote:

Before Roland appears and mentions it, try and avoid putting a stateful firewall in front of stuff :)

Naming calls. ;> Also, it's not just monlist which can be abused, but various level-6/level-7 commands such as monlist, sysstats, showpeer, peers, listpeers, et. al. They should all be disallowed on public interfaces of boxen running various ntpds. ----------------------------------------------------------------------- Roland Dobbins // http://www.arbornetworks.com Luck is the residue of opportunity and design. -- John Milton