Same here on Telstra Fibre. traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets 1 192.168.64.254 1.129 ms 0.843 ms 1.355 ms 2 10.255.1.9 0.512 ms 0.482 ms 0.500 ms 3 192.168.255.1 0.855 ms 0.693 ms 0.702 ms 4 203.97.40.1 18.487 ms 6.760 ms 3.610 ms 5 218.101.61.50 2.832 ms 2.457 ms 2.185 ms 6 203.98.50.2 11.479 ms 11.595 ms 11.396 ms 7 203.98.50.251 12.533 ms 12.012 ms 12.252 ms 8 203.98.42.18 12.438 ms 12.381 ms 12.312 ms 9 203.109.130.33 13.124 ms 13.500 ms 13.398 ms 10 * * * 11 203.109.130.6 175.716 ms 173.951 ms 172.882 ms 12 209.85.249.3 191.824 ms 192.721 ms 209.85.249.5 191.274 ms 13 209.85.250.64 191.326 ms 209.85.250.66 191.357 ms 191.684 ms 14 216.239.49.198 195.631 ms 72.14.232.63 199.293 ms 216.239.49.198 195.777 ms 15 72.14.233.200 202.215 ms 72.14.233.138 198.278 ms 198.168 ms 16 216.239.48.165 190.833 ms 64.233.174.131 193.692 ms 64.233.174.129 194.517 ms 17 * * * 18 8.8.8.8 190.718 ms 190.846 ms 190.563 ms And on Telstra wholesale traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets 1 218.185.224.129 0.594 ms 0.581 ms 0.570 ms 2 218.185.224.65 1.111 ms 1.109 ms 1.098 ms 3 203.167.136.129 15.814 ms 15.841 ms 15.977 ms 4 203.98.42.18 15.974 ms 16.098 ms 16.096 ms 5 203.109.130.33 17.088 ms 17.087 ms 17.083 ms 6 * * * 7 203.109.130.6 166.600 ms 166.643 ms 166.622 ms 8 209.85.249.5 144.881 ms 144.961 ms 144.418 ms 9 209.85.250.64 167.404 ms 167.395 ms 209.85.250.60 167.722 ms 10 216.239.49.198 167.142 ms 72.14.232.63 160.291 ms 160.214 ms 11 72.14.233.140 161.390 ms 72.14.233.200 185.323 ms 72.14.233.202 160.620 ms 12 216.239.48.167 163.625 ms 163.100 ms 216.239.48.165 186.152 ms 13 8.8.8.8 163.081 ms 163.295 ms 164.068 ms On 17/09/2013, at 8:46 AM, Stephen Farrar wrote:

Hi Matthew,

Not sure if this helps at all, but I get similar results from a TCL Cable connection here in Wellington:

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets 1 gateway (192.168.1.1) 0.956 ms 0.697 ms 0.635 ms 2 * * * 3 lo0.internet.ivpn.pe19.telstraclear.net (218.101.61.100) 11.469 ms 11.214 ms 9.799 ms 4 ie2-g-0-0-0.telstraclear.net (203.98.50.2) 18.642 ms 20.183 ms 18.590 ms 5 ge-0-2-0-1.xcore1.acld.telstraclear.net (203.98.50.251) 20.239 ms 23.168 ms 21.402 ms 6 ge-0-2-0-914.xcore2.acld.telstraclear.net (203.98.42.18) 20.222 ms 20.372 ms 37.712 ms 7 gi1-2.akl-grafton-bdr2.ihug.net (203.109.130.33) 32.948 ms 20.299 ms 19.991 ms 8 * * * 9 tu20232901.akl-grafton-core1.ihug.net (203.109.130.6) 150.555 ms 151.977 ms 151.098 ms 10 209.85.249.3 (209.85.249.3) 198.609 ms 200.060 ms 199.597 ms 11 209.85.250.64 (209.85.250.64) 199.295 ms 209.85.250.66 (209.85.250.66) 195.563 ms 209.85.250.64 (209.85.250.64) 198.689 ms 12 72.14.232.63 (72.14.232.63) 203.499 ms 203.307 ms 216.239.49.198 (216.239.49.198) 200.577 ms 13 72.14.233.200 (72.14.233.200) 206.175 ms 72.14.233.140 (72.14.233.140) 207.716 ms 209.516 ms 14 64.233.174.131 (64.233.174.131) 198.878 ms 216.239.48.167 (216.239.48.167) 198.319 ms 216.239.48.165 (216.239.48.165) 199.101 ms 15 * * * 16 google-public-dns-a.google.com (8.8.8.8) 433.268 ms 202.334 ms 198.946 ms

For reference, I'm on their 'WarpSpeed' 130Mbps connection.

Good luck getting it sorted!

Cheers, Stephen

On 17 September 2013 08:25, Matthew Poole wrote: Hi all

Breaking with recent tradition and discussing something related to the operation of the NZ tubes...

My employer has a TVH business fibre connection, and since around midnight Saturday/Sunday the latency to 8.8.8.8 (one of my Smokeping test subjects) has been consistently > 160ms. mtr tells me: |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | firewall - 0 | 124 | 124 | 0 | 0 | 1 | 0 | | 203.167.222.5 - 1 | 120 | 119 | 0 | 2 | 51 | 51 | | ge-0-2-0-1.xcore1.acld.telstraclear.net - 0 | 124 | 124 | 0 | 3 | 62 | 1 | |ge-0-2-0-914.xcore2.acld.telstraclear.net - 0 | 124 | 124 | 0 | 3 | 48 | 2 | | gi1-2.akl-grafton-bdr2.ihug.net - 0 | 124 | 124 | 1 | 2 | 29 | 11 | | No response from host - 100 | 24 | 0 | 0 | 0 | 0 | 0 | | tu20232901.akl-grafton-core1.ihug.net - 0 | 124 | 124 | 129 | 132 | 192 | 164 | | 209.85.249.5 - 0 | 124 | 124 | 176 | 178 | 242 | 200 | | 209.85.250.63 - 0 | 124 | 124 | 176 | 187 | 252 | 195 | | 72.14.232.63 - 0 | 124 | 124 | 181 | 183 | 246 | 199 | | 72.14.233.200 - 0 | 124 | 124 | 189 | 191 | 251 | 213 | | 216.239.48.165 - 0 | 124 | 124 | 179 | 182 | 244 | 197 | | No response from host - 100 | 24 | 0 | 0 | 0 | 0 | 0 | | google-public-dns-a.google.com - 0 | 124 | 124 | 179 | 183 | 246 | 200 | |________________________________________________|______|______|______|______|______|______|

The sudden latency jump at akl-grafton-core1.ihug.net kinda leaps out. Is anyone else seeing this? And is this something that the TVH admins are aware of? Please don't make me deal with helpdesk!

Cheers

-- Matthew Poole "The difference between theory and practice is that practice is easier in theory than theory is in practice" _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

It's merely a network status probe for me, but the sudden leap in latency which has now been sustained for ~50 hours doesn't point to something healthy. When my ping to 8.8.8.8 was sitting around 120ms I could accept that as ordinary (if very average) connection latency. Now it's sitting another 40+ms higher. Until the weekend my latency right to the end was about the same as the latency I'm now seeing to the intermediate hop where it suddenly gets ugly. If it's simply a routing change so be it, but it's one that's introduced a whole lot of extra latency. On 17/09/2013 09:02, Liam Farr wrote:

You lot sure its not just being served from the USA rather than domestically or Sydney. If so I don't think there is any real problem with this. Not an optimised route sure, but that's Telstra/voda/hug's prerogative.

If your using it for DNS resolution (its intended purpose), don't. Due to the anycast nature of 8.8.8.8 using it will break most regional CDN's and stuff like your Akamai will end up being served from somewhere close by like Singapore.

Is there any real impact to your service, or have you just noticed "an interesting change in routing"?

-- Liam Farr +64-22-6107884 +64-27-5222624

Sent from my iPhone

On 17/09/2013, at 8:55 AM, Tim Price mailto:tim(a)initech.co.nz> wrote:

...

Same here on Telstra Fibre.

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets 1 192.168.64.254 1.129 ms 0.843 ms 1.355 ms 2 10.255.1.9 0.512 ms 0.482 ms 0.500 ms 3 192.168.255.1 0.855 ms 0.693 ms 0.702 ms 4 203.97.40.1 18.487 ms 6.760 ms 3.610 ms 5 218.101.61.50 2.832 ms 2.457 ms 2.185 ms 6 203.98.50.2 11.479 ms 11.595 ms 11.396 ms 7 203.98.50.251 12.533 ms 12.012 ms 12.252 ms 8 203.98.42.18 12.438 ms 12.381 ms 12.312 ms 9 203.109.130.33 13.124 ms 13.500 ms 13.398 ms 10 * * * 11 203.109.130.6 175.716 ms 173.951 ms 172.882 ms 12 209.85.249.3 191.824 ms 192.721 ms 209.85.249.5 191.274 ms 13 209.85.250.64 191.326 ms 209.85.250.66 191.357 ms 191.684 ms 14 216.239.49.198 195.631 ms 72.14.232.63 199.293 ms 216.239.49.198 195.777 ms 15 72.14.233.200 202.215 ms 72.14.233.138 198.278 ms 198.168 ms 16 216.239.48.165 190.833 ms 64.233.174.131 193.692 ms 64.233.174.129 194.517 ms 17 * * * 18 8.8.8.8 190.718 ms 190.846 ms 190.563 ms

And on Telstra wholesale

traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets 1 218.185.224.129 0.594 ms 0.581 ms 0.570 ms 2 218.185.224.65 1.111 ms 1.109 ms 1.098 ms 3 203.167.136.129 15.814 ms 15.841 ms 15.977 ms 4 203.98.42.18 15.974 ms 16.098 ms 16.096 ms 5 203.109.130.33 17.088 ms 17.087 ms 17.083 ms 6 * * * 7 203.109.130.6 166.600 ms 166.643 ms 166.622 ms 8 209.85.249.5 144.881 ms 144.961 ms 144.418 ms 9 209.85.250.64 167.404 ms 167.395 ms 209.85.250.60 167.722 ms 10 216.239.49.198 167.142 ms 72.14.232.63 160.291 ms 160.214 ms 11 72.14.233.140 161.390 ms 72.14.233.200 185.323 ms 72.14.233.202 160.620 ms 12 216.239.48.167 163.625 ms 163.100 ms 216.239.48.165 186.152 ms 13 8.8.8.8 163.081 ms 163.295 ms 164.068 ms

On 17/09/2013, at 8:46 AM, Stephen Farrar mailto:stephen(a)sf.net.nz> wrote:

...

Hi Matthew,

Not sure if this helps at all, but I get similar results from a TCL Cable connection here in Wellington:

traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets 1 gateway (192.168.1.1) 0.956 ms 0.697 ms 0.635 ms 2 * * * 3 lo0.internet.ivpn.pe19.telstraclear.net http://lo0.internet.ivpn.pe19.telstraclear.net/ (218.101.61.100) 11.469 ms 11.214 ms 9.799 ms 4 ie2-g-0-0-0.telstraclear.net http://ie2-g-0-0-0.telstraclear.net/ (203.98.50.2) 18.642 ms 20.183 ms 18.590 ms 5 ge-0-2-0-1.xcore1.acld.telstraclear.net http://ge-0-2-0-1.xcore1.acld.telstraclear.net/ (203.98.50.251) 20.239 ms 23.168 ms 21.402 ms 6 ge-0-2-0-914.xcore2.acld.telstraclear.net http://ge-0-2-0-914.xcore2.acld.telstraclear.net/ (203.98.42.18) 20.222 ms 20.372 ms 37.712 ms 7 gi1-2.akl-grafton-bdr2.ihug.net http://gi1-2.akl-grafton-bdr2.ihug.net/ (203.109.130.33) 32.948 ms 20.299 ms 19.991 ms 8 * * * 9 tu20232901.akl-grafton-core1.ihug.net http://tu20232901.akl-grafton-core1.ihug.net/ (203.109.130.6) 150.555 ms 151.977 ms 151.098 ms 10 209.85.249.3 (209.85.249.3) 198.609 ms 200.060 ms 199.597 ms 11 209.85.250.64 (209.85.250.64) 199.295 ms 209.85.250.66 (209.85.250.66) 195.563 ms 209.85.250.64 (209.85.250.64) 198.689 ms 12 72.14.232.63 (72.14.232.63) 203.499 ms 203.307 ms 216.239.49.198 (216.239.49.198) 200.577 ms 13 72.14.233.200 (72.14.233.200) 206.175 ms 72.14.233.140 (72.14.233.140) 207.716 ms 209.516 ms 14 64.233.174.131 (64.233.174.131) 198.878 ms 216.239.48.167 (216.239.48.167) 198.319 ms 216.239.48.165 (216.239.48.165) 199.101 ms 15 * * * 16 google-public-dns-a.google.com http://google-public-dns-a.google.com/ (8.8.8.8) 433.268 ms 202.334 ms 198.946 ms

For reference, I'm on their 'WarpSpeed' 130Mbps connection.

Good luck getting it sorted!

Cheers, Stephen

On 17 September 2013 08:25, Matthew Poole mailto:matt(a)p00le.net> wrote:

Hi all

Breaking with recent tradition and discussing something related to the operation of the NZ tubes...

My employer has a TVH business fibre connection, and since around midnight Saturday/Sunday the latency to 8.8.8.8 (one of my Smokeping test subjects) has been consistently > 160ms. mtr tells me: |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | firewall - 0 | 124 | 124 | 0 | 0 | 1 | 0 | | 203.167.222.5 - 1 | 120 | 119 | 0 | 2 | 51 | 51 | | ge-0-2-0-1.xcore1.acld.telstraclear.net http://ge-0-2-0-1.xcore1.acld.telstraclear.net/ - 0 | 124 | 124 | 0 | 3 | 62 | 1 | |ge-0-2-0-914.xcore2.acld.telstraclear.net http://ge-0-2-0-914.xcore2.acld.telstraclear.net/ - 0 | 124 | 124 | 0 | 3 | 48 | 2 | | gi1-2.akl-grafton-bdr2.ihug.net http://gi1-2.akl-grafton-bdr2.ihug.net/ - 0 | 124 | 124 | 1 | 2 | 29 | 11 | | No response from host - 100 | 24 | 0 | 0 | 0 | 0 | 0 | | tu20232901.akl-grafton-core1.ihug.net http://tu20232901.akl-grafton-core1.ihug.net/ - 0 | 124 | 124 | 129 | 132 | 192 | 164 | | 209.85.249.5 - 0 | 124 | 124 | 176 | 178 | 242 | 200 | | 209.85.250.63 tel:209.85.250.63 - 0 | 124 | 124 | 176 | 187 | 252 | 195 | | 72.14.232.63 - 0 | 124 | 124 | 181 | 183 | 246 | 199 | | 72.14.233.200 - 0 | 124 | 124 | 189 | 191 | 251 | 213 | | 216.239.48.165 - 0 | 124 | 124 | 179 | 182 | 244 | 197 | | No response from host - 100 | 24 | 0 | 0 | 0 | 0 | 0 | | google-public-dns-a.google.com http://google-public-dns-a.google.com/ - 0 | 124 | 124 | 179 | 183 | 246 | 200 | |________________________________________________|______|______|______|______|______|______|

The sudden latency jump at akl-grafton-core1.ihug.net http://akl-grafton-core1.ihug.net/ kinda leaps out. Is anyone else seeing this? And is this something that the TVH admins are aware of? Please don't make me deal with helpdesk!

Cheers

-- Matthew Poole "The difference between theory and practice is that practice is easier in theory than theory is in practice" _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Matthew Poole "The difference between theory and practice is that practice is easier in theory than theory is in practice"

On Tue, Sep 17, 2013 at 09:12:22AM +1200, Joel Wirāmu Pauling wrote:

Using anything at googles SDN managed ip network as a metric/benchmark is foolish at best.

My first thought was that this is kind of off-topic for NZNOG. But I think it would be valid information for what hosts are actually good to test to. And I don't know if there's any kind of worldwide list of sites that are ok with being monitored by a lot of people, and want to publicise... I'm sure that it's very common to receive a lot of pings to 8.8.8.8, 4.2.2.2, www.google.com, www.facebook.com etc. but these are either anycast or give different ip's depending on source. It's reasonably common to want to be able to monitor performance, but to me it's packet loss that is the real concern rather than ping time as packet loss can easily translate into delays and timeouts. Ben.

I can confirm the following from my home ADSL (Through my $DAYJOB, we have VodaClear as our main upstream provider) 3 113.197.97.1 (113.197.97.1) 24.178 ms 39.475 ms 23.745 ms 4 103.9.216.18 (103.9.216.18) 21.193 ms 21.312 ms 174.437 ms 5 103.9.216.17 (103.9.216.17) 21.224 ms 21.651 ms 21.355 ms 6 203.167.223.89 (203.167.223.89) 23.213 ms 21.759 ms 21.567 ms 7 gi1-2.akl-grafton-bdr2.ihug.net (203.109.130.33) 23.224 ms 23.269 ms 22.376 ms 8 * * * 9 tu20232901.akl-grafton-core1.ihug.net (203.109.130.6) 157.319 ms 163.857 ms 153.209 ms 10 209.85.249.5 (209.85.249.5) 174.993 ms 176.616 ms 209.85.249.3 (209.85.249.3) 154.141 ms 11 209.85.250.63 (209.85.250.63) 152.099 ms 209.85.250.66 (209.85.250.66) 151.924 ms 209.85.250.64 (209.85.250.64) 154.291 ms 12 216.239.49.198 (216.239.49.198) 174.872 ms 167.880 ms 72.14.232.63 (72.14.232.63) 336.555 ms 13 72.14.233.140 (72.14.233.140) 173.993 ms 72.14.233.200 (72.14.233.200) 175.369 ms 72.14.233.138 (72.14.233.138) 175.282 ms 14 216.239.48.167 (216.239.48.167) 194.713 ms 64.233.174.131 (64.233.174.131) 202.080 ms 216.239.48.167 (216.239.48.167) 202.312 ms 15 * * * 16 google-public-dns-a.google.com (8.8.8.8) 170.125 ms 341.309 ms 170.395 ms I am positing (and I am by no means using any known info) that this could be a change related to the Vodafone ownership. Can anyone on "Vodafone" ADSL (actually from vodafone) or someone with a vodafone mobile (I can't check with mine for a little while) give us a trace route so we can see their path? Regards Alexander On 17/09/2013, at 9:59 AM, "Tony Wicks" wrote:

If this isn't on topic (one of the major transit provides routing) nothing is and we should pack up the list and go home. One useful aspect of this list is to allow people to ask if an issue (perceived or otherwise) is a general one or just effects them or their providers. In this case, seeing as no one else has confirmed it for the OP, I can confirm that traffic to 8.8.8.8 via the Global Gateway, Vocus and CallPlus networks is still routing to Sydney at sub 30ms rates. I would suggest that the OP (and others if desired) might want to log a call with Vodafone and ask them to look at it.

cheers

-----Original Message----- From: Ben Sent: Tuesday, September 17, 2013 9:31 AM To: JoelWirāmuPauling Cc: Nznog List Subject: Re: [nznog] TelstraVodaHug core lag

On Tue, Sep 17, 2013 at 09:12:22AM +1200, Joel Wirāmu Pauling wrote:

...

Using anything at googles SDN managed ip network as a metric/benchmark is foolish at best.

My first thought was that this is kind of off-topic for NZNOG. But I think it would be valid information for what hosts are actually good to test to. And I don't know if there's any kind of worldwide list of sites that are ok with being monitored by a lot of people, and want to publicise...

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Not for me... Voda ADSL near Chch... 3 be3-100.bras1ctp.chc.vf.net.nz.129.109.203.in-addr.arpa (203.109.129.145) 21.380 ms 21.987 ms 22.691 ms 4 203-118-190-12.dsl.dyn.ihug.co.nz (203.118.190.12) 23.536 ms 23.626 ms 23.876 ms 5 203-118-190-13.dsl.dyn.ihug.co.nz (203.118.190.13) 25.989 ms 13.190.109.203.core.vf.net.nz (203.109.190.13) 26.125 ms 203-118-190-13.dsl.dyn.ihug.co.nz (203.118.190.13) 27.048 ms 6 te0-0-0-1.ppnzctp02.chc.vf.net.nz.182.109.203.in-addr.arpa (203.109.182.17) 27.734 ms 23.282 ms 23.941 ms 7 ggl-router.syd.vf.net.nz.130.109.203.in-addr.arpa (203.109.130.2) 136.410 ms 59.936 ms 116.548 ms 8 72.14.237.21 (72.14.237.21) 61.837 ms 59.950 ms 60.526 ms 9 google-public-dns-a.google.com (8.8.8.8) 61.534 ms 62.472 ms 63.684 ms On Tue, 2013-09-17 at 10:07 +1200, Alexander Neilson wrote:

I can confirm the following from my home ADSL (Through my $DAYJOB, we have VodaClear as our main upstream provider)

3 113.197.97.1 (113.197.97.1) 24.178 ms 39.475 ms 23.745 ms 4 103.9.216.18 (103.9.216.18) 21.193 ms 21.312 ms 174.437 ms 5 103.9.216.17 (103.9.216.17) 21.224 ms 21.651 ms 21.355 ms 6 203.167.223.89 (203.167.223.89) 23.213 ms 21.759 ms 21.567 ms 7 gi1-2.akl-grafton-bdr2.ihug.net (203.109.130.33) 23.224 ms 23.269 ms 22.376 ms 8 * * * 9 tu20232901.akl-grafton-core1.ihug.net (203.109.130.6) 157.319 ms 163.857 ms 153.209 ms 10 209.85.249.5 (209.85.249.5) 174.993 ms 176.616 ms 209.85.249.3 (209.85.249.3) 154.141 ms 11 209.85.250.63 (209.85.250.63) 152.099 ms 209.85.250.66 (209.85.250.66) 151.924 ms 209.85.250.64 (209.85.250.64) 154.291 ms 12 216.239.49.198 (216.239.49.198) 174.872 ms 167.880 ms 72.14.232.63 (72.14.232.63) 336.555 ms 13 72.14.233.140 (72.14.233.140) 173.993 ms 72.14.233.200 (72.14.233.200) 175.369 ms 72.14.233.138 (72.14.233.138) 175.282 ms 14 216.239.48.167 (216.239.48.167) 194.713 ms 64.233.174.131 (64.233.174.131) 202.080 ms 216.239.48.167 (216.239.48.167) 202.312 ms 15 * * * 16 google-public-dns-a.google.com (8.8.8.8) 170.125 ms 341.309 ms 170.395 ms

I am positing (and I am by no means using any known info) that this could be a change related to the Vodafone ownership. Can anyone on "Vodafone" ADSL (actually from vodafone) or someone with a vodafone mobile (I can't check with mine for a little while) give us a trace route so we can see their path?

Regards Alexander

On 17/09/2013, at 9:59 AM, "Tony Wicks" wrote:

...

If this isn't on topic (one of the major transit provides routing) nothing is and we should pack up the list and go home. One useful aspect of this list is to allow people to ask if an issue (perceived or otherwise) is a general one or just effects them or their providers. In this case, seeing as no one else has confirmed it for the OP, I can confirm that traffic to 8.8.8.8 via the Global Gateway, Vocus and CallPlus networks is still routing to Sydney at sub 30ms rates. I would suggest that the OP (and others if desired) might want to log a call with Vodafone and ask them to look at it.

cheers

-----Original Message----- From: Ben Sent: Tuesday, September 17, 2013 9:31 AM To: JoelWirāmuPauling Cc: Nznog List Subject: Re: [nznog] TelstraVodaHug core lag

On Tue, Sep 17, 2013 at 09:12:22AM +1200, Joel Wirāmu Pauling wrote:

...

Using anything at googles SDN managed ip network as a metric/benchmark is foolish at best.

My first thought was that this is kind of off-topic for NZNOG. But I think it would be valid information for what hosts are actually good to test to. And I don't know if there's any kind of worldwide list of sites that are ok with being monitored by a lot of people, and want to publicise...

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Steve Holdoway BSc(Hons) MIITP http://www.greengecko.co.nz Linkedin: http://www.linkedin.com/in/steveholdoway Skype: sholdowa

On 17/09/2013, at 4:04 PM, Peter Lambrechtsen wrote:

I’ve forwarded this email trail onto a friend of mine at Voda…. I’ll let you know if I hear anything back.

Its still the only proven way to solve problems with telcos. Don't forget beer. regards Peter Mott LocalCloud Limited Business Critical Application Hosting +64 9 280 0925 -/-

8.8.8.8 is an anycast address that appears in multiple locations around the world and usually you'll find it take the best path. the best path to 8.8.8.8 may actually be via the US as a lot of providers in NZ have had bandwidth problems to Australia from excessive Youtube and other complications. although 8.8.8.8 has had a sydney location for a while, at times it seems to shift to somewhere else in asia, and even when you do a lookup you'll find that the latency is often ~180 msec. i don't think they actually do lookups from sydney, it's just a small cache. and if australia asia performance is anything like new zealand asia performance there will be significant performance degredation making people not wanting to use it, meaning the cache pool will be small. i think the dns lookups originate from taiwan, but i could be mistaken. if you want a static probe you should probably use something static, like telstra's sydney dns server rather than google. try this address instead: 61.9.194.49 Ben. On Tue, Sep 17, 2013 at 09:10:11AM +1200, Matthew Poole wrote:

It's merely a network status probe for me, but the sudden leap in latency which has now been sustained for ~50 hours doesn't point to something healthy. When my ping to 8.8.8.8 was sitting around 120ms I could accept that as ordinary (if very average) connection latency. Now it's sitting another 40+ms higher. Until the weekend my latency right to the end was about the same as the latency I'm now seeing to the intermediate hop where it suddenly gets ugly. If it's simply a routing change so be it, but it's one that's introduced a whole lot of extra latency.

On 17/09/2013 09:02, Liam Farr wrote:

You lot sure its not just being served from the USA rather than domestically or Sydney. If so I don't think there is any real problem with this. Not an optimised route sure, but that's Telstra/voda/hug's prerogative. If your using it for DNS resolution (its intended purpose), don't. Due to the anycast nature of 8.8.8.8 using it will break most regional CDN's and stuff like your Akamai will end up being served from somewhere close by like Singapore. Is there any real impact to your service, or have you just noticed "an interesting change in routing"? -- Liam Farr +64-22-6107884 +64-27-5222624 Sent from my iPhone On 17/09/2013, at 8:55 AM, Tim Price wrote:

Same here on Telstra Fibre. traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets 1 192.168.64.254 1.129 ms 0.843 ms 1.355 ms 2 10.255.1.9 0.512 ms 0.482 ms 0.500 ms 3 192.168.255.1 0.855 ms 0.693 ms 0.702 ms 4 203.97.40.1 18.487 ms 6.760 ms 3.610 ms 5 218.101.61.50 2.832 ms 2.457 ms 2.185 ms 6 203.98.50.2 11.479 ms 11.595 ms 11.396 ms 7 203.98.50.251 12.533 ms 12.012 ms 12.252 ms 8 203.98.42.18 12.438 ms 12.381 ms 12.312 ms 9 203.109.130.33 13.124 ms 13.500 ms 13.398 ms 10 * * * 11 203.109.130.6 175.716 ms 173.951 ms 172.882 ms 12 209.85.249.3 191.824 ms 192.721 ms 209.85.249.5 191.274 ms 13 209.85.250.64 191.326 ms 209.85.250.66 191.357 ms 191.684 ms 14 216.239.49.198 195.631 ms 72.14.232.63 199.293 ms 216.239.49.198 195.777 ms 15 72.14.233.200 202.215 ms 72.14.233.138 198.278 ms 198.168 ms 16 216.239.48.165 190.833 ms 64.233.174.131 193.692 ms 64.233.174.129 194.517 ms 17 * * * 18 8.8.8.8 190.718 ms 190.846 ms 190.563 ms And on Telstra wholesale traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets 1 218.185.224.129 0.594 ms 0.581 ms 0.570 ms 2 218.185.224.65 1.111 ms 1.109 ms 1.098 ms 3 203.167.136.129 15.814 ms 15.841 ms 15.977 ms 4 203.98.42.18 15.974 ms 16.098 ms 16.096 ms 5 203.109.130.33 17.088 ms 17.087 ms 17.083 ms 6 * * * 7 203.109.130.6 166.600 ms 166.643 ms 166.622 ms 8 209.85.249.5 144.881 ms 144.961 ms 144.418 ms 9 209.85.250.64 167.404 ms 167.395 ms 209.85.250.60 167.722 ms 10 216.239.49.198 167.142 ms 72.14.232.63 160.291 ms 160.214 ms 11 72.14.233.140 161.390 ms 72.14.233.200 185.323 ms 72.14.233.202 160.620 ms 12 216.239.48.167 163.625 ms 163.100 ms 216.239.48.165 186.152 ms 13 8.8.8.8 163.081 ms 163.295 ms 164.068 ms On 17/09/2013, at 8:46 AM, Stephen Farrar wrote:

Hi Matthew, Not sure if this helps at all, but I get similar results from a TCL Cable connection here in Wellington: traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets 1 gateway (192.168.1.1) 0.956 ms 0.697 ms 0.635 ms 2 * * * 3 lo0.internet.ivpn.pe19.telstraclear.net (218.101.61.100) 11.469 ms 11.214 ms 9.799 ms 4 ie2-g-0-0-0.telstraclear.net (203.98.50.2) 18.642 ms 20.183 ms 18.590 ms 5 ge-0-2-0-1.xcore1.acld.telstraclear.net (203.98.50.251) 20.239 ms 23.168 ms 21.402 ms 6 ge-0-2-0-914.xcore2.acld.telstraclear.net (203.98.42.18) 20.222 ms 20.372 ms 37.712 ms 7 gi1-2.akl-grafton-bdr2.ihug.net (203.109.130.33) 32.948 ms 20.299 ms 19.991 ms 8 * * * 9 tu20232901.akl-grafton-core1.ihug.net (203.109.130.6) 150.555 ms 151.977 ms 151.098 ms 10 209.85.249.3 (209.85.249.3) 198.609 ms 200.060 ms 199.597 ms 11 209.85.250.64 (209.85.250.64) 199.295 ms 209.85.250.66 (209.85.250.66) 195.563 ms 209.85.250.64 (209.85.250.64) 198.689 ms 12 72.14.232.63 (72.14.232.63) 203.499 ms 203.307 ms 216.239.49.198 (216.239.49.198) 200.577 ms 13 72.14.233.200 (72.14.233.200) 206.175 ms 72.14.233.140 (72.14.233.140) 207.716 ms 209.516 ms 14 64.233.174.131 (64.233.174.131) 198.878 ms 216.239.48.167 (216.239.48.167) 198.319 ms 216.239.48.165 (216.239.48.165) 199.101 ms 15 * * * 16 google-public-dns-a.google.com (8.8.8.8) 433.268 ms 202.334 ms 198.946 ms For reference, I'm on their 'WarpSpeed' 130Mbps connection. Good luck getting it sorted! Cheers, Stephen

On 17 September 2013 08:25, Matthew Poole wrote:

Hi all

Breaking with recent tradition and discussing something related to the operation of the NZ tubes...

My employer has a TVH business fibre connection, and since around midnight Saturday/Sunday the latency to 8.8.8.8 (one of my Smokeping test subjects) has been consistently > 160ms. mtr tells me: |------------------------------------------------------------------------------------------| | WinMTR statistics | | Host - % | Sent | Recv | Best | Avrg | Wrst | Last | |------------------------------------------------|------|------|------|------|------|------| | firewall - 0 | 124 | 124 | 0 | 0 | 1 | 0 | | 203.167.222.5 - 1 | 120 | 119 | 0 | 2 | 51 | 51 | | ge-0-2-0-1.xcore1.acld.telstraclear.net - 0 | 124 | 124 | 0 | 3 | 62 | 1 | |ge-0-2-0-914.xcore2.acld.telstraclear.net - 0 | 124 | 124 | 0 | 3 | 48 | 2 | | gi1-2.akl-grafton-bdr2.ihug.net - 0 | 124 | 124 | 1 | 2 | 29 | 11 | | No response from host - 100 | 24 | 0 | 0 | 0 | 0 | 0 | | tu20232901.akl-grafton-core1.ihug.net - 0 | 124 | 124 | 129 | 132 | 192 | 164 | | 209.85.249.5 - 0 | 124 | 124 | 176 | 178 | 242 | 200 | | 209.85.250.63 - 0 | 124 | 124 | 176 | 187 | 252 | 195 | | 72.14.232.63 - 0 | 124 | 124 | 181 | 183 | 246 | 199 | | 72.14.233.200 - 0 | 124 | 124 | 189 | 191 | 251 | 213 | | 216.239.48.165 - 0 | 124 | 124 | 179 | 182 | 244 | 197 | | No response from host - 100 | 24 | 0 | 0 | 0 | 0 | 0 | | google-public-dns-a.google.com - 0 | 124 | 124 | 179 | 183 | 246 | 200 | |________________________________________________|______|______|______|______|______|______|

The sudden latency jump at akl-grafton-core1.ihug.net kinda leaps out. Is anyone else seeing this? And is this something that the TVH admins are aware of? Please don't make me deal with helpdesk!

Cheers

-- Matthew Poole "The difference between theory and practice is that practice is easier in theory than theory is in practice" _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

-- Matthew Poole "The difference between theory and practice is that practice is easier in theory than theory is in practice"

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

On 17/09/2013, at 4:45 PM, Jonathan Brewer wrote:

Keep in mind:

1.) As Ben mentioned, 8.8.8.8 appears a great number of places on the Internet; your path is unlikely to be the same as Matthew's 2.) Do not assume that Vodafone's network has anything to do with TelstraClear's network.

Otherwise an interesting thread. The tin-foil-hat wearer in me says to always be suspicious of sudden increases in latency over known paths. Port mirroring doesn't add latency, but if you have to re-route traffic to a box with a spare port to keep the TLAs happy...

I was thinking of replying to this and subtly implying that that's exactly what it was, to see who bit. Isn't it an FLA these days, though? :-) Not sure how you'd get that much latency though - maybe round trip to Waihopai, twice or maybe even thrice (to catch any packets they missed the first time)? Oh or a single round trip to Pine Gap or something? Anyway, reality is large networks do changes that are sometimes in several stages over a number of days, and sometimes result in less than optimal routing for a period of time, often for a sub-set of the network. Given the networks in question, you'd expect that there's some of these sorts of changes going on, related to improving on Jon's point no. 2 above. Pretty sure what you're seeing is Google learning the route for the network you're in over a link in the US. Certainly your outbound path is going that way - the latency step of about 130ms before leaving the network is enough evidence of that without me even needing to check routing tables. Ignore the reverse DNS bits. Check the same latency again tomorrow, would be my informed guess. :-) As always, if you have performance impacting or functionality problems you'll want to call your ISP's support guys so they know and can pass on any customer impacting issues. Sounds like things are working OK though, yeah? As has been mentioned by others, don't use 8.8.8.8 for DNS if you're in NZ because Akamai will make your life hard. Using these sort of things for latency or connectivity measurements is probably OK, but just like spam filtering, use them all together to avoid false positives. -- Nathan Ward

Your ISP's DNS servers? On 18/09/2013, at 7:32 PM, Juha Saarinen wrote:

Out of curiosity, what would be recommended alternative to 8.8.8.8 for casual DNS serving in NZ? -- Juha Saarinen twitter: juhasaarinen

On 18/09/2013, at 6:58 PM, Nathan Ward wrote:

...

As has been mentioned by others, don't use 8.8.8.8 for DNS if you're in NZ because Akamai will make your life hard.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

On 18/09/2013, at 7:39 PM, Nathan Ward wrote:

Out of curiosity, what is "casual DNS serving"? :P

A "technical term". Will explain at the next NZNOG meet if I ever get to go to one...

I assume you mean to ask what a good open DNS recurser is? (recursor?) Let me turn that around - what are you trying to achieve that you don't by using your ISP's recursive DNS server? (that's a much better than recurser)

For those situations when either the ISP is known, but there are DNS problems, or when it isn't, and you'd want to try with another server. One of those things that I've come across a great many times lately, with subcontractors putting in 8.8.8.8 etc as a QAD fix. -- Juha

On 18/09/2013, at 8:10 PM, Juha Saarinen wrote:

On 18/09/2013, at 7:39 PM, Nathan Ward wrote:

...

Out of curiosity, what is "casual DNS serving"? :P

A "technical term". Will explain at the next NZNOG meet if I ever get to go to one…

So what you're saying is, you made it up and are never going to tell us? :-P

...

I assume you mean to ask what a good open DNS recurser is? (recursor?) Let me turn that around - what are you trying to achieve that you don't by using your ISP's recursive DNS server? (that's a much better than recurser)

For those situations when either the ISP is known, but there are DNS problems, or when it isn't, and you'd want to try with another server.

I use 8.8.8.8 for quick hacks like that, then I turn it off again.

One of those things that I've come across a great many times lately, with subcontractors putting in 8.8.8.8 etc as a QAD fix.

I guess it's better than them putting in another ISPs DNS servers. There's a lot of people doing that. Some ISPs even put other ISPs' DNS servers in their BRASes.

On 18/09/13 18:58, Nathan Ward wrote:

As has been mentioned by others, don't use 8.8.8.8 for DNS if you're in NZ because Akamai will make your life hard.

Random thought ... how bad would it be for a service provider to locally redirect 8.8.8.8/32 into their own DNS forwarders? I mean it would probably violate the Principle of Least Astonishment, at least for folk who are paying attention, but random users who have been told that "it's better to use 8.8.8.8" without fully understanding the implications might get better performance... -- don

On 18/09/2013, at 7:46 PM, Tim Price wrote:

Apart from the obvious net neutrality rhetoric why would you have a stateful device in your forwarding path. And you'd better hope that google don't decide to do authoritive hosting on those servers now or in the future (although we all know they shouldn't)

Yeah, why /would/ you have a stateful device in your forwarding path, in order to achieve Don's suggestion? :-) Loopback/dummy interface + static route. -- Nathan Ward

We already do this with the blackhole-*.iana.org addresses - we just announce the /32 routes into our internal routing table, just like any other internal service IP address. All I need to do is add 8.8.8.8 to my local forwarders, and they'll anycast the address to our downstreams. In the blackhole*.iana.org case we're doing something recommended to prevent bogus traffic from going outside our provider net, not hijacking another provider's service. That's the philosophical problem - it's not "can I do it without breaking the internet?", but "should I?", given that those using 8.8.8.8 probably don't understand that it's - in most cases - probably giving them a worse experience than asking locally. You could argue that it's "fixing the Internet" for the naive user. But Google DNS is not "standard" DNS. How non-standard it is isn't clear, but ... well, it's Google ... so providing something that behaves "differently" to Google DNS at the Google DNS address does strike me as rude. But is it actually bad? Note that I'm not proposing to do this on our infrastructure, because we have a customer base that for the most part either has clue, or punts clue to us. But other providers might think differently... -- don On 18/09/13 19:53, Tim Price wrote:

Well I guess if you want to get all technical ;). Blame the flu for the missing of the obvious, I know I will.

Cheers,

Tim

On 18/09/2013, at 7:51 PM, Nathan Ward wrote:

...

On 18/09/2013, at 7:46 PM, Tim Price wrote:

...

Apart from the obvious net neutrality rhetoric why would you have a stateful device in your forwarding path. And you'd better hope that google don't decide to do authoritive hosting on those servers now or in the future (although we all know they shouldn't) Yeah, why /would/ you have a stateful device in your forwarding path, in order to achieve Don's suggestion? :-)

Loopback/dummy interface + static route.

-- Nathan Ward

On 2013-09-18, at 04:36, Nathan Ward wrote:

Maybe something better would be getting an IANA assigned address for providers to stick on their DNS servers, and everyone can anycast it.

I'm interested in why people think this would be a good idea. Running a caching resolver at an ISP is a twitchy business. There's nothing more likely to make the helpdesk phone ring (or for people to declare your network to be dead, and to take steps to move elsewhere) than a flaky resolver. A resolver service distributed using anycast across multiple/many ISPs seems likely to result in your customers using someone else's resolver infrastructure at least some of the time. Given the tight bond between resolver service quality and customer experience, this seems like an idea with curious business characteristics (unless everybody does it, but that doesn't seem like a particularly low-energy state for the world to be in). Resolvers can be mined extensively and the resulting data streams can be monetised. If you think that's evil, but you're happy for your customers to use someone else's infrastructure, then perhaps you don't think it's that evil. If you don't think it's that evil, sending your customer data to someone else's box so that you can't easily mine it yourself seems equally curious. Servers that I provision have a local resolver bound to localhost, because I don't like the dependency explosion that comes from doing it any other way. Customers get DNS server addresses handed to them in DHCP options or IPCP. I don't see much incentive from that angle to provision a well-known, cross-provider address. Google's success with 8.8.8.8 and 8.8.4.4 seem to mainly result from customers getting frustrated with flaky ISP resolvers. If you can afford the operational cost and can provide the dedicated team required to run a resolver service that is supremely reliable, why not use that opex on your own infrastructure rather than propping up your competitors' deficiencies? What am I missing? (Incidentally, if someone *was* to do this, running the idea through the IETF and an IANA Considerations section in order to get addresses assigned seems like the right way only if you are anxious to spend a few years arguing your case in front of a skeptical audience before you find out the answer is no. The more pragmatic approach is for someone with a spare /24 and/or /48 to make local arrangements over a beer for the address to be used to advertise an anycast service, and encourage people to turn up servers with the appropriate config.) Joe

On 19/09/2013 02:30, Joe Abley wrote:

Google's success with 8.8.8.8 and 8.8.4.4 seem to mainly result from customers getting frustrated with flaky ISP resolvers.

Bingo. The merger into TVH saw the DNS servers that had been put into our WiFi config "go away", resulting in all kinds of challenging behaviour. Rather than trying to hunt down the correct name servers, with unpleasant past memories of ISPs running resolving and authoritative on the same servers, I just put in 8.8.8.8 and 8.8.4.4. If resolution takes a bit longer, well, big whoop - it's not a critical service. I've actually been surprised at the general hostility in this thread to using pings to 8.8.8.8 as a way to track international connectivity and get a vague idea of latency. I don't happen to "just know" international sites which respond to ping requests, but I've used 8.8.8.8 many times for establishing if a connection can get external ping; it's ISP agnostic, which is great. If people have better destinations for ping monitoring, toss them out there. I can't be the only person in here who isn't running an ISP network but does want to know if their connection to the tubes is working in all directions. As an aside, the TVH DNS server I'm using for ping monitoring to the ISP went away (after first spiking pings to nearly 200ms) for a period last night. 8.8.8.8 works, or your internet doesn't. Mumble mumble flaky ISP resolvers? -- Matthew Poole "The difference between theory and practice is that practice is easier in theory than theory is in practice"

On 2013-09-18, at 16:40, Matthew Poole wrote:

I don't happen to "just know" international sites which respond to ping requests,

You're very welcome to ping L.ROOT-SERVERS.NET at any time, although ideally (but not really, yet) that's always local :-)

As an aside, the TVH DNS server I'm using for ping monitoring to the ISP went away (after first spiking pings to nearly 200ms) for a period last night. 8.8.8.8 works, or your internet doesn't. Mumble mumble flaky ISP resolvers?

If anybody here thinks that it'd be nice to be able to buy inexpensive, 1U, high-performance, supported, etc appliances which could hang in whatever parts of your network make sense to you and provide recursive service to customers with or without DNSSEC validation, drop me a note off-list and let me know what you think you want. I may have some ideas. Joe

On 2013-09-18, at 17:41, Ben wrote:

anyway, i think in general it's better to ping non any cast addresses, and a quick check suggested that there are at least nodes in scotland, los angeles, and auckland, but i don't know if you'd be keen to expose the non any cast ip addresses of these locations, and if that'd be ok to ping.

[krill:~]% dig nodes.l.root-servers.org txt +tcp | grep -i zealand nodes.l.root-servers.org. 21566 IN TXT "wlg41.l.root-servers.org" "Wellington" "" "New Zealand" "AsiaPacific" nodes.l.root-servers.org. 21566 IN TXT "wlg42.l.root-servers.org" "Wellington" "" "New Zealand" "AsiaPacific" nodes.l.root-servers.org. 21566 IN TXT "wlg43.l.root-servers.org" "Wellington" "" "New Zealand" "AsiaPacific" nodes.l.root-servers.org. 21566 IN TXT "wlg44.l.root-servers.org" "Wellington" "" "New Zealand" "AsiaPacific" nodes.l.root-servers.org. 21566 IN TXT "akl01.l.root-servers.org" "Mangere" "" "New Zealand" "AsiaPacific" nodes.l.root-servers.org. 21566 IN TXT "akl41.l.root-servers.org" "Mangere" "" "New Zealand" "AsiaPacific" nodes.l.root-servers.org. 21566 IN TXT "akl42.l.root-servers.org" "Mangere" "" "New Zealand" "AsiaPacific" nodes.l.root-servers.org. 21566 IN TXT "akl43.l.root-servers.org" "Mangere" "" "New Zealand" "AsiaPacific" nodes.l.root-servers.org. 21566 IN TXT "akl44.l.root-servers.org" "Mangere" "" "New Zealand" "AsiaPacific" nodes.l.root-servers.org. 21566 IN TXT "chc01.l.root-servers.org" "Christchurch" "" "New Zealand" "AsiaPacific" [krill:~]% [krill:~]% host chc01.l.root-servers.org chc01.l.root-servers.org has address 202.53.188.114 [krill:~]% [krill:~]% host akl41.l.root-servers.org akl41.l.root-servers.org has address 74.80.125.64 akl41.l.root-servers.org has IPv6 address 2001:500:15:1700::64 [krill:~]% See also http://tools.ietf.org/html/draft-jabley-dnsop-anycast-mapping-02 (and if you're interested in reviewing that document for clarity, please do let me know -- it's stuck in the individual submission queue right now with Nevil having difficulty finding people to look at it, and I'd like to get it published). Joe

Hi Ben, On 2013-09-18, at 17:28, Ben wrote:

Also sometimes root servers seem to disappear from locations, like f.root-servers.net used to have a location in NZ, but now seems to go to Brisbane...

I hear from my friends at ISC that work is afoot to repair the Auckland F-Root node, so hopefully it becomes more local for you in due course. L-Root ought to be local for you though (send me a traceroute if not, and we'll see what we can do).

The second is that it should be considered that this will perform no better than ISP name servers, but add diversity, and/or be more likely to work if on mobile network, remote wifi network, or if usual place of connection changed upstreams. This means if the ISP name server does mirrored caching between multiple POP's then this one could too, but the intention shouldn't be to extend beyond normal ISP DNS capabilities and create any kind of improved performance.

I think what a lot of people are discovering is that running a really reliable resolver service (especially if DNSSEC validation is involved) requires active operational management. We are past the point where you could start the shipping DNS resolver service on the OS of your choice and leave it running unattended for ten years without there ever being a problem; you need dedicated staff, active monitoring, disaster recovery plans, hardware support, an ear to the ground in the usual open and closed/secret forums, etc, etc which is easier to achieve if you're Comcast with eleventy-gazillion customers than if you're smallisp.net.nz. So this is not a question of technology or technical capability; it's a question of how much it makes sense to spend on DNS operations.

The third is that ISP's should still dish out their primary DNS servers over DHCP, and tell their users to use their own DNS servers. Support for such should be hidden, and not required for an ISP, but optimise the performance for end users that have had their DNS set to use this as primary/secondary/tertiary.

Sure, but perhaps an ISP could also hand out the addresses of a managed/supported DNS resolver service provisioned through appliances in the ISP's own network and find some comfortable middle ground.

Something not completely related or unrelated, is that when people are doing Anycast for DNS in general (both authoriative and recursive) it seems pretty common to send both primary and secondary DNS to the same region. I've seen this with both recursive and authorative, and it icnreases the chance of problems. The simple solution is to make secondary not be anycast and in an unrelated region, as I understand it can be complicated to shift traffic around right to use anycast in a secondary region while optimising all other paths.

The right thing here I think is to provision multiple clouds on the same set of anycast nodes where the overlap between clouds is managed, and provision each nameserver service (the name and associated addresses for each NS record) on a different cloud. That way you get useful diversity, but the individual services are still massively redundant. This requires lots of anycast nodes to be able to do well, of course. Another approach followed by people such as Afilias is to use more than one anycast service for the zones you care about. ORG is hosted on one cloud provisioned by PCH and another one run directly by Afilias, for example. I agree that having the path to two apparently-different nameservers for the same zone land on the same anycast node is probably a bad idea :-) Joe

On 2013-09-18, at 04:25, Don Stokes wrote:

But Google DNS is not "standard" DNS.

In what way? On 2013-09-18, at 03:38, Don Stokes wrote:

Random thought ... how bad would it be for a service provider to locally redirect 8.8.8.8/32 into their own DNS forwarders?

I think you'd find out when Google's lawyers called you. Joe

Hi Don, On 2013-09-18, at 16:19, Don Stokes wrote:

On 19/09/13 02:14, Joe Abley wrote:

...

...

But Google DNS is not "standard" DNS. In what way?

The short answer is, "I don't know." In Google's words, "Google Public DNS takes some new approaches that we believe offer more valid results, increased security, and, in most cases, better performance." I don't know what those "new approaches" are. Do you?

I don't, but I don't necessarily read into that text that their approach is non-standard. It's clearly standard enough to interop well with a large variety of client implementations that run entirely different code.

"Not 'standard'" was perhaps a little unfair, but it is fair to say that Google DNS is not implemented the way one would expect a "normal", stand-alone BIND or similar name server to be,

Depends what you mean by similar, I guess. YADIFA, knot, NSD, PowerDNS, Microsoft DNS, etc, etc all follow usefully different approaches, and are implemented to varying degrees differently from BIND9. I don't think that means any of them are necessarily non-standard for that reason, though. (They might well be non-standard in other ways, of course.) An enduring complaint about the DNS in general is that the specifications are smeared liberally over a boatload of different RFCs, seasoned with implementation decisions made by early implementors which are not written down. A consequence of this mess is that it's very difficult to design a compliance test suite with any teeth. (A related problem is that nobody seems to have much interest in designing a compliance test suite anyway.) I've definitely found non-standard behaviour with particular implementations from time to time, and I thought perhaps you had seen something interesting with 8.8.8.8. But I understand what you were getting at. Joe

On 18/09/2013 7:47 p.m., Nathan Ward wrote:

On 18/09/2013, at 7:38 PM, Don Stokes wrote:

...

On 18/09/13 18:58, Nathan Ward wrote:

...

As has been mentioned by others, don't use 8.8.8.8 for DNS if you're in NZ because Akamai will make your life hard. Random thought ... how bad would it be for a service provider to locally redirect 8.8.8.8/32 into their own DNS forwarders?

I mean it would probably violate the Principle of Least Astonishment, at least for folk who are paying attention, but random users who have been told that "it's better to use 8.8.8.8" without fully understanding the implications might get better performance…

This is why we can't have nice things.

Incidentally, on some networks, for HTTP it doesn't change too much, because their proxy servers do DNS lookups based on whatever you put in your requests' Host header, instead of whatever is the original destination IP address.

Here's an example of this sort of thing in the past: http://list.waikato.ac.nz/pipermail/nznog/2009-August/015712.html

Anything in the Host header is a bit of an overstatement for the current generation of HTTP proxies. Since 2009 there has been significant cross-vendor effort to eradicate that ability and validate the Host header against destination IP before using either one. AYJ -- "That Squid Guy"