- NZNOG - lists.nznog.org

Re: [nznog] Random number generators
by Nathan Ward 23 May '15

23 May '15

> On 24/05/2015, at 15:12, Dean Pemberton <dean(a)deanpemberton.com> wrote: > > > I can't imagine that if you were an ISP you wouldn't know exactly where any TrueNet probes were in your network. That is possibly true if you’ve paid TrueNet for probes. If your customers have opted to have one without your involvement (which I assume is possible, given ComCom give them money to do just that across NZ) then that’s unlikely. > The only question then is which of the following strategies you employ. > > 1) do nothing. > 2) make this user the best on your network > 3) randomise the results > 4) tweek the results to spell out "what is six times nine” As I’m sure you can imagine, I’m keen to work on solutions to implement (4) if any network providers who have TrueNet portal access are keen for some fun. > Nathan, have you had a look at any other systems (SamKnows) to see if they are any closer to measuring Internet Good-ness? Briefly, but not in a while. My understanding is that SamKnows is much the same as TrueNet - a set of contrived tests, though they do a much better simulation of a web page load than TrueNet and it’s use of “URL Downloads using the WGET command” - but I don’t believe it’s anywhere near perfect in today’s javascript heavy world, where clients cache static content etc. etc. -- Nathan Ward

1 0

Anyone else having GGIS speed problems?
by Nathan Ward 19 May '15

19 May '15

Hi all, Beware, operational content. A couple of hours ago I started noticing weird things getting to stuff overseas. Looks like GGIS have some sort of path flapping that doesn’t care about IP flows. I see RTT to sites in the US jump from ~125ms to ~160ms several times per second, which to me is indicative of going the short and long way around the SXC network, respectively. # hping3 -S -p 80 216.146.46.10 HPING 216.146.46.10 (eth0 216.146.46.10): S set, 40 headers + 0 data bytes len=46 ip=216.146.46.10 ttl=55 DF id=33739 sport=80 flags=SA seq=0 win=65535 rtt=129.1 ms len=46 ip=216.146.46.10 ttl=55 DF id=47972 sport=80 flags=SA seq=1 win=65535 rtt=132.3 ms len=46 ip=216.146.46.10 ttl=53 DF id=48561 sport=80 flags=SA seq=2 win=65535 rtt=159.6 ms len=46 ip=216.146.46.10 ttl=55 DF id=35481 sport=80 flags=SA seq=3 win=65535 rtt=128.1 ms len=46 ip=216.146.46.10 ttl=54 DF id=49791 sport=80 flags=SA seq=4 win=65535 rtt=158.7 ms len=46 ip=216.146.46.10 ttl=55 DF id=36729 sport=80 flags=SA seq=5 win=65535 rtt=132.2 ms len=46 ip=216.146.46.10 ttl=53 DF id=50960 sport=80 flags=SA seq=6 win=65535 rtt=158.8 ms len=46 ip=216.146.46.10 ttl=55 DF id=37839 sport=80 flags=SA seq=7 win=65535 rtt=129.1 ms len=46 ip=216.146.46.10 ttl=55 DF id=38397 sport=80 flags=SA seq=8 win=65535 rtt=129.2 ms len=46 ip=216.146.46.10 ttl=54 DF id=39075 sport=80 flags=SA seq=9 win=65535 rtt=158.8 ms The above example is a bunch of SYN packets so are different L4 flows so it’d be *sort of* reasonable for them to go different ways. However, you see plenty of out of order TCP packets if you do a long running flow (like say trying to load a web page with large objects, such as speedtest.net) Out of order packets means TCP cries a bunch and goes slow. This is the effect I see. Note also the TTL differences and correlation to RTT. I do not see the TTL hopping around when looking at a single TCP stream, though I do see different TTLs on two concurrent streams, so I suspect that’s some other ECMP happening that isn’t a problem. Who knows where that might be, but I don’t believe it’s important. This particular host I have tested to is the speedtest.net main web server. I see the out of order packets when doing a regular speedtest.net test to the Sonic.net endpoint in California (I think San Jose?). Here is what mtr sees when tracing to the same host as above. It is unclear to me why hop 9 doesn’t see the same weird latency. It sees it sometimes, but not all the time. Perhaps there’s something about the way mtr sends packets that means the hashing is different over some sort of aggregate interface/ECMP thing. Maybe that’s where our TTL difference from above happens. I *always* see the frequent latency changes (jitter, I guess) on hop 8. 7.|-- ae1-10.tkbr12.global-gate 0.0% 10 3.3 3.5 2.2 5.3 0.7 8.|-- xe-0-0-7-3.sjbr3.global-g 0.0% 10 130.5 136.1 126.3 163.1 14.2 9.|-- ae0.pabr5.global-gateway. 0.0% 10 154.8 154.7 154.1 155.9 0.5 Here are some more samples that show the differences in later hops, but always hop 8 having jitter: 7.|-- 202.50.232.37 0.0% 10 3.5 3.2 1.5 4.1 0.7 8.|-- 122.56.127.22 0.0% 10 160.2 148.0 127.7 160.2 15.0 9.|-- 122.56.127.25 0.0% 10 158.2 148.6 132.2 166.9 14.3 10.|-- 203.96.120.74 10.0% 10 159.0 159.0 158.2 160.0 0.4 11.|-- 4.28.172.109 0.0% 10 167.6 171.3 166.3 207.7 12.8 7.|-- 202.50.232.37 0.0% 10 2.9 3.3 2.7 4.2 0.3 8.|-- 122.56.127.22 10.0% 10 159.3 147.1 126.6 168.7 17.1 9.|-- 122.56.127.25 10.0% 10 158.7 146.9 132.5 166.3 13.8 10.|-- 203.96.120.74 10.0% 10 159.7 158.8 158.0 159.7 0.0 11.|-- 4.28.172.109 20.0% 10 167.0 167.4 166.8 168.1 0.0 At least two providers have (or are in the process of) logging faults, so I figured it’s appropriate to bring it on to the list. Is anyone else seeing this behaviour? Do you have any feedback from GGIS yet? Have you had any customers complain about poor speeds/user experience? -- Nathan Ward

1 0

Situations Vacant: Network Operations Team Leader
by Mark Foster 07 May '15

07 May '15

'evening folks. A quick 'sits vacant' advertisement; my Network Operations Team Leader is heading across the ditch for a new role shortly and I'm beginning the recruiting process to replace him. Looking for someone who speaks BGP and MPLS reasonably fluently and can help manage our network, managed security and datacentre operations, and also has some leadership potential (with two direct-reports to manage, in addition to the technical skills sought). If you're interested or know someone who might be, please contact me off-list for further information (keeping this deliberately light on advertisey-type-speak). The role is located in the Auckland CBD. Thanks, Mark. -- Mark Foster Technical Services Manager - Australia & New Zealand [Description: Description: Description: Description: Description: iconz-webvisions-logo-png] www.iconz-webvisions.com<http://www.iconz-webvisions.com/> Follow us on LinkedIn<https://www.linkedin.com/company/iconz-webvisions-group?trk=company_logo>

1 0

Bitcoin/DDoS Ransoms
by Dean Pemberton 06 May '15

06 May '15

Hi all, Wearing my NZITF hat this time. We've put out a release this morning to get the message our about the latest round of "Pay us Bitcoin or your network gets DDoSed' You can see the release here: http://nzitf.org.nz/news.html But why I really wanted to address this audience was that one of our suggestions to organisations is... · Establish points of contact with your Internet Service Providers (ISP) in the event that you need them to perform traffic filtering. Defence against many attack types is most effective when performed before it reaches your network. To date NZITF has had reports of organisations being able to handle these attacks effectively through collaboration with their ISPs. And I wanted to make sure that all the Network Operators on the list had a heads up that A) Organisations might call you asking for help and B) you should work out what you'd do to help them. In fact free to take the release, tweek it and send it out to your customers if that helps. Regards, Dean

3 3

Cisco Fellow Fred Baker - OpenStack on IPv6: Authorization vs Routing
by Nathan Ward 05 May '15

05 May '15

Hi, Apologies to those who are on InternetNZ members-discuss who’re seeing this for a second time. This looks like it would be fairly interesting for some people on the list. I’d be going myself if I’d known about it sooner! Thanks to Hamish MacEwan for posting it originally. -- Fri, May 8, 10:00 – 12:00 Cotton Club, Cotton 350 ECS Seminars Description Fred Baker will discuss concepts and a project in an architecture for OpenStack/IPv6, with a view to simplification and scaling in large datacenters, multi datacenter operations, multi-administration projects, and hybrid datacenters. Bio: Cisco Fellow Fred Baker has been involved in data communications since 1978 and the development of the Internet since the 1980's. He has been involved in standards development in the IETF, both contributing technically and managing processes. His current interests are in routing, the management of latency and data flow, the universal deployment of IPv6 (especially in OpenStack data centers), and the public policy implications these issues bring with them. http://ecs.victoria.ac.nz/Events/Seminars?rm=details&id=973 -- Nathan Ward

1 0

APNIC 40 Conference - Call for Papers
by Elly Tawhai 29 Apr '15

29 Apr '15

________________________________________________________________________ APNIC 40 Conference - Call for Papers ________________________________________________________________________ Hi NZNOG Community, The APNIC 40 Program Committee is now seeking presentations for the APNIC 40 Conference in Jakarta, Indonesia. We are looking for content that would suit the APOPS Plenary, technical conference sessions, lightning talks and Birds of Feather (BoF) sessions. Key Dates --------- The timeline for submissions is: Call for Papers Opens Now First Round Paper Acceptance 17 July 2015 Final Deadline for Submissions 14 August 2015 Final Round Paper Acceptance 21 August 2015 Final Slides Received 4 September 2015 Submitting Proposals -------------------- http://submission.apnic.net/user/login.php?event=28 Program Material ---------------- The technical sessions at APNIC 40 will include presentations relevant to Internet operations and technologies. The following topics are examples of possible areas of interest. - ISP and Carrier Services - IXPs and Peering - Network and infrastructure security - Access and Transport Technologies - Content and Service Delivery - Software Defined Networking and Network Function Virtualization If you have any other ideas or proposals for panel or BoF sessions, please feel free to submit your ideas via the submission system. If you have any questions, please email the Programme Committee: pc-chairs(a)apricot.net For more information about APNIC 40, please visit: https://conference.apnic.net/40 On behalf of: ________________________________________________________________________ Dean Pemberton & Mike Jager APNIC 40 Program Committee Co-chairs ________________________________________________________________________

1 0

IANA AS Numbers registry update
by Selina Harrington 29 Apr '15

29 Apr '15

Hi, The IANA AS Numbers registry has been updated to reflect the allocation of the following block to ARIN in April 2015: 64198-64296 Assigned by ARIN 2015-04-29 394240-395164 Assigned by ARIN 2015-04-29 You can find the IANA AS Numbers registry at: http://www.iana.org/assignments/as-numbers/as-numbers.xml The allocation was made in accordance with the Policy for Allocation of ASN Blocks to Regional Internet Registries: https://www.icann.org/resources/pages/global-policy-asn-blocks-2010-09-21-en Regards, Selina Harrington ******************************************* Internet Assigned Numbers Authority (IANA) Internet Corporation for Assigned Names & Numbers 12025 Waterfront Drive, Suite 300 Los Angeles, CA 90094 Phone: +1 310 301 5800 Fax: +1-310-823-8649 *******************************************

1 0

Re: [nznog] Looking for data on how much Netflix/SVOD data NZ ISPs are seeing
by Richard Naylor 28 Apr '15

28 Apr '15

At 09:23 a.m. 29/04/2015, Barry Murphy wrote: >Obviously the cache fill that happens between >midnight and mid-day consumes some bandwidth, we >find the fill capacity (after the initial fill) >is about 1.2gbps for on average 2-4 hours per day. >We currently share our cache on Megaport IX >Auckland, in doing so we find weâre filling a >few other providers caches so they donât have >to fetch it from AU, so this may be helping some providers. Thanks Barry - Thats REALLY useful. I'm involved in downstream activity in 2 areas. First - Uploading We have a current project moving some older videos from a server in NZ to Vimeo. Theres a weekly 20GB limit on uploads and an ugly web uploader (no restarts and a break means a complete re work). Using ADSL its pretty ugly. It takes 3.5 days to get 20-30GB up and ADSL 'breaks' make life a real pain. So I pay a local church to use their UFB. We find that uploading during the day conflicts with US based uploaders, so only get 10Mbps upload. If I wait until 5pm, we get a pretty good 100Mbps and the 20GB is up in about an hour. The first time we did it a 2.5G file that took 12 hours ( and failed) on ADSL went up in 4 minutes. Looking at the graph, it seems we have a nice window between 5pm and 8 or 9pm to do our uploads. Thats great to know. The second issue - Live feeds. We're doing quite a bit of live TV feeding using the net now. We have RX points with both TVNZ and TV3 - and also Sky Sport UK. The local TV companies are OK - its local (bounces out of SYD) and only SD and they don't really trust this IP stuff. Sky UK is different. We send them HD at 4Mbps and after some small adjustments for latency, can do live crosses with 3 sec delay (4 sec total allowing for return audio for the talent earpiece). NZ to UK is pretty solid. The other night we tried UK to NZ. NZ RX was via ADSL. We found we could only get a 1Mbps feed going. (vs 4Mbps going the other way) We need to do more testing as there are feed aggregation points and they are different depending on direction, but we have been wondering if there was congestion along the way causing the issue. If so we need to avoid doing crosses during the 'flix' peak. We are also going to test where we do the feed aggregation to see if that helps. Currently NZ to UK has aggregation in London, while UK to NZ has aggregation in LAX. We think its the LAX to NZ leg that suffers. More testing required. Richard

1 0

Looking for data on how much Netflix/SVOD data NZ ISPs are seeing
by Dean Pemberton 28 Apr '15

28 Apr '15

Hi Everyone. Following recent media articles making mention of "the Netflix effect" and broadband slowdowns at peak times, we've had a number of requests to see if we could pull together some facts from NZ ISPs detailing what they are actually seeing. So the question is... How much Netflix/SVOD traffic are you seeing, either as a direct measurement or percentage of total traffic. Does this percentage change at all during the day/week. I realise that this may be considered commercially sensitive but I'm sure we can find a way to make it work. If you're happy to give number on the list that would be great. Good on you for sharing. If you're not comfortable with that then drop me a line and we can discuss how you might feel more comfortable about sharing the facts. I'm happy to agree to only give an aggregated number from people who contact me privately for eg "ISPs are seeing between X and Y % SVOD traffic" Or you could just tell me and I'll keep it to myself (not quite as helpful but I can use it for context). You could also tell me "we have no way of knowing" and I guess that's a data point too :). Thanks -- Dean Pemberton Technical Policy Advisor InternetNZ +64 21 920 363 (mob) dean(a)internetnz.net.nz To promote the Internet's benefits and uses, and protect its potential.

2 1

Need assistance troubleshooting a DNS issue
by Glen Eustace 26 Apr '15

26 Apr '15

>From about October/November last year we have been getting the odd call from some of our customers to report that our servers are ‘not found’. So far, each report has been from an Xtra broadband user. When investigating these reports we have found that the servers were fine and DNS lookups from various sources (except Xtra) were as well. Last night the issue happened to a member of the family so I was able to jump onto their computer using Teamviewer and do some more thorough diagnostics. My results were; Name servers in use: ns1.xtra.co.nz <http://ns1.xtra.co.nz/> and ns2.xtra.co.nz <http://ns2.xtra.co.nz/> Name servers pingable from PC: Yes Does smtp.godzone.net.nz <http://smtp.godzone.net.nz/> resolve: No, times out Do other resources in our DNS resolve: No, time out Do resources in other DNS servers resolve: Yes Are all 4 of our name servers pingable from PC: Yes, 2 in NZ and 2 offshore Use nslookup to query our name servers directly, does smtp.godzone.net.nz <http://smtp.godzone.net.nz/> resolve: Yes Use nslookup to query our name servers using 8.8.8.8, does smtp.godzone.net.nz <http://smtp.godzone.net.nz/> resolve: Yes To fix the issue on the PC, I manually set the name servers to 8.8.8.8 and 8.8.4.4, all our services were then resolvable and useable. So, whats going on ? How are our servers different ? Well, we are one of the few ISPs that are using DNSSec to sign zones. Is DNSSec broken ? Not according to dnsviz.net <http://dnsviz.net/> The problem is also intermittent, I have heard that the two Xtra servers are actually LB VIPs in front of a farm of name servers. With the intermittent nature of the issue I wonder whether one server in the farm might be broken/misconfigured, just a thought. I have tried, without success to contact appropriate people at Xtra to either comment or assist and have failed to get past the level one helpdesk people. Their only response being “Sorry we can’t help you”. I am not saying this is an issue with Xtra’s internal recursive DNS servers but so far I have been unable to replicate the issue and have had no reports from any of our other customers using alternative broadband suppliers. I have run out of ideas now on how to continue to investigate this and just changing to Google’s DNS servers might work but isn’t a great solution. If anyone has any suggestions I’d appreciate hearing from you. Glen. -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Glen and Rosanne Eustace, GodZone Internet Services, a division of AGRE Enterprises Ltd., P.O. Box 8020, Palmerston North, New Zealand 4446 Ph: +64 6 357 8168 <tel:%2B64%206%20357%208168>, Fax: +64 6 357 8165 <tel:%2B64%206%20357%208165>, Mob: +64 27 542 4015 <tel:%2B64%2027%20542%204015> "Specialising in providing low-cost professional Internet Services since 1997"

8 12