- NZNOG - lists.nznog.org

Where can congestion occur on a UFB (GPON) network?
by Joel van Velden 03 Jun '15

03 Jun '15

Hi all, At Auckland Thursday Night Curry I asked what I thought was a simple enough question: "In this this new world of GPON fibre that is UFB, /where can congestion occur/?" Obviously at handovers, no question, but where else? From what I can figure, given a hypothetical GPON network using a 24-way optical split (seems to be number quoted in a NZ context) Standard GPON suggests 2.488 Gbps down, so split 24 ways is 103.7 Mbps (Not sure what packet sizes) So if all customers behind a splitter are provisioned at a maximum of 100Mbps down, there could be /no/ congestion? Obviously this is not the case with 200 and 1000 down being sold. Are there other possible points of congestion between the point of handover and the GPON switch? (forgive me I'm not up with the exact terminology. OLT?) What have I got wrong? -- Regards, Joel van Velden

1 0

RE Legal Text on TICSA - Visit to Wellington/ Seeking input
by Beau Murrah 27 May '15

27 May '15

Hello NZNOG list, Some months ago I emailed to the list about my plan to produce a legal manual/critical to guide to the TICSA legislation for network operators/the list. I got some feedback from people on this list and outside of it that there was interest in such analysis and some highlighted a number of particular issues to look into/affecting them. My original hope was to have it completely edited and ready to share to the list about this time - I know updated guidance was/is planned to be released by the NCSC soon (or has already been released). Well, Im still going at it but it is proving difficult topic to write about and finish into a usable and credible form. My text is currently 6500+ words - and I think there are definitely some genuinely some useful or novel things for consideration but it is also all over the place. Initial feedback on the very rough draft was that it doesn’t seem clear exactly exactly who I am writing for - seeming like a mix of legal academic essay and direct practical consideration for network operators and also partly political content about the nature of bureaucracy in this area (the latter Ive decided to cut out as much as possible. I believe writing about this topic is difficult because I realised I don’t know much about what the worklife of a network operator of a truly large network is actually like at all. I have read/seen most all of the publicly available information - close reading of the legislation and related legislation, NCSC guidance, information from this list and so on. None of the technical people I know are network operators. It is hard to gauge what the level/type of interaction with the Agency is actually like broadly. I really don’t want to inadvertently publish any bullcrap technical claims and there is plenty of cyber-paranoid stuff out there already. So, I digress- I am visiting Wellington later this month (28th-31st May) and I would love to get insight from a network operator(s) over beer/coffee/water. I am actually less interested in what TICSA compliance has been like for you - and I don’t expect or want you to “leak” anything or disclose anything - but to hear more about the worklife of a network operator in general day to day that would come under examination in any potential legal proceeding under TICSA. To give a quick preview of my text, it focuses on the fundamental importance of good faith in the legislation to protecting you from any worst case scenario from alleged non-compliance. It considers what remaining in good faith could look like in a situation of alleged non-compliance and what variety of reasons or events could lead to non-compliance while good faith has been retained by the network operator. This depends on a lot on getting an understanding your workflow and obligations and what kind of people are you expected to report to and way that evidence accumulates in your day - much like in any employment situation or potential examination/dispute. There is very limited legal precedent to what Good Faith actually is from legislation and case law around the issue of employment and law and collective bargaining. How good faith might be interpreted involving network operators dealings with the Agency is a very open question and something all network operators ultimately have an interest in. I will say that the guidance provided to you by the NCSC guidance is not legal binding guidance and makes particular assumptions about the nature of good faith. It is very possible that this entire law will never effectively see a day in court if the narrative that this regulation is intrinsically a piece of national security and good faith flatly means following the Agency’s commands or striving to meet their interpretation of compliance. I do not think this is correct nor is anyway certain. I will add though, an actual Court case under this legislation’s confidentiality and evidential provisions will attempt to turn a civil court into a “Secret Court” like FISA this Country has never really seen and will intimidate the most experienced of lawyers, Judges and Court staff - there is a huge impetus to avoid this situation, and hopefully this can be put more in the network operators favour, rather than to scare them. I invite you to email me off-list if you want to meet, or to email on-list to propose a venue/time on-list for people to chat as a group. Happy Budget day, ___________ Beau Murrah Ph: +64 27 375 7897 Email: bmurrah(a)icloud.com <mailto:bmurrah(a)icloud.com> - PGP: https://keybase.io/airbridge <https://keybase.io/airbridge> Linkedin: https://www.linkedin.com/in/bmorganmurrah <https://www.linkedin.com/in/bmorganmurrah> Enrolled Barrister and Solicitor of the High Court of New Zealand (NB: not a lawyer <https://www.lawsociety.org.nz/for-lawyers/joining-the-legal-profession/admi…>)

2 1

Re: [nznog] Need assistance troubleshooting a DNS issue
by Glen Eustace 27 May '15

27 May '15

To satisfy the curious and to provide some closure on this incident, I have included a statement from Ryan Joe. The incident began approx last Oct/Nov and was resolved last week. I would like to thank Peter Lambrechtsen for getting the right people at Spark to contact me and for their perseverance working through what turned out to be more than one non-trivial issue. The impact, unfortunately, was not wide spread and hence had gone largely un-noticed but had the potential to impact a lot more people. I don’t believe that it is my place to go into the details of the issues, what was found and how they were fixed. Perhaps a Spark engineer would be prepared to do so as I believe it would be of interest to many on this list. "As discussed on Friday, this email is to acknowledge that Spark experienced a problem with our DNS which resulted in some Spark customers being intermittently unable to access content hosted at godzone.net.nz <http://godzone.net.nz/>. I’d like to sincerely apologise for any frustration or inconvenience caused by this – we have now fully resolved the issue” Ryan Joe Senior Product Manager - xDSL -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Glen and Rosanne Eustace, GodZone Internet Services, a division of AGRE Enterprises Ltd., P.O. Box 8020, Palmerston North, New Zealand 4446 Ph: +64 6 357 8168 <tel:%2B64%206%20357%208168>, Fax: +64 6 357 8165 <tel:%2B64%206%20357%208165>, Mob: +64 27 542 4015 <tel:%2B64%2027%20542%204015> "Specialising in providing low-cost professional Internet Services since 1997"

1 0

Random number generators
by Nathan Ward 25 May '15

25 May '15

Hi all, As we all know, random numbers are an important part of cryptography, which is required for tools that are important to network operators like the widely deployed RPKI. Most systems today generate random numbers using a PRNG, rather than a dedicated hardware random number generator. A PRNG is only as good as its entropy source - if you have a small amount of entropy, an attacker can start to predict the output of your system’s PRNG, which is obviously bad. There are a number of existing tools for generating entropy: - HAVEGE implementations (i.e. http://www.issihosts.com/haveged/) - Audio/video sources (audio_entropyd and video_entropyd) - Some other things like mouse/keyboard input (i.e. when you wiggle your mouse over that window in whatever that app was that generated keys, puttygen maybe?) - Network interrupts is also a common source A week or two back I was a little hungover and didn’t want to do any real work, and decided to write my own entropy harvesting tool, and have the code available at the below URL: https://github.com/nward/trueentropy I strongly encourage you NOT to run this on production systems, until it has been certified (a good standard to shoot for would be NIST SP 800-90B perhaps), but I would like to get feedback, so please read the code and suggest improvements. Perhaps you have some additional sources of entropy data that would be useful. I would particularly like a good way to “boil down” the entropy generated. The Linux kernel by default can take up to 4096 bits, and this provides far, far more than that, which just seems like a bit of overkill. Please feel free to submit pull requests with ideas for that - it would be a great project to learn a bit of Python. Generally, entropy should not be sourced from a system you don’t control, or sourced over a network you don’t control, for obviously reasons. However, the quality of the data is so good in this case that I believe it negates those concerns. The data is also fetched with HTTPS, so as long as the source doesn't also starting using this as their entropy source (known as entroception) we should be OK - see my recommendation about production systems. -- Nathan Ward (opinions all my own, blah blah blah)

8 14

Dark fibre in the outer wilderness of... Parnell
by Matthew Poole 24 May '15

24 May '15

Hi all I'm getting discouraging information on the availability of dark fibre around the bottom end of St George's Bay Rd in Parnell, near Xero and the Textile Centre. Seems that nobody has any, or at least nobody is mentioning any. If anyone knows of dark fibre capacity in that area could they please contact me off-list. Cheers -- Matthew Poole "The difference between theory and practice is that practice is easier in theory than theory is in practice"

3 3

Re: [nznog] Random number generators
by Nathan Ward 23 May '15

23 May '15

> On 24/05/2015, at 15:12, Dean Pemberton <dean(a)deanpemberton.com> wrote: > > > I can't imagine that if you were an ISP you wouldn't know exactly where any TrueNet probes were in your network. That is possibly true if you’ve paid TrueNet for probes. If your customers have opted to have one without your involvement (which I assume is possible, given ComCom give them money to do just that across NZ) then that’s unlikely. > The only question then is which of the following strategies you employ. > > 1) do nothing. > 2) make this user the best on your network > 3) randomise the results > 4) tweek the results to spell out "what is six times nine” As I’m sure you can imagine, I’m keen to work on solutions to implement (4) if any network providers who have TrueNet portal access are keen for some fun. > Nathan, have you had a look at any other systems (SamKnows) to see if they are any closer to measuring Internet Good-ness? Briefly, but not in a while. My understanding is that SamKnows is much the same as TrueNet - a set of contrived tests, though they do a much better simulation of a web page load than TrueNet and it’s use of “URL Downloads using the WGET command” - but I don’t believe it’s anywhere near perfect in today’s javascript heavy world, where clients cache static content etc. etc. -- Nathan Ward

1 0

Anyone else having GGIS speed problems?
by Nathan Ward 19 May '15

19 May '15

Hi all, Beware, operational content. A couple of hours ago I started noticing weird things getting to stuff overseas. Looks like GGIS have some sort of path flapping that doesn’t care about IP flows. I see RTT to sites in the US jump from ~125ms to ~160ms several times per second, which to me is indicative of going the short and long way around the SXC network, respectively. # hping3 -S -p 80 216.146.46.10 HPING 216.146.46.10 (eth0 216.146.46.10): S set, 40 headers + 0 data bytes len=46 ip=216.146.46.10 ttl=55 DF id=33739 sport=80 flags=SA seq=0 win=65535 rtt=129.1 ms len=46 ip=216.146.46.10 ttl=55 DF id=47972 sport=80 flags=SA seq=1 win=65535 rtt=132.3 ms len=46 ip=216.146.46.10 ttl=53 DF id=48561 sport=80 flags=SA seq=2 win=65535 rtt=159.6 ms len=46 ip=216.146.46.10 ttl=55 DF id=35481 sport=80 flags=SA seq=3 win=65535 rtt=128.1 ms len=46 ip=216.146.46.10 ttl=54 DF id=49791 sport=80 flags=SA seq=4 win=65535 rtt=158.7 ms len=46 ip=216.146.46.10 ttl=55 DF id=36729 sport=80 flags=SA seq=5 win=65535 rtt=132.2 ms len=46 ip=216.146.46.10 ttl=53 DF id=50960 sport=80 flags=SA seq=6 win=65535 rtt=158.8 ms len=46 ip=216.146.46.10 ttl=55 DF id=37839 sport=80 flags=SA seq=7 win=65535 rtt=129.1 ms len=46 ip=216.146.46.10 ttl=55 DF id=38397 sport=80 flags=SA seq=8 win=65535 rtt=129.2 ms len=46 ip=216.146.46.10 ttl=54 DF id=39075 sport=80 flags=SA seq=9 win=65535 rtt=158.8 ms The above example is a bunch of SYN packets so are different L4 flows so it’d be *sort of* reasonable for them to go different ways. However, you see plenty of out of order TCP packets if you do a long running flow (like say trying to load a web page with large objects, such as speedtest.net) Out of order packets means TCP cries a bunch and goes slow. This is the effect I see. Note also the TTL differences and correlation to RTT. I do not see the TTL hopping around when looking at a single TCP stream, though I do see different TTLs on two concurrent streams, so I suspect that’s some other ECMP happening that isn’t a problem. Who knows where that might be, but I don’t believe it’s important. This particular host I have tested to is the speedtest.net main web server. I see the out of order packets when doing a regular speedtest.net test to the Sonic.net endpoint in California (I think San Jose?). Here is what mtr sees when tracing to the same host as above. It is unclear to me why hop 9 doesn’t see the same weird latency. It sees it sometimes, but not all the time. Perhaps there’s something about the way mtr sends packets that means the hashing is different over some sort of aggregate interface/ECMP thing. Maybe that’s where our TTL difference from above happens. I *always* see the frequent latency changes (jitter, I guess) on hop 8. 7.|-- ae1-10.tkbr12.global-gate 0.0% 10 3.3 3.5 2.2 5.3 0.7 8.|-- xe-0-0-7-3.sjbr3.global-g 0.0% 10 130.5 136.1 126.3 163.1 14.2 9.|-- ae0.pabr5.global-gateway. 0.0% 10 154.8 154.7 154.1 155.9 0.5 Here are some more samples that show the differences in later hops, but always hop 8 having jitter: 7.|-- 202.50.232.37 0.0% 10 3.5 3.2 1.5 4.1 0.7 8.|-- 122.56.127.22 0.0% 10 160.2 148.0 127.7 160.2 15.0 9.|-- 122.56.127.25 0.0% 10 158.2 148.6 132.2 166.9 14.3 10.|-- 203.96.120.74 10.0% 10 159.0 159.0 158.2 160.0 0.4 11.|-- 4.28.172.109 0.0% 10 167.6 171.3 166.3 207.7 12.8 7.|-- 202.50.232.37 0.0% 10 2.9 3.3 2.7 4.2 0.3 8.|-- 122.56.127.22 10.0% 10 159.3 147.1 126.6 168.7 17.1 9.|-- 122.56.127.25 10.0% 10 158.7 146.9 132.5 166.3 13.8 10.|-- 203.96.120.74 10.0% 10 159.7 158.8 158.0 159.7 0.0 11.|-- 4.28.172.109 20.0% 10 167.0 167.4 166.8 168.1 0.0 At least two providers have (or are in the process of) logging faults, so I figured it’s appropriate to bring it on to the list. Is anyone else seeing this behaviour? Do you have any feedback from GGIS yet? Have you had any customers complain about poor speeds/user experience? -- Nathan Ward

1 0

Situations Vacant: Network Operations Team Leader
by Mark Foster 07 May '15

07 May '15

'evening folks. A quick 'sits vacant' advertisement; my Network Operations Team Leader is heading across the ditch for a new role shortly and I'm beginning the recruiting process to replace him. Looking for someone who speaks BGP and MPLS reasonably fluently and can help manage our network, managed security and datacentre operations, and also has some leadership potential (with two direct-reports to manage, in addition to the technical skills sought). If you're interested or know someone who might be, please contact me off-list for further information (keeping this deliberately light on advertisey-type-speak). The role is located in the Auckland CBD. Thanks, Mark. -- Mark Foster Technical Services Manager - Australia & New Zealand [Description: Description: Description: Description: Description: iconz-webvisions-logo-png] www.iconz-webvisions.com<http://www.iconz-webvisions.com/> Follow us on LinkedIn<https://www.linkedin.com/company/iconz-webvisions-group?trk=company_logo>

1 0

Bitcoin/DDoS Ransoms
by Dean Pemberton 06 May '15

06 May '15

Hi all, Wearing my NZITF hat this time. We've put out a release this morning to get the message our about the latest round of "Pay us Bitcoin or your network gets DDoSed' You can see the release here: http://nzitf.org.nz/news.html But why I really wanted to address this audience was that one of our suggestions to organisations is... · Establish points of contact with your Internet Service Providers (ISP) in the event that you need them to perform traffic filtering. Defence against many attack types is most effective when performed before it reaches your network. To date NZITF has had reports of organisations being able to handle these attacks effectively through collaboration with their ISPs. And I wanted to make sure that all the Network Operators on the list had a heads up that A) Organisations might call you asking for help and B) you should work out what you'd do to help them. In fact free to take the release, tweek it and send it out to your customers if that helps. Regards, Dean

3 3

Cisco Fellow Fred Baker - OpenStack on IPv6: Authorization vs Routing
by Nathan Ward 05 May '15

05 May '15

Hi, Apologies to those who are on InternetNZ members-discuss who’re seeing this for a second time. This looks like it would be fairly interesting for some people on the list. I’d be going myself if I’d known about it sooner! Thanks to Hamish MacEwan for posting it originally. -- Fri, May 8, 10:00 – 12:00 Cotton Club, Cotton 350 ECS Seminars Description Fred Baker will discuss concepts and a project in an architecture for OpenStack/IPv6, with a view to simplification and scaling in large datacenters, multi datacenter operations, multi-administration projects, and hybrid datacenters. Bio: Cisco Fellow Fred Baker has been involved in data communications since 1978 and the development of the Internet since the 1980's. He has been involved in standards development in the IETF, both contributing technically and managing processes. His current interests are in routing, the management of latency and data flow, the universal deployment of IPv6 (especially in OpenStack data centers), and the public policy implications these issues bring with them. http://ecs.victoria.ac.nz/Events/Seminars?rm=details&id=973 -- Nathan Ward

1 0