Re: [nznog] NOC status (was 2Degrees (SNAP) issues at 220 Queen Street this morning)

4 Jul 2016

      <html>
<head>
<title>BigTelcoVeryImportantStatusPage</title>
</head>
<table>
 <tr>
  <td>Service</td>
  <td>Status</td>
 </tr>
 <tr>
    <td>Everything</td>
    <td><img src=“thisisfine.jpg”> </td>
 </tr>
</table>
</html>

Sadly most vendor status pages are worth less than the cost of the bandwidth to download them… this includes some of the very large players whom should know better. Ideally we should have data-driven status pages in 2016 that get fed from internal monitoring systems, but I’m sure that becomes too political about disclosing failure, especially when media read things incorrectly (“redundant link B down" -> “all internet in NZ on fire!”).

regards,
Jethro
...
On 5/07/2016, at 20:33, Tim Price  wrote:
I think that the idea was more just having a link off to each operator’s network status page from the NOC list rather than an aggregate centralized status page.  When I first read it I also misinterpreted the intent and it got me thinking along the lines of “why should I trust this status page more than the person at the NOC that just lied to me?” and “sounds like an unmaintainable politically charged clusterf*ck waiting to happen” which made me go back and read it again.
From:  on behalf of Jamie Baddeley 
Date: Tuesday, 5 July 2016 at 7:54 PM
To: Ian Lewis 
Cc: NZNOG Mailing-List 
Subject: [nznog] NOC status (was 2Degrees (SNAP) issues at 220 Queen Street this morning)
I think it's a great idea.
Would be a good paper/lightning talk from someone at the next NZNOG conference.
The key issue is a standard API that network operators could present to an aggregator location like http://www.nznog.org/noc-list
and for it to include info that operators need to know and care about.
If some sort of API spec was published then at least folks would know what to aim for. Jeez, even a set of twitter handles to aggregate would be a start. But then we'd rely on Timmeh not breaking shit.
Anyone up for talking about that sort of stuff in February?
Jamie
On 5 July 2016 at 14:07, Ian Lewis  wrote:
...
Just a thought… What do you folks think about adding “Network Status” page links to the existing NOC list? http://www.nznog.org/noc-list
Could provide a quick and easy zero-touch way of getting the word out about network faults without tying up hard-working NOC people with serial phone calls / emails?
Disclaimer: my company doesn’t actually *have* a Network Status page yet (working on it), but many on the NOC list do…
Thoughts?
PS Thanks for the idea Jesse Archer. J
Ian Lewis 
T  +64 7  850 3841 | M  +64 27  405 7765 | E  ian.lewis(a)ultrafast.co.nz
From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Liam Farr
Sent: Tuesday, 5 July 2016 2:01 p.m.
To: Tim Hoffman
Cc: NZNOG Mailing-List
Subject: Re: [nznog] 2Degrees (SNAP) issues at 220 Queen Street this morning
Indeed shutting down all (flapping) circuits from the affected provider at that location allowed traffic to traverse alternate paths and carriers and normal operation to resume.
Unfortunately not all downstream customers choose to buy 100% diverse services due to cost constraints, and as such are vulnerable to network interruptions affecting a particular POP or hand over point, this is an accepted risk they take, but it doesn't make it any less impacting when it breaks. (I believe you actually built some of the affected non-diverse circuits back in the day bro).
Sometimes when your shit is on fire, and you don't have enough information about whats happening, in a communications vacuum it can help to reach out to the community as someone else may know more than you, and sometimes when you do that you don't always write the most comprehensive or eloquent emails as your a little pre-occupied with the issues at hand.
Thankfully the issues are mostly resolved or routed around, although a RCA is going to take some time.
On 5 July 2016 at 13:39, Tim Hoffman  wrote:
...
Sounds like your network is lacking in carrier diversity if they can cause you that much bother bro
Sent from my iPhone
On Jul 4, 2016, at 6:32 PM, Liam Farr  wrote:
...
Your right Tim, its not the NOC.
At the time the NOC was being uncommunicative, and as a customer I was left in the dark about a major P1 fault affecting me.
Due to communication difficulties with the NOC and in exasperation as a customer I decided to post here to see if anyone else was affected, or had any further information they could provide on the issue, (i.e. is anyone else affected, I'm trying to find out what the fuck is actually going on), had the NOC been more communicative at the time I would not have resorted to posting here.
Subsequently I escalated this to my account manager and then the CTO, and the flow of communication from NOC to customer has resumed in a satisfactory manner.
Thanks for your concern.
On 5 July 2016 at 13:22, Tim Hoffman  wrote:
...
I wasn't aware that this mailing list was the 2degrees NOC email.
On Mon, Jul 4, 2016 at 5:52 PM, Jesse Archer  wrote:
...
We also noticed some issues.
2Degrees NOC are aware.
Jesse Archer
General Manager
Full Flavour
p. 07 577 0099  ddi. 07 281 1391
s. Skype "myfullflavour"
e. jesse(a)fullflavour.nz
w. fullflavour.nz
a. Basestation, 148 Durham Street, Tauranga
a. PO Box 13403, Tauranga Central, Tauranga 3141, New Zealand
On Tue, Jul 5, 2016 at 11:02 AM, Liam Farr  wrote:
> Hi,
>  
> Just wondering if anyone else had issues with services delivered by 2Degrees (SNAP) at 220 Queen Street this morning from 9AM onwards?
>  
> 
>  
> -- 
> Kind Regards
>  
>  
> Liam Farr
>  
> Maxum Data
> +64-9-950-5302
>  
> _______________________________________________
> NZNOG mailing list
> NZNOG(a)list.waikato.ac.nz
> https://list.waikato.ac.nz/mailman/listinfo/nznog
>
_______________________________________________
NZNOG mailing list
NZNOG(a)list.waikato.ac.nz
https://list.waikato.ac.nz/mailman/listinfo/nznog
-- 
Kind Regards
Liam Farr
Maxum Data
+64-9-950-5302
-- 
Kind Regards
Liam Farr
Maxum Data
+64-9-950-5302
Attention: This e-mail message is intended for the intended recipient only and it may contain Ultrafast Fibre Limited confidential or legally privileged information or both. No confidentiality or privilege is waived or lost if this email has been delivered to the wrong recipient. If you have received this email in error, please immediately delete it from your system and notify Ultrafast Fibre Limited. You must not disclose, copy or relay any part of this correspondence if you are not the intended recipient. Any views expressed in this message are those of the individual sender and not Ultrafast Fibre Limited. This email has been checked for viruses. However, Ultrafast Fibre Limited makes no warranty that this email or any attachments are free from viruses or other conditions which may damage or interfere with recipient data, hardware or software. The recipient relies on its own procedures and assumes all risk of use and of opening any attachments.
_______________________________________________
NZNOG mailing list
NZNOG(a)list.waikato.ac.nz
https://list.waikato.ac.nz/mailman/listinfo/nznog
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog 
_______________________________________________
NZNOG mailing list
NZNOG(a)list.waikato.ac.nz
https://list.waikato.ac.nz/mailman/listinfo/nznog

Re: [nznog] NOC status (was 2Degrees (SNAP) issues at 220 Queen Street this morning)

Jethro Carr