Re: [nznog] UFB Upload Issues

Well done on being this approachable open about this issue! I look forward to your imminent NZNOG conference paper suggestion to talks(a)nznog.org =P Dean On Thu, Oct 9, 2014 at 9:43 AM, Brent Marquis wrote:

Thanks for pushing all that info back into the list.

I know of 1 RSP which has Juniper equipment that seems to be discarding CFI/DEI=1. I guess some switches need a bit of config to recognise CFI as DEI or to re-write it entirely (luckily there haven't been any hardware issues yet)

Given that impacted RSPs have solutions, or are working with vendors to find solutions - I don't think I'll progress DEI re-write on our side. After looking into it, the changes for that would either be a) painful, by manually changing each customer on a per-RSP basis or b) network wide, changing how the service operates for everyone. Both of these options would be unpalatable for either Chorus or some RSPs.

I do think we, at Chorus, need to make the DEI bit usage explicit in our documentation and, based on some questions in the last couple of days, explain exactly how low/high priority and the CIR on low priority operates - I'll take that on board and work on some documentation updates.

In the meantime, I'm comfortable taking non-operational questions anytime (probably off list is best) about our services. Anyone wellington based is welcome to catch up over a coffee (or, as it seems to be more commonly mentioned here, beer!) to discuss. I do travel to Auckland, although not often...

Thanks, Brent

Brent Marquis | Layer 2 Network Specialist Chorus | T : +6448964169 | M :+64272290923 Brent.marquis(a)chorus.co.nz

From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of NGW Network Operations Centre Sent: Wednesday, 8 October 2014 8:11 p.m. To: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues

I can confirm with working with Brent and the Chorus team that yes this is related to what Dylan is referring to.

In particular we tracked our issue down to some Cisco switching gear and found:

The Layer 2 packet header with CFI / DFI bit set to 0 indicates a standard Ethernet frame and passes to other Ethernet ports without issue. As soon as this is set to 1 then the packets get dropped when forwarding to an Ethernet port which is set as an access port. This is due to the switches thinking CFI bit 1 being set is a Token Ring frame which does not confirm to Ethernet MAC standards.

On ME 3600x and 3800x switches you can rewrite the DEI bit with some policy maps from 1 to 0 and it should happily pass this along. Link for those interested: http://www.cisco.com/c/en/us/td/docs/switches/metro/me3600x_3800x/software/r...

The above is also likely to be subject to IOS running so you may need to do some further checking or late night upgrades.

As far as Juniper gear goes our transport network passed the traffic without issue through trunks and access ports to the Cisco's. Running 12.3 firmware.

With the MX's it looks like at least 12.3 or higher supports also rewriting the DEI bit on ingress. Link for those interested: http://www.juniper.net/techpubs/en_US/junos12.3/topics/usage-guidelines/cos-...

I also believe version 14 (which is not JTAC recommended) supports DEI bit rewrite on any of the EX equipment also.

Anyway hopefully the above may help anyone out there experiencing issue with this.

On 8/10/2014 5:35 p.m., Dylan Hall wrote: While tinkering with CityLink we hit the same issue. Anything above 2.5Mbps was marked discard eligible and was being discarded by CPE Cisco switches (ME3400).

We figured out this is because in the old days the discard eligible bit used to mean "this packet has originated on a token ring network, the byte order of the mac addresses is different". The Cisco was happy to forward the frames across trunks, but would drop the packets on access ports on the assumption the workstation connected wouldn't know how to understand the mac addresses. This was on IOS 12.2.50ish.

I don't recall the details, but a change was made by Chorus that resolved the issue. Pete - do you recall the details?

Thanks,

Dylan

On 8 October 2014 16:46, Brent Marquis wrote: Result! Having been inundated with replies yesterday (which was great!) we've narrowed this issue down to a probable cause.

The new Chorus 'Accelerate' plans use Two Rate Three Colour marking on the low priority part of the service, which includes a low priority CIR of 2.5mbps. The EIR above this, up to the low priority plan speed (i.e 100mbps, 200mbps, 1gig, etc), is marked as discard eligible with the VLAN header DEI(CFI) bit.

Collaborative debug/information sharing with Chorus, Fastcom, Callplus, Inspire and DTS has shown that Chorus is, in fact, forwarding all traffic from our handover as expected, but traffic marked with DEI=1 is lost/discarded in RSP networks. The affected RSPs are discussing this with equipment vendors to find solutions or workarounds. I'm not keen to name names, as it's the RSPs equipment, but if anyone has experience with DEI marking causing issues on various switches, I'm sure help would be appreciated.

In parallel, we are looking at disabling DEI marking, but being that this was an integral part of the new plans, I need to engage a few people around chorus to find a way forward here. Considering some RSPs are consuming DEI=1 without issue, we don't want to break things that they may be doing. Not to mention this would be a change to a lot of our QoS policies, so need some sanity testing and deployment to production, etc.

Thanks to the RSPs involved, although I suspect we all have some work to do to get clear of this issue completely.

Thanks, Brent

Brent Marquis | Layer 2 Network Specialist Chorus | T : +6448964169 | M :+64272290923

From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Brent Marquis Sent: Tuesday, 7 October 2014 3:38 p.m. To: noc(a)ngw.co.nz; nznog(a)list.waikato.ac.nz

Subject: Re: [nznog] UFB Upload Issues

I've only heard about it recently. I'm investigating at the moment - instead of me running back via account managers to track down technical peoples details, can RSPs experiencing this email me please? Particularly if you have some kind of test connection where you can push some traffic through while I debug at our end.

I have some theories, but I'm double checking all our configuration at the moment to rule that out definitively. I certainly can't replicate it in our lab.

Thanks, Brent

Brent Marquis | Layer 2 Network Specialist Chorus | T : +6448964169 | M :+64272290923

From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of "NOC(a)NGWnoc"@ngw.co.nz Sent: Tuesday, 7 October 2014 2:43 p.m. To: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues

Yes, we have the same issues but also on the 100/50 plans, not just limited to auckland pops, it's with Chorus, but they are scratching heads still, ETA to be confrimed, so far we have been waiting over 5 weeks for a answer. -- NOC @ Next Generation Wholesale - ngw.co.nz On 7/10/2014 2:36 p.m., Dave Mill wrote: Hi all

If any RSPs currently have any reports of 100/20 UFB Right Performing customers having only 2M upload could you contact me offlist? If you do contact me I'm really interested in where your BNG is and where your customers are with the issue.

If any end users are on 100/20 with only 2M upload also feel free to contact me. I'm more interested in your location and your ISP in this case.

Cheers Dave

(representing Inspire Net in this post)

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog