I've only heard about it recently. I'm investigating at the moment - instead of me running back via account managers to track down technical peoples details, can RSPs experiencing this email me please? Particularly if you have some kind of test connection where you can push some traffic through while I debug at our end. I have some theories, but I'm double checking all our configuration at the moment to rule that out definitively. I certainly can't replicate it in our lab. Thanks, Brent Brent Marquis | Layer 2 Network Specialist [cid:image001.png(a)01CFE244.A3C601B0] Chorus | T : +6448964169 | M :+64272290923 From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of "NOC(a)NGWnoc"@ngw.co.nz Sent: Tuesday, 7 October 2014 2:43 p.m. To: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues Yes, we have the same issues but also on the 100/50 plans, not just limited to auckland pops, it's with Chorus, but they are scratching heads still, ETA to be confrimed, so far we have been waiting over 5 weeks for a answer. -- NOC @ Next Generation Wholesale - ngw.co.nz On 7/10/2014 2:36 p.m., Dave Mill wrote: Hi all If any RSPs currently have any reports of 100/20 UFB Right Performing customers having only 2M upload could you contact me offlist? If you do contact me I'm really interested in where your BNG is and where your customers are with the issue. If any end users are on 100/20 with only 2M upload also feel free to contact me. I'm more interested in your location and your ISP in this case. Cheers Dave (representing Inspire Net in this post) _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nzmailto:NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

Nice sleuthing Brent. -Joel On 8 October 2014 16:46, Brent Marquis wrote:

Result!

Having been inundated with replies yesterday (which was great!) we’ve narrowed this issue down to a probable cause…

The new Chorus ‘Accelerate’ plans use Two Rate Three Colour marking on the low priority part of the service, which includes a low priority CIR of 2.5mbps.

The EIR above this, up to the low priority plan speed (i.e 100mbps, 200mbps, 1gig, etc), is marked as discard eligible with the VLAN header DEI(CFI) bit.

Collaborative debug/information sharing with Chorus, Fastcom, Callplus, Inspire and DTS has shown that Chorus is, in fact, forwarding all traffic from our handover as expected, but traffic marked with DEI=1 is lost/discarded in RSP networks.

The affected RSPs are discussing this with equipment vendors to find solutions or workarounds. I’m not keen to name names, as it’s the RSPs equipment, but if anyone has experience with DEI marking causing issues on various switches, I’m sure help would be appreciated.

In parallel, we are looking at disabling DEI marking, but being that this was an integral part of the new plans, I need to engage a few people around chorus to find a way forward here. Considering some RSPs are consuming DEI=1 without issue, we don’t want to break things that they may be doing. Not to mention this would be a change to a lot of our QoS policies, so need some sanity testing and deployment to production, etc.

Thanks to the RSPs involved, although I suspect we all have some work to do to get clear of this issue completely…

Thanks,

Brent

Brent Marquis *|* Layer 2 Network Specialist

Chorus *|* *T : *+6448964169 *| **M :*+64272290923

*From:* nznog-bounces(a)list.waikato.ac.nz [mailto: nznog-bounces(a)list.waikato.ac.nz] *On Behalf Of *Brent Marquis *Sent:* Tuesday, 7 October 2014 3:38 p.m. *To:* noc(a)ngw.co.nz; nznog(a)list.waikato.ac.nz

*Subject:* Re: [nznog] UFB Upload Issues

I’ve only heard about it recently.

I’m investigating at the moment – instead of me running back via account managers to track down technical peoples details, can RSPs experiencing this email me please? Particularly if you have some kind of test connection where you can push some traffic through while I debug at our end.

I have some theories, but I’m double checking all our configuration at the moment to rule that out definitively. I certainly can’t replicate it in our lab.

Thanks,

Brent

Brent Marquis *|* Layer 2 Network Specialist

Chorus *|* *T : *+6448964169 *| **M :*+64272290923

*From:* nznog-bounces(a)list.waikato.ac.nz [ mailto:nznog-bounces(a)list.waikato.ac.nz ] *On Behalf Of *"NOC(a)NGWnoc"@ngw.co.nz *Sent:* Tuesday, 7 October 2014 2:43 p.m. *To:* nznog(a)list.waikato.ac.nz *Subject:* Re: [nznog] UFB Upload Issues

Yes, we have the same issues but also on the 100/50 plans, not just limited to auckland pops, it's with Chorus, but they are scratching heads still, ETA to be confrimed, so far we have been waiting over 5 weeks for a answer.

-- NOC @ Next Generation Wholesale - ngw.co.nz

On 7/10/2014 2:36 p.m., Dave Mill wrote:

Hi all

If any RSPs currently have any reports of 100/20 UFB Right Performing customers having only 2M upload could you contact me offlist? If you do contact me I'm really interested in where your BNG is and where your customers are with the issue.

If any end users are on 100/20 with only 2M upload also feel free to contact me. I'm more interested in your location and your ISP in this case.

Cheers

Dave

(representing Inspire Net in this post)

_______________________________________________

NZNOG mailing list

NZNOG(a)list.waikato.ac.nz

http://list.waikato.ac.nz/mailman/listinfo/nznog

This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

While tinkering with CityLink we hit the same issue. Anything above 2.5Mbps was marked discard eligible and was being discarded by CPE Cisco switches (ME3400). We figured out this is because in the old days the discard eligible bit used to mean "this packet has originated on a token ring network, the byte order of the mac addresses is different". The Cisco was happy to forward the frames across trunks, but would drop the packets on access ports on the assumption the workstation connected wouldn't know how to understand the mac addresses. This was on IOS 12.2.50ish. I don't recall the details, but a change was made by Chorus that resolved the issue. Pete - do you recall the details? Thanks, Dylan On 8 October 2014 16:46, Brent Marquis wrote:

Result!

Having been inundated with replies yesterday (which was great!) we’ve narrowed this issue down to a probable cause…

The new Chorus ‘Accelerate’ plans use Two Rate Three Colour marking on the low priority part of the service, which includes a low priority CIR of 2.5mbps.

The EIR above this, up to the low priority plan speed (i.e 100mbps, 200mbps, 1gig, etc), is marked as discard eligible with the VLAN header DEI(CFI) bit.

Collaborative debug/information sharing with Chorus, Fastcom, Callplus, Inspire and DTS has shown that Chorus is, in fact, forwarding all traffic from our handover as expected, but traffic marked with DEI=1 is lost/discarded in RSP networks.

The affected RSPs are discussing this with equipment vendors to find solutions or workarounds. I’m not keen to name names, as it’s the RSPs equipment, but if anyone has experience with DEI marking causing issues on various switches, I’m sure help would be appreciated.

In parallel, we are looking at disabling DEI marking, but being that this was an integral part of the new plans, I need to engage a few people around chorus to find a way forward here. Considering some RSPs are consuming DEI=1 without issue, we don’t want to break things that they may be doing. Not to mention this would be a change to a lot of our QoS policies, so need some sanity testing and deployment to production, etc.

Thanks to the RSPs involved, although I suspect we all have some work to do to get clear of this issue completely…

Thanks,

Brent

Brent Marquis *|* Layer 2 Network Specialist

Chorus *|* *T : *+6448964169 *| **M :*+64272290923

*From:* nznog-bounces(a)list.waikato.ac.nz [mailto: nznog-bounces(a)list.waikato.ac.nz] *On Behalf Of *Brent Marquis *Sent:* Tuesday, 7 October 2014 3:38 p.m. *To:* noc(a)ngw.co.nz; nznog(a)list.waikato.ac.nz

*Subject:* Re: [nznog] UFB Upload Issues

I’ve only heard about it recently.

I’m investigating at the moment – instead of me running back via account managers to track down technical peoples details, can RSPs experiencing this email me please? Particularly if you have some kind of test connection where you can push some traffic through while I debug at our end.

I have some theories, but I’m double checking all our configuration at the moment to rule that out definitively. I certainly can’t replicate it in our lab.

Thanks,

Brent

Brent Marquis *|* Layer 2 Network Specialist

Chorus *|* *T : *+6448964169 *| **M :*+64272290923

*From:* nznog-bounces(a)list.waikato.ac.nz [ mailto:nznog-bounces(a)list.waikato.ac.nz ] *On Behalf Of *"NOC(a)NGWnoc"@ngw.co.nz *Sent:* Tuesday, 7 October 2014 2:43 p.m. *To:* nznog(a)list.waikato.ac.nz *Subject:* Re: [nznog] UFB Upload Issues

Yes, we have the same issues but also on the 100/50 plans, not just limited to auckland pops, it's with Chorus, but they are scratching heads still, ETA to be confrimed, so far we have been waiting over 5 weeks for a answer.

-- NOC @ Next Generation Wholesale - ngw.co.nz

On 7/10/2014 2:36 p.m., Dave Mill wrote:

Hi all

If any RSPs currently have any reports of 100/20 UFB Right Performing customers having only 2M upload could you contact me offlist? If you do contact me I'm really interested in where your BNG is and where your customers are with the issue.

If any end users are on 100/20 with only 2M upload also feel free to contact me. I'm more interested in your location and your ISP in this case.

Cheers

Dave

(representing Inspire Net in this post)

_______________________________________________

NZNOG mailing list

NZNOG(a)list.waikato.ac.nz

http://list.waikato.ac.nz/mailman/listinfo/nznog

This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Thanks for pushing all that info back into the list. I know of 1 RSP which has Juniper equipment that seems to be discarding CFI/DEI=1. I guess some switches need a bit of config to recognise CFI as DEI or to re-write it entirely (luckily there haven't been any hardware issues yet) Given that impacted RSPs have solutions, or are working with vendors to find solutions - I don't think I'll progress DEI re-write on our side. After looking into it, the changes for that would either be a) painful, by manually changing each customer on a per-RSP basis or b) network wide, changing how the service operates for everyone. Both of these options would be unpalatable for either Chorus or some RSPs. I do think we, at Chorus, need to make the DEI bit usage explicit in our documentation and, based on some questions in the last couple of days, explain exactly how low/high priority and the CIR on low priority operates - I'll take that on board and work on some documentation updates. In the meantime, I'm comfortable taking non-operational questions anytime (probably off list is best) about our services. Anyone wellington based is welcome to catch up over a coffee (or, as it seems to be more commonly mentioned here, beer!) to discuss. I do travel to Auckland, although not often... Thanks, Brent Brent Marquis | Layer 2 Network Specialist  Chorus | T : +6448964169 | M :+64272290923 Brent.marquis(a)chorus.co.nz      From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of NGW Network Operations Centre Sent: Wednesday, 8 October 2014 8:11 p.m. To: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues I can confirm with working with Brent and the Chorus team that yes this is related to what Dylan is referring to. In particular we tracked our issue down to some Cisco switching gear and found: The Layer 2 packet header with CFI / DFI bit set to 0 indicates a standard Ethernet frame and passes to other Ethernet ports without issue. As soon as this is set to 1 then the packets get dropped when forwarding to an Ethernet port which is set as an access port. This is due to the switches thinking CFI bit 1 being set is a Token Ring frame which does not confirm to Ethernet MAC standards. On ME 3600x and 3800x switches you can rewrite the DEI bit with some policy maps from 1 to 0 and it should happily pass this along. Link for those interested: http://www.cisco.com/c/en/us/td/docs/switches/metro/me3600x_3800x/software/r... The above is also likely to be subject to IOS running so you may need to do some further checking or late night upgrades. As far as Juniper gear goes our transport network passed the traffic without issue through trunks and access ports to the Cisco's. Running 12.3 firmware. With the MX's it looks like at least 12.3 or higher supports also rewriting the DEI bit on ingress. Link for those interested: http://www.juniper.net/techpubs/en_US/junos12.3/topics/usage-guidelines/cos-... I also believe version 14 (which is not JTAC recommended) supports DEI bit rewrite on any of the EX equipment also. Anyway hopefully the above may help anyone out there experiencing issue with this. On 8/10/2014 5:35 p.m., Dylan Hall wrote: While tinkering with CityLink we hit the same issue. Anything above 2.5Mbps was marked discard eligible and was being discarded by CPE Cisco switches (ME3400).  We figured out this is because in the old days the discard eligible bit used to mean "this packet has originated on a token ring network, the byte order of the mac addresses is different". The Cisco was happy to forward the frames across trunks, but would drop the packets on access ports on the assumption the workstation connected wouldn't know how to understand the mac addresses. This was on IOS 12.2.50ish.  I don't recall the details, but a change was made by Chorus that resolved the issue. Pete - do you recall the details? Thanks, Dylan On 8 October 2014 16:46, Brent Marquis wrote: Result! Having been inundated with replies yesterday (which was great!) we've narrowed this issue down to a probable cause.   The new Chorus 'Accelerate' plans use Two Rate Three Colour marking on the low priority part of the service, which includes a low priority CIR of 2.5mbps. The EIR above this, up to the low priority plan speed (i.e 100mbps, 200mbps, 1gig, etc), is marked as discard eligible with the VLAN header DEI(CFI) bit.   Collaborative debug/information sharing with Chorus, Fastcom, Callplus, Inspire and DTS has shown that Chorus is, in fact, forwarding all traffic from our handover as expected, but traffic marked with DEI=1 is lost/discarded in RSP networks. The affected RSPs are discussing this with equipment vendors to find solutions or workarounds. I'm not keen to name names, as it's the RSPs equipment, but if anyone has experience with DEI marking causing issues on various switches, I'm sure help would be appreciated.   In parallel, we are looking at disabling DEI marking, but being that this was an integral part of the new plans, I need to engage a few people around chorus to find a way forward here.  Considering some RSPs are consuming DEI=1 without issue, we don't want to break things that they may be doing.  Not to mention this would be a change to a lot of our QoS policies, so need some sanity testing and deployment to production, etc.   Thanks to the RSPs involved, although I suspect we all have some work to do to get clear of this issue completely.   Thanks, Brent   Brent Marquis | Layer 2 Network Specialist  Chorus | T : +6448964169 | M :+64272290923      From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Brent Marquis Sent: Tuesday, 7 October 2014 3:38 p.m. To: noc(a)ngw.co.nz; nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues   I've only heard about it recently. I'm investigating at the moment - instead of me running back via account managers to track down technical peoples details, can RSPs experiencing this email me please?  Particularly if you have some kind of test connection where you can push some traffic through while I debug at our end.   I have some theories, but I'm double checking all our configuration at the moment to rule that out definitively.  I certainly can't replicate it in our lab.   Thanks, Brent   Brent Marquis | Layer 2 Network Specialist  Chorus | T : +6448964169 | M :+64272290923      From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of "NOC(a)NGWnoc"@ngw.co.nz Sent: Tuesday, 7 October 2014 2:43 p.m. To: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues   Yes, we have the same issues but also on the 100/50 plans, not just limited to auckland pops, it's with Chorus, but they are scratching heads still, ETA to be confrimed, so far we have been waiting over 5 weeks for a answer. -- NOC @ Next Generation Wholesale - ngw.co.nz On 7/10/2014 2:36 p.m., Dave Mill wrote: Hi all   If any RSPs currently have any reports of 100/20 UFB Right Performing customers having only 2M upload could you contact me offlist? If you do contact me I'm really interested in where your BNG is and where your customers are with the issue.   If any end users are on 100/20 with only 2M upload also feel free to contact me. I'm more interested in your location and your ISP in this case.   Cheers Dave   (representing Inspire Net in this post) _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog   This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.   _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Hi Don We are using 0x8100 as the outer TPID in all cases for UFB. We're not certain at the moment if our equipment is dropping this traffic or if a provider could be. We have a bit of work to do to establish this still. Would be interested to see Brent's thoughts on this on or off list :) Cheers Dave On Thu, Oct 9, 2014 at 11:52 AM, Don Stokes wrote:

On 09/10/14 09:43, Brent Marquis wrote: (stuff about CFI/DEI=1 traffic being dropped by Certain Hardware Vendors)

Quick question: are the handovers where this problem is occurring using tag type 88a8 or 8100?

If it's using 8100, then the bit should be interpreted as CFI, whereas if it's using 88a8 it should be treated as DEI. Thus, if the type is 88a8, the DEI bit should be treated as such, and equipment that unconditionally drops DEI=1 traffic is Broken And Needs To Be Fixed.

On the other hand, if an 8100 tagged handover is not clearing CFI, it's that end that is Behaving Badly and should be fixed. A box dropping CFI=1 traffic on traffic received with a tag type of 8100 is behaving "correctly".

-- don _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Sorry for the quick reply to myself! It actually seems like Don might not be 100% correct. I don’t have IEEE access to get the .1q standard… But Wikipedia suggests it has been updated in 2005 for CFI to be DEI: http://en.wikipedia.org/wiki/IEEE_802.1ad Drop eligible indicator (DEI): a 1-bit field. (formerly CFI[note 1]http://en.wikipedia.org/wiki/IEEE_802.1Q#cite_note-2[2]http://en.wikipedia.org/wiki/IEEE_802.1Q#cite_note-3) May be used separately or in conjunction with PCP to indicate frames eligible to be dropped in the presence of congestion.[3]http://en.wikipedia.org/wiki/IEEE_802.1Q#cite_note-4 With the note suggesting – “IEEE 802.1Q-2005 clause 9.6” If it’s on Wikipedia, it must be true…. Right? ☺ From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Brent Marquis Sent: Thursday, 9 October 2014 1:02 p.m. To: Dave Mill; Don Stokes Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues Hi Dave, Don, Yes, Don is correct. However, our handover is an 802.1ad interface. In .1ad CFI==DEI My understanding is that this is regardless of SVID ethertype (which is selectable by RSPs when a handover is ordered) it is still an 802.1ad interface. Alternatively - an RSP requesting us to change the SVID ethertype, doesn’t stop our interface from being 802.1ad. I’m probably getting on some shaky ground here, but that is certainly how it appears to work on ALU’s implementation. If I’m wrong above, I’m happy to go back to ALU for clarification, etc. Cheers, Brent Brent Marquis | Layer 2 Network Specialist [cid:image001.png(a)01CFE3BD.F1F1E640] Chorus | T : +6448964169 | M :+64272290923 From: nznog-bounces(a)list.waikato.ac.nzmailto:nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Dave Mill Sent: Thursday, 9 October 2014 12:14 p.m. To: Don Stokes Cc: nznog(a)list.waikato.ac.nzmailto:nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues Hi Don We are using 0x8100 as the outer TPID in all cases for UFB. We're not certain at the moment if our equipment is dropping this traffic or if a provider could be. We have a bit of work to do to establish this still. Would be interested to see Brent's thoughts on this on or off list :) Cheers Dave On Thu, Oct 9, 2014 at 11:52 AM, Don Stokes mailto:don(a)daedalus.co.nz> wrote: On 09/10/14 09:43, Brent Marquis wrote: (stuff about CFI/DEI=1 traffic being dropped by Certain Hardware Vendors) Quick question: are the handovers where this problem is occurring using tag type 88a8 or 8100? If it's using 8100, then the bit should be interpreted as CFI, whereas if it's using 88a8 it should be treated as DEI. Thus, if the type is 88a8, the DEI bit should be treated as such, and equipment that unconditionally drops DEI=1 traffic is Broken And Needs To Be Fixed. On the other hand, if an 8100 tagged handover is not clearing CFI, it's that end that is Behaving Badly and should be fixed. A box dropping CFI=1 traffic on traffic received with a tag type of 8100 is behaving "correctly". -- don _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nzmailto:NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

In our testing about 2 years ago our juniper bng wouldn't handle 0x88a8 as the outer tag when doing the auto conf demux stuff. Dave On Thursday, October 9, 2014, Nathan Ward wrote:

On 9 October 2014 at 1:09:42 pm, Brent Marquis (brent.marquis(a)chorus.co.nz javascript:_e(%7B%7D,'cvml','brent.marquis(a)chorus.co.nz');(mailto: brent.marquis(a)chorus.co.nz javascript:_e(%7B%7D,'cvml','brent.marquis(a)chorus.co.nz');)) wrote:

...

Sorry for the quick reply to myself!

It actually seems like Don might not be 100% correct.

I don’t have IEEE access to get the .1q standard… But Wikipedia suggests

it has been updated in 2005 for CFI to be DEI:

...

http://en.wikipedia.org/wiki/IEEE_802.1ad

Drop eligible indicator (DEI): a 1-bit field. (formerly CFI[note 1](

http://en.wikipedia.org/wiki/IEEE_802.1Q#cite_note-2)[2](http://en.wikipedia.org/wiki/IEEE_802.1Q#cite_note-3)) May be used separately or in conjunction with PCP to indicate frames eligible to be dropped in the presence of congestion.[3]( http://en.wikipedia.org/wiki/IEEE_802.1Q#cite_note-4)

...

With the note suggesting – “IEEE 802.1Q-2005 clause 9.6”

If it’s on Wikipedia, it must be true…. Right? J

The quote above is from the 802.1q page. If you read the page you link to, the 802.1ad page, you get:

In IEEE 802.1ad the CFI is replaced by a Drop Eligibility Indicator (DEI), increasing the functionality of the PCP field.

Key bit is “802.1ad”, not 802.1q. Using 0x88a8 vs 0x8100/0x9100 is signalling that you’re using 802.1ad vs. stacked 802.1q, so should set this bit appropriate to the tag type. I’m with Don on this one - the frame type bits signal how to interpret the following bits, you can’t just swap them around.

People should really just use 0x88a8 - those who aren’t, can I ask why not? Is it because you’re trying to tunnel it over a switch that doesn’t support 802.1ad or something? I’m not saying it’s wrong, I’m interested in understanding the situations in which you might do this.

-- Nathan Ward

Oops, copy and passed the wrong link, but the right quote. http://en.wikipedia.org/wiki/IEEE_802.1Q 802.1Q does show DEI, rather than CFI and appears to be a 2005 change to 802.1q standard. Sorry for the confusion, but it would seem from the wiki information that Don would have been right if we were in 2004 (and/or possibly for anyone running software on switches from pre-2005 amendment to 802.1Q) Thanks, Brent Brent Marquis | Layer 2 Network Specialist [cid:image001.png(a)01CFE3C8.F7184140] Chorus | T : +6448964169 | M :+64272290923 From: Nathan Ward [mailto:nznog(a)daork.net] Sent: Thursday, 9 October 2014 1:26 p.m. To: Brent Marquis; Dave Mill; Don Stokes Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues On 9 October 2014 at 1:09:42 pm, Brent Marquis (brent.marquis(a)chorus.co.nz(mailto:brent.marquis(a)chorus.co.nz))mailto:brent.marquis(a)chorus.co.nz(mailto:brent.marquis(a)chorus.co.nz)) wrote:

Sorry for the quick reply to myself!

It actually seems like Don might not be 100% correct.

I don’t have IEEE access to get the .1q standard… But Wikipedia suggests it has been updated in 2005 for CFI to be DEI:

http://en.wikipedia.org/wiki/IEEE_802.1ad

Drop eligible indicator (DEI): a 1-bit field. (formerly CFI[note 1](http://en.wikipedia.org/wiki/IEEE_802.1Q#cite_note-2)[2](http://en.wikipedia.org/wiki/IEEE_802.1Q#cite_note-3)) May be used separately or in conjunction with PCP to indicate frames eligible to be dropped in the presence of congestion.[3](http://en.wikipedia.org/wiki/IEEE_802.1Q#cite_note-4)

With the note suggesting – “IEEE 802.1Q-2005 clause 9.6”

If it’s on Wikipedia, it must be true…. Right? J

The quote above is from the 802.1q page. If you read the page you link to, the 802.1ad page, you get: In IEEE 802.1ad the CFI is replaced by a Drop Eligibility Indicator (DEI), increasing the functionality of the PCP field. Key bit is “802.1ad”, not 802.1q. Using 0x88a8 vs 0x8100/0x9100 is signalling that you’re using 802.1ad vs. stacked 802.1q, so should set this bit appropriate to the tag type. I’m with Don on this one - the frame type bits signal how to interpret the following bits, you can’t just swap them around. People should really just use 0x88a8 - those who aren’t, can I ask why not? Is it because you’re trying to tunnel it over a switch that doesn’t support 802.1ad or something? I’m not saying it’s wrong, I’m interested in understanding the situations in which you might do this. -- Nathan Ward This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

I have done some reading, it looks like 802.1ad is an amendment to 802.1q. I assumed it was a separate protocol. tl;dr however, 0x8100 means you have a CFI bit, so, set it to 0 please. On 9 October 2014 at 2:02:35 pm, Ewen McNeill (nznog(a)ewen.mcneill.gen.nz) wrote: On 9/10/14 13:26, Nathan Ward wrote: 

Key bit is “802.1ad”, not 802.1q. Using 0x88a8 vs 0x8100/0x9100 is  signalling that you’re using 802.1ad vs. stacked 802.1q, so should set  this bit appropriate to the tag type. 

For those playing along at home:  - ethertype = 0x8100 means 802.1Q or 802.1aq or some pre-standard tag  stacking  - ethertype = 0x88a8 means 802.1ad (standardised double tagging)  - ethertype = 0x9100, 0x9200, ... is pre-standard double tagging (looks  like 0x9100 was the most common of those alternatives).  See, eg,  http://perlmonkey.blogspot.co.nz/2010/07/cisco-versus-ieee-qinq.html  AFAICT the bit in that position in 802.1Q (0x8100) used to mean CFI (eg,  802.1Q-2003) and was changed by 802.1Q-2005 to mean DEI; the bit in that  position in 802.1ad (0x88a8) seems to have always meant DEI. If the bit  is clear (0), then the difference between 802.1Q-2003 and 802.1Q-2005  probably doesn't matter here; if it is set, it obviously does. Joy.  This is not strictly true. 802.1ad updates 802.1q. It’s perhaps unfortunate that the article on Wikipedia is worded the way it is, as though CFI is deprecated. CFI is still valid in CVLAN tags, which 802.1ad specifies as having TPID 0x8100. DEI is only valid in SVLAN frames, which have TPID 0x88a8. <quote> A distinct Ethertype has been allocated (Table 9-1) for use in the TPID field (9.4) of each tag type so they can be distinguished from each other, and from other protocols. </quote> <quote> The semantics and structure of the S-TAG is identical to that of the C-TAG, with the exception that bit 5 in octet 1, the Drop Eligible Indicator (DEI) bit, does not convey a CFI (9.6). The priority and drop_eligible parameters are conveyed in the 3-bit PCP field and the DEI bit as specified in 6.7.3. </quote> If you’re sending 0x8100, you’re setting CFI, so, make it 0.

I’m with Don on this one - the frame type bits signal how to interpret  the following bits, you can’t just swap them around. 

Sadly it looks to be even less clear than that. Using 0x88a8 (as Nathan  suggests) appears to clear up the confusion. But using 0x8100 appears  to require asking "umm, which 802.1Q/802.1aq/pre-standard double-tagging  thing was the sender following" if the CFI/DEI bit was set.  So, eg, force clearing that bit as you receive it into your 0x8100  network (and, eg, apply your own QoS), upgrading everything to speak  only modern 802.1Q (ie assume bit is always DEI), or using 0x88a8  (802.1ad) seem to be the only reasonably compatible options to avoid  equipment confusion.  Definitely looks like something that would benefit from being more  clearly documented by Chorus as a technical item RSPs that choose 0x8100  as their ethertype and use this particular product should watch out for.  My view remains the same - LFCs should implement 802.1ad as written, and use CFI for 0x8100 frames and DEI for 0x88a8 frames. Ewen  PS: It appears one can get actual IEEE 802.1 standards as PDFs here:  http://standards.ieee.org/about/get/802/802.1.html  if one is willing to give up an email address to IEEE (I haven't  actually tried to give them an email address though).  Already done :-) You can also get them from: http://www.stephan-robert.ch/attachments/File/TIN08/802-1Q-2005.pdf https://github.com/yhaz/homeenv/blob/master/specs/802.1ad-2005.pdf --  --  Nathan Ward

Hi Brent, On 9/10/14 16:03, Brent Marquis wrote:

It just gets slightly more confusing…. [...] Which would mean your interpretation is that CFI on 802.1q only applies if there is also an 88a8 802.1ad tag preceding it.

Based on looking at the 802.1Q-2005 specification and 802.1ad-2005 specification that Nathan linked to, it seems to me that the intent of both 802.1Q-2005 and 802.1ad-2005 was that ethertype=0x8100 would mean "interpret the header as a 802.1Q VLAN Tag/802.1ad CVLAN Tag", both of which have the same bit pattern, and semantics -- including a CFI bit rather than a DEI bit. This is section 9.6 in both specifications (which makes sense since 802.1ad was based on 802.1Q). By contrast, in 802.1ad there is a SVLAN tag, with a separate ethertype (0x88a8), which is the same except that it has a DEI bit in the same bit position that 802.1Q/802.1ad has the CFI bit. This is section 9.7 in the 802.1ad specification. It seems to me that the intent was: if ethertype=0x8100 then bit is interpreted as CFI, else if ethertype=0x88a8 then bit is interpreted as DEI. Which gives an unambiguous interpretation, without needing any context (much easier for an ASCI implementation). So I'm pleased IEEE seem to have thought of this problem in advance and tried to avoid ambiguity. However, setting DEI on an 802.1ad frame with ethertype 0x88a8, then changing the ethertype of the outer tag to 0x8100 ("for compatibility" :-) ) seems to either (a) require force-clearing that bit to avoid confusion, or (b) lead to a potential for misinterpretation that causes traffic to be discarded.

IMO, our interface is .1ad and we don’t believe being nice and tweaking the ethertype at RSP request changes that :) but let me see what I can find out from our Vendor.

FWIW, it appears to me that sending two 0x8100 tags stacked is not 802.1ad, but some "pre-standard" double tagging. Even if the frames started out 802.1ad compliant. 802.1ad seems to require a SVLAN tag (0x88a8) around a CVLAN tag (0x8100). Although it appears the difference mostly only matters for the interpretation of this one bit (DEI or CFI). If Chorus is going to offer 0x8100-double-tagging (ie, two 0x8100 tags stacked, pre-802.1ad Q-in-Q) it seems like it'd maximise compatibility if you were able to clear bit 5 (CFI on 0x8100/DEI on 0x88a8) of the tag in that specific case (ie, two stacked 0x8100 tags). Ewen PS: For reference, Nathan's links: http://www.stephan-robert.ch/attachments/File/TIN08/802-1Q-2005.pdf https://github.com/yhaz/homeenv/blob/master/specs/802.1ad-2005.pdf beware the 802.1ad one is a revision 4 draft (they went to revision 6 before the release AFAICT: http://www.ieee802.org/1/pages/802.1ad.html); if you want to argue with vendors it might pay to get the official standard from IEEE directly.

On 09/10/14 13:26, Nathan Ward wrote:

People should really just use 0x88a8 - those who aren’t, can I ask why not? Is it because you’re trying to tunnel it over a switch that doesn’t support 802.1ad or something? I’m not saying it’s wrong, I’m interested in understanding the situations in which you might do this.

There's a bunch of stuff that still doesn't support 802.1ad, even if it does support stacked Q - most Linux distros for starters. If you're looking to do the kind of neat stuff that Linux & friends can do for you at layer 3, then you're pretty much stuck with TPID=8100 and pre-802.1ad semantics. I can't see why you'd use or expect 802.1ad semantics (including DEI) without using the 802.1ad TPID of 88a8. To me, the whole point of offering an outer TPID of 8100 is for backward compatibility with pre-802.1ad compatible devices. -- don

All, I have a question about the converse side of the issue we have struck... Are there any RSPs out there that are classifying on ingress from us based on DEI? i.e.*IF* we did remove DEI=1, is it going to cause any pain? I guess this would most likely be the guys with ALU kit which are using the DEI bit from us to classify, but I suspect that no one is doing this... As has been mentioned, it’s a global change for us to remove DEI. Given the impact to our customers (you guys) and your customers, we're still investigating the possibility of removing DEI. Which has technical and commercial issues around it. Thanks, Brent -----Original Message----- From: Simon Allard [mailto:Simon.Allard(a)team.orcon.net.nz] Sent: Monday, 13 October 2014 12:06 p.m. To: Nathan Ward; Brent Marquis; Dave Mill; Don Stokes Cc: nznog(a)list.waikato.ac.nz Subject: RE: [nznog] UFB Upload Issues

People should really just use 0x88a8 - those who aren’t, can I ask why not? Is it because you’re trying to tunnel it over a switch that doesn’t support 802.1ad or something? I’m not saying it’s wrong, I’m > interested in understanding the situations in which you might do this.

Some devices just don't support it, even modern kit. We use Pseudowires to transport handovers to central ALU 7750's. The ALU 7750 drops the packets if we use 08x88 on the handovers in the Psuedowire code. Apparently 0x8100 is hardcoded! Hopefully one day they will fix it. This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

Hi Brent, I thought the interface specs clearly say that they don't honour the DEI bit (classification is done using the PCP bits) on ingress; while it doesn't say anything about what colour the DEI bit will be on egress, I would have thought that the clear implication is that DEI is not part of that specification. Any RSP doing classification on it is therefore making an unwarranted assumption. I'd be really surprised if anyone is actually doing so, but if you turned it off and it broke something, that would be the RSP's problem, not yours. (Still, it's a fair question to ask.) The robust thing to do, given that the DEI bit behaviour is not defined, and given that setting it has caused problems, is to treat DEI as a Must Be Zero field, ignore any incorrectly set DEI bits on egress, and reliably set DEI=0 on egress. That both defines the DEI behaviour and interoperates with pre 802.1Q-2011 compatible gear. Otherwise, you really need to update the interface specification, with full consultation with RSPs / TCF, to redefine the DEI bit behaviour. If you're going to do that, you should revisit DEI behaviour on ingress as well to allow for (optional?) DEI colouring instead of or as well as PCP colouring. I do think the ingress and egress behaviour should be consistent. -- don On 16/10/14 08:50, Brent Marquis wrote:

All,

I have a question about the converse side of the issue we have struck... Are there any RSPs out there that are classifying on ingress from us based on DEI?

i.e.*IF* we did remove DEI=1, is it going to cause any pain? I guess this would most likely be the guys with ALU kit which are using the DEI bit from us to classify, but I suspect that no one is doing this...

As has been mentioned, it’s a global change for us to remove DEI. Given the impact to our customers (you guys) and your customers, we're still investigating the possibility of removing DEI. Which has technical and commercial issues around it.

Thanks, Brent

-----Original Message----- From: Simon Allard [mailto:Simon.Allard(a)team.orcon.net.nz] Sent: Monday, 13 October 2014 12:06 p.m. To: Nathan Ward; Brent Marquis; Dave Mill; Don Stokes Cc: nznog(a)list.waikato.ac.nz Subject: RE: [nznog] UFB Upload Issues

...

People should really just use 0x88a8 - those who aren’t, can I ask why not? Is it because you’re trying to tunnel it over a switch that doesn’t support 802.1ad or something? I’m not saying it’s wrong, I’m > interested in understanding the situations in which you might do this. Some devices just don't support it, even modern kit.

We use Pseudowires to transport handovers to central ALU 7750's. The ALU 7750 drops the packets if we use 08x88 on the handovers in the Psuedowire code. Apparently 0x8100 is hardcoded!

Hopefully one day they will fix it.

This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Thanks Don,

but if you turned it off and it broke something, that would be the RSP's problem, not yours. (Still, it's a fair question to ask.)

To me, this is the same case as leaving it on - It's the RSPs problem :) but that’s not very helpful either. I was just hoping to see if we might break the service for anyone while fixing it for others.

The robust thing to do, given that the DEI bit behaviour is not defined, and given that setting it has caused problems, is to treat DEI as a Must Be Zero field, ignore any incorrectly set DEI bits on egress, and reliably set DEI=0 on egress. That both defines the DEI behaviour and interoperates with pre 802.1Q-2011 compatible gear.

Yes, that’s the intention of the change we would make - if we can! (assuming your 'ignore on egress' was a typo and you meant 'ignore on ingress'). We currently ignore ingress DEI. Given that [some] RSPs have tested the services where we do have DEI on egress - my concern was that someone might be using it. Also by turning DEI off we, essentially, change the product and may have to jump through some hoops (which could actually involve consultation, rather than a notification). I'm working with our product and other internal teams to see the extent of what we would have to do for turning off DEI on egress. We do have a technical solution to do this, but it is global.

Otherwise, you really need to update the interface specification, with full consultation with RSPs / TCF, to redefine >the DEI bit behaviour. If you're going to do that, you should revisit DEI behaviour on ingress as well to allow for (optional?) DEI colouring instead of or as well as PCP colouring. I do think the ingress and egress behaviour should be consistent.

yes, our product guys are working on the TCF service re-definition at the moment anyway, so I'd expect the new TCF ELAS (if it's still called that) will solidify the specification and provide industry agreement going forward. Our Product team have always intended to include colour awareness on ingress in the future, provided RSPs actually want it. Regards, Brent -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Don Stokes Sent: Thursday, 16 October 2014 11:53 a.m. To: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues Hi Brent, I thought the interface specs clearly say that they don't honour the DEI bit (classification is done using the PCP bits) on ingress; while it doesn't say anything about what colour the DEI bit will be on egress, I would have thought that the clear implication is that DEI is not part of that specification. Any RSP doing classification on it is therefore making an unwarranted assumption. I'd be really surprised if anyone is actually doing so, but if you turned it off and it broke something, that would be the RSP's problem, not yours. (Still, it's a fair question to ask.) The robust thing to do, given that the DEI bit behaviour is not defined, and given that setting it has caused problems, is to treat DEI as a Must Be Zero field, ignore any incorrectly set DEI bits on egress, and reliably set DEI=0 on egress. That both defines the DEI behaviour and interoperates with pre 802.1Q-2011 compatible gear. Otherwise, you really need to update the interface specification, with full consultation with RSPs / TCF, to redefine the DEI bit behaviour. If you're going to do that, you should revisit DEI behaviour on ingress as well to allow for (optional?) DEI colouring instead of or as well as PCP colouring. I do think the ingress and egress behaviour should be consistent. -- don On 16/10/14 08:50, Brent Marquis wrote:

All,

I have a question about the converse side of the issue we have struck... Are there any RSPs out there that are classifying on ingress from us based on DEI?

i.e.*IF* we did remove DEI=1, is it going to cause any pain? I guess this would most likely be the guys with ALU kit which are using the DEI bit from us to classify, but I suspect that no one is doing this...

As has been mentioned, it’s a global change for us to remove DEI. Given the impact to our customers (you guys) and your customers, we're still investigating the possibility of removing DEI. Which has technical and commercial issues around it.

Thanks, Brent

-----Original Message----- From: Simon Allard [mailto:Simon.Allard(a)team.orcon.net.nz] Sent: Monday, 13 October 2014 12:06 p.m. To: Nathan Ward; Brent Marquis; Dave Mill; Don Stokes Cc: nznog(a)list.waikato.ac.nz Subject: RE: [nznog] UFB Upload Issues

...

People should really just use 0x88a8 - those who aren’t, can I ask why not? Is it because you’re trying to tunnel it over a switch that doesn’t support 802.1ad or something? I’m not saying it’s wrong, I’m > interested in understanding the situations in which you might do this. Some devices just don't support it, even modern kit.

We use Pseudowires to transport handovers to central ALU 7750's. The ALU 7750 drops the packets if we use 08x88 on the handovers in the Psuedowire code. Apparently 0x8100 is hardcoded!

Hopefully one day they will fix it.

This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

Hi Brent, Coming from an ALU background that’s my understanding of it also. Providers with ALU based subscriber termination (7750’s etc) are probably unlikely to see this occur. This reminds me of the good old days where mixing different vendors in a deployment would have all sorts of fun unintended ‘quirks’ (Spanning tree issues or LACP flapping, anyone??) If we lived in a perfect world all vendors manufactured hardware compliant with open standards then would never have issues, but then where would the fun be? Cheers, Lance HD NET From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Brent Marquis Sent: Thursday, 9 October 2014 1:02 p.m. To: Dave Mill; Don Stokes Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues Hi Dave, Don, Yes, Don is correct. However, our handover is an 802.1ad interface. In .1ad CFI==DEI My understanding is that this is regardless of SVID ethertype (which is selectable by RSPs when a handover is ordered) it is still an 802.1ad interface. Alternatively - an RSP requesting us to change the SVID ethertype, doesn’t stop our interface from being 802.1ad. I’m probably getting on some shaky ground here, but that is certainly how it appears to work on ALU’s implementation. If I’m wrong above, I’m happy to go back to ALU for clarification, etc. Cheers, Brent Brent Marquis | Layer 2 Network Specialist Chorus | T : +6448964169 | M :+64272290923 From: nznog-bounces(a)list.waikato.ac.nz mailto:nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Dave Mill Sent: Thursday, 9 October 2014 12:14 p.m. To: Don Stokes Cc: nznog(a)list.waikato.ac.nz mailto:nznog(a)list.waikato.ac.nz Subject: Re: [nznog] UFB Upload Issues Hi Don We are using 0x8100 as the outer TPID in all cases for UFB. We're not certain at the moment if our equipment is dropping this traffic or if a provider could be. We have a bit of work to do to establish this still. Would be interested to see Brent's thoughts on this on or off list :) Cheers Dave On Thu, Oct 9, 2014 at 11:52 AM, Don Stokes mailto:don(a)daedalus.co.nz > wrote: On 09/10/14 09:43, Brent Marquis wrote: (stuff about CFI/DEI=1 traffic being dropped by Certain Hardware Vendors) Quick question: are the handovers where this problem is occurring using tag type 88a8 or 8100? If it's using 8100, then the bit should be interpreted as CFI, whereas if it's using 88a8 it should be treated as DEI. Thus, if the type is 88a8, the DEI bit should be treated as such, and equipment that unconditionally drops DEI=1 traffic is Broken And Needs To Be Fixed. On the other hand, if an 8100 tagged handover is not clearing CFI, it's that end that is Behaving Badly and should be fixed. A box dropping CFI=1 traffic on traffic received with a tag type of 8100 is behaving "correctly". -- don _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

Apologies for the BUMP so long after this is all done, but, Does anyone have a working config for re-marking DEI=0, incoming, on a Cisco ME3660, with service-instances, running IOS 15.3 (me360x-universalk9-mz.153) ? (and as I write this, I'm thinking that I might have to upgrade to 15.4 to get the 'set dei 0' command inside a service policy...) Any help at all would be appreciated, My Lab-fu is failing hard today :-) Mike -- On 08/10/14 20:11, NGW Network Operations Centre wrote:

I can confirm with working with Brent and the Chorus team that yes this is related to what Dylan is referring to.

In particular we tracked our issue down to some Cisco switching gear and found:

The Layer 2 packet header with CFI / DFI bit set to 0 indicates a standard Ethernet frame and passes to other Ethernet ports without issue. As soon as this is set to 1 then the packets get dropped when forwarding to an Ethernet port which is set as an access port. This is due to the switches thinking CFI bit 1 being set is a Token Ring frame which does not confirm to Ethernet MAC standards.

On ME 3600x and 3800x switches you can rewrite the DEI bit with some policy maps from 1 to 0 and it should happily pass this along. Link for those interested: http://www.cisco.com/c/en/us/td/docs/switches/metro/me3600x_3800x/software/r...

The above is also likely to be subject to IOS running so you may need to do some further checking or late night upgrades.

As far as Juniper gear goes our transport network passed the traffic without issue through trunks and access ports to the Cisco's. Running 12.3 firmware.

With the MX's it looks like at least 12.3 or higher supports also rewriting the DEI bit on ingress. Link for those interested: http://www.juniper.net/techpubs/en_US/junos12.3/topics/usage-guidelines/cos-...

I also believe version 14 (which is not JTAC recommended) supports DEI bit rewrite on any of the EX equipment also.

Anyway hopefully the above may help anyone out there experiencing issue with this.

On 8/10/2014 5:35 p.m., Dylan Hall wrote:

...

While tinkering with CityLink we hit the same issue. Anything above 2.5Mbps was marked discard eligible and was being discarded by CPE Cisco switches (ME3400).

We figured out this is because in the old days the discard eligible bit used to mean "this packet has originated on a token ring network, the byte order of the mac addresses is different". The Cisco was happy to forward the frames across trunks, but would drop the packets on access ports on the assumption the workstation connected wouldn't know how to understand the mac addresses. This was on IOS 12.2.50ish.

I don't recall the details, but a change was made by Chorus that resolved the issue. Pete - do you recall the details?

Thanks,

Dylan

On 8 October 2014 16:46, Brent Marquis mailto:Brent.Marquis(a)chorus.co.nz> wrote:

Result!

Having been inundated with replies yesterday (which was great!) we’ve narrowed this issue down to a probable cause…

The new Chorus ‘Accelerate’ plans use Two Rate Three Colour marking on the low priority part of the service, which includes a low priority CIR of 2.5mbps.

The EIR above this, up to the low priority plan speed (i.e 100mbps, 200mbps, 1gig, etc), is marked as discard eligible with the VLAN header DEI(CFI) bit.

Collaborative debug/information sharing with Chorus, Fastcom, Callplus, Inspire and DTS has shown that Chorus is, in fact, forwarding all traffic from our handover as expected, but traffic marked with DEI=1 is lost/discarded in RSP networks.

The affected RSPs are discussing this with equipment vendors to find solutions or workarounds. I’m not keen to name names, as it’s the RSPs equipment, but if anyone has experience with DEI marking causing issues on various switches, I’m sure help would be appreciated.

In parallel, we are looking at disabling DEI marking, but being that this was an integral part of the new plans, I need to engage a few people around chorus to find a way forward here. Considering some RSPs are consuming DEI=1 without issue, we don’t want to break things that they may be doing. Not to mention this would be a change to a lot of our QoS policies, so need some sanity testing and deployment to production, etc.

Thanks to the RSPs involved, although I suspect we all have some work to do to get clear of this issue completely…

Thanks,

Brent

Brent Marquis *|* Layer 2 Network Specialist

Chorus *|* *T : *+6448964169 tel:%2B6448964169 *| **M :*+64272290923 tel:%2B64272290923

*From:*nznog-bounces(a)list.waikato.ac.nz mailto:nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz mailto:nznog-bounces(a)list.waikato.ac.nz] *On Behalf Of *Brent Marquis *Sent:* Tuesday, 7 October 2014 3:38 p.m. *To:* noc(a)ngw.co.nz mailto:noc(a)ngw.co.nz; nznog(a)list.waikato.ac.nz mailto:nznog(a)list.waikato.ac.nz

*Subject:* Re: [nznog] UFB Upload Issues

I’ve only heard about it recently.

I’m investigating at the moment – instead of me running back via account managers to track down technical peoples details, can RSPs experiencing this email me please? Particularly if you have some kind of test connection where you can push some traffic through while I debug at our end.

I have some theories, but I’m double checking all our configuration at the moment to rule that out definitively. I certainly can’t replicate it in our lab.

Thanks,

Brent

Brent Marquis *|* Layer 2 Network Specialist

Chorus *|* *T : *+6448964169 tel:%2B6448964169 *| **M :*+64272290923 tel:%2B64272290923

*From:*nznog-bounces(a)list.waikato.ac.nz mailto:nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] *On Behalf Of *"NOC(a)NGWnoc"@ngw.co.nz mailto:%22NOC(a)NGWnoc%22(a)ngw.co.nz *Sent:* Tuesday, 7 October 2014 2:43 p.m. *To:* nznog(a)list.waikato.ac.nz mailto:nznog(a)list.waikato.ac.nz *Subject:* Re: [nznog] UFB Upload Issues

Yes, we have the same issues but also on the 100/50 plans, not just limited to auckland pops, it's with Chorus, but they are scratching heads still, ETA to be confrimed, so far we have been waiting over 5 weeks for a answer.

-- NOC @ Next Generation Wholesale - ngw.co.nz http://ngw.co.nz

On 7/10/2014 2:36 p.m., Dave Mill wrote:

Hi all

If any RSPs currently have any reports of 100/20 UFB Right Performing customers having only 2M upload could you contact me offlist? If you do contact me I'm really interested in where your BNG is and where your customers are with the issue.

If any end users are on 100/20 with only 2M upload also feel free to contact me. I'm more interested in your location and your ISP in this case.

Cheers

Dave

(representing Inspire Net in this post)

_______________________________________________

NZNOG mailing list

NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz

http://list.waikato.ac.nz/mailman/listinfo/nznog

This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

A reply to the list with the solution; Huge thanks to Brent from Chorus, you sir are a legend... The solution, in case anyone missed it (like I did); Chorus re-designed the service slightly so that you can order UFB with DEI = 0 (can't remember which plans are affected off the top of my head) So, a bit of a facepalm on our side, I had this thread put aside to read when convienient... 10 months later I wish I'd read the thread last year... :-) Thanks again Brent, thanks list, very helpful, as always. Regards, Mike On 20/08/15 14:25, Mike Taylor wrote:

Apologies for the BUMP so long after this is all done, but,

Does anyone have a working config for re-marking DEI=0, incoming, on a Cisco ME3660, with service-instances, running IOS 15.3 (me360x-universalk9-mz.153) ?

(and as I write this, I'm thinking that I might have to upgrade to 15.4 to get the 'set dei 0' command inside a service policy...)

Any help at all would be appreciated, My Lab-fu is failing hard today :-)

Mike --

On 08/10/14 20:11, NGW Network Operations Centre wrote:

...

I can confirm with working with Brent and the Chorus team that yes this is related to what Dylan is referring to.

In particular we tracked our issue down to some Cisco switching gear and found:

The Layer 2 packet header with CFI / DFI bit set to 0 indicates a standard Ethernet frame and passes to other Ethernet ports without issue. As soon as this is set to 1 then the packets get dropped when forwarding to an Ethernet port which is set as an access port. This is due to the switches thinking CFI bit 1 being set is a Token Ring frame which does not confirm to Ethernet MAC standards.

On ME 3600x and 3800x switches you can rewrite the DEI bit with some policy maps from 1 to 0 and it should happily pass this along. Link for those interested: http://www.cisco.com/c/en/us/td/docs/switches/metro/me3600x_3800x/software/r...

The above is also likely to be subject to IOS running so you may need to do some further checking or late night upgrades.

As far as Juniper gear goes our transport network passed the traffic without issue through trunks and access ports to the Cisco's. Running 12.3 firmware.

With the MX's it looks like at least 12.3 or higher supports also rewriting the DEI bit on ingress. Link for those interested: http://www.juniper.net/techpubs/en_US/junos12.3/topics/usage-guidelines/cos-...

I also believe version 14 (which is not JTAC recommended) supports DEI bit rewrite on any of the EX equipment also.

Anyway hopefully the above may help anyone out there experiencing issue with this.

On 8/10/2014 5:35 p.m., Dylan Hall wrote:

...

While tinkering with CityLink we hit the same issue. Anything above 2.5Mbps was marked discard eligible and was being discarded by CPE Cisco switches (ME3400).

We figured out this is because in the old days the discard eligible bit used to mean "this packet has originated on a token ring network, the byte order of the mac addresses is different". The Cisco was happy to forward the frames across trunks, but would drop the packets on access ports on the assumption the workstation connected wouldn't know how to understand the mac addresses. This was on IOS 12.2.50ish.

I don't recall the details, but a change was made by Chorus that resolved the issue. Pete - do you recall the details?

Thanks,

Dylan

On 8 October 2014 16:46, Brent Marquis mailto:Brent.Marquis(a)chorus.co.nz> wrote:

Result!

Having been inundated with replies yesterday (which was great!) we’ve narrowed this issue down to a probable cause…

The new Chorus ‘Accelerate’ plans use Two Rate Three Colour marking on the low priority part of the service, which includes a low priority CIR of 2.5mbps.

The EIR above this, up to the low priority plan speed (i.e 100mbps, 200mbps, 1gig, etc), is marked as discard eligible with the VLAN header DEI(CFI) bit.

Collaborative debug/information sharing with Chorus, Fastcom, Callplus, Inspire and DTS has shown that Chorus is, in fact, forwarding all traffic from our handover as expected, but traffic marked with DEI=1 is lost/discarded in RSP networks.

The affected RSPs are discussing this with equipment vendors to find solutions or workarounds. I’m not keen to name names, as it’s the RSPs equipment, but if anyone has experience with DEI marking causing issues on various switches, I’m sure help would be appreciated.

In parallel, we are looking at disabling DEI marking, but being that this was an integral part of the new plans, I need to engage a few people around chorus to find a way forward here. Considering some RSPs are consuming DEI=1 without issue, we don’t want to break things that they may be doing. Not to mention this would be a change to a lot of our QoS policies, so need some sanity testing and deployment to production, etc.

Thanks to the RSPs involved, although I suspect we all have some work to do to get clear of this issue completely…

Thanks,

Brent

Brent Marquis *|* Layer 2 Network Specialist

Chorus *|* *T : *+6448964169 tel:%2B6448964169 *| **M :*+64272290923 tel:%2B64272290923

*From:*nznog-bounces(a)list.waikato.ac.nz mailto:nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz mailto:nznog-bounces(a)list.waikato.ac.nz] *On Behalf Of *Brent Marquis *Sent:* Tuesday, 7 October 2014 3:38 p.m. *To:* noc(a)ngw.co.nz mailto:noc(a)ngw.co.nz; nznog(a)list.waikato.ac.nz mailto:nznog(a)list.waikato.ac.nz

*Subject:* Re: [nznog] UFB Upload Issues

I’ve only heard about it recently.

I’m investigating at the moment – instead of me running back via account managers to track down technical peoples details, can RSPs experiencing this email me please? Particularly if you have some kind of test connection where you can push some traffic through while I debug at our end.

I have some theories, but I’m double checking all our configuration at the moment to rule that out definitively. I certainly can’t replicate it in our lab.

Thanks,

Brent

Brent Marquis *|* Layer 2 Network Specialist

Chorus *|* *T : *+6448964169 tel:%2B6448964169 *| **M :*+64272290923 tel:%2B64272290923

*From:*nznog-bounces(a)list.waikato.ac.nz mailto:nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] *On Behalf Of *"NOC(a)NGWnoc"@ngw.co.nz mailto:%22NOC(a)NGWnoc%22(a)ngw.co.nz *Sent:* Tuesday, 7 October 2014 2:43 p.m. *To:* nznog(a)list.waikato.ac.nz mailto:nznog(a)list.waikato.ac.nz *Subject:* Re: [nznog] UFB Upload Issues

Yes, we have the same issues but also on the 100/50 plans, not just limited to auckland pops, it's with Chorus, but they are scratching heads still, ETA to be confrimed, so far we have been waiting over 5 weeks for a answer.

-- NOC @ Next Generation Wholesale - ngw.co.nz http://ngw.co.nz

On 7/10/2014 2:36 p.m., Dave Mill wrote:

Hi all

If any RSPs currently have any reports of 100/20 UFB Right Performing customers having only 2M upload could you contact me offlist? If you do contact me I'm really interested in where your BNG is and where your customers are with the issue.

If any end users are on 100/20 with only 2M upload also feel free to contact me. I'm more interested in your location and your ISP in this case.

Cheers

Dave

(representing Inspire Net in this post)

_______________________________________________

NZNOG mailing list

NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz

http://list.waikato.ac.nz/mailman/listinfo/nznog

This communication, including any attachments, is confidential and may be legally privileged. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. No confidentiality or privilege is waived or lost by any mis-transmission or error. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz mailto:NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog

---

NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog