Hi All, Curious if anyone has seen this issue previously or could offer any advice on the issue: We have a VLL/EOMPLS with Pipe from Sydney to NZ which then utilises a City Link QinQ service across town in Auckland. We've had an issue ongoing for some time and it only relates to multicast traffic traversing the path. Essentially this issue started when we couldn't establish a OSPF session over the link - yet all other traffic flows as expected. When we broke this link down to basics (laptop either end) we send one multicast packet in the Sydney end - we then receive that same packet 97 times out the Auckland end. (Over the Pipe and City link path) We see this issue when we ping 224.0.0.5 over the link and get that same exact packet 97 times on the other end. We've seen this when doing WireShark captures on the port. The captures show the exact same packet 97 times (ie same payload/sequence number). Pipe seem to be running out of ideas to track the issue. Being a EoMPLS circuit I doubt the issue is on the Pipe side but more on the City Link side. When CityLink were engaged in the issue they did what they referred to as an RFC test - as I understand it, unicast testing only and never engaged further. This issue only seems to effect link local multicast traffic - ie 224.0.0.x/24 All other traffic appears to flow normally/as expected. Obviously we are chasing pipe to test each side individually to work out who is doing the packet replication - however I am curious if anyone has had issues forming OSPF sessions over a City link service ?? Or if anyone can offer advice or seen a similar issue ? My brain boggles thinking how this is even possible or what would cause it. Ben Cornish General Manager [Description: Description: cid:image001.png(a)01CCB4C2.B40A97F0] Over the Wire Pty Ltd - An Impirical Company Level 3, 24 Little Edward St, Spring Hill QLD 4000 t +61 7 3847 9292 m +61 417 617 204 e ben.cornish(a)overthewire.com.aumailto:ben.cornish(a)overthewire.com.au www.overthewire.com.auhttp://www.overthewire.com.au/
My thought would be the opposite - if there's a layer 2 loop then you'd see
a lot more than 97 of each packet, but a routing loop in the MPLS cloud
could drop your packets out until the TTL runs out
On 16 Dec 2014 16:15, "Ben Cornish"
Hi All,
Curious if anyone has seen this issue previously or could offer any advice on the issue:
We have a VLL/EOMPLS with Pipe from Sydney to NZ which then utilises a City Link QinQ service across town in Auckland.
We’ve had an issue ongoing for some time and it only relates to multicast traffic traversing the path.
Essentially this issue started when we couldn’t establish a OSPF session over the link – yet all other traffic flows as expected.
When we broke this link down to basics (laptop either end) we send one multicast packet in the Sydney end – we then receive that same packet 97 times out the Auckland end. (Over the Pipe and City link path)
We see this issue when we ping 224.0.0.5 over the link and get that same exact packet 97 times on the other end.
We’ve seen this when doing WireShark captures on the port. The captures show the exact same packet 97 times (ie same payload/sequence number).
Pipe seem to be running out of ideas to track the issue. Being a EoMPLS circuit I doubt the issue is on the Pipe side but more on the City Link side.
When CityLink were engaged in the issue they did what they referred to as an RFC test – as I understand it, unicast testing only and never engaged further.
This issue only seems to effect link local multicast traffic – ie 224.0.0.x/24
All other traffic appears to flow normally/as expected.
Obviously we are chasing pipe to test each side individually to work out who is doing the packet replication – however I am curious if anyone has had issues forming OSPF sessions over a City link service ??
Or if anyone can offer advice or seen a similar issue ?
My brain boggles thinking how this is even possible or what would cause it.
Ben Cornish General Manager
[image: Description: Description: cid:image001.png(a)01CCB4C2.B40A97F0]
Over the Wire Pty Ltd – An Impirical Company Level 3, 24 Little Edward St, Spring Hill QLD 4000 t +61 7 3847 9292
m +61 417 617 204 e ben.cornish(a)overthewire.com.au www.overthewire.com.au
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
And by drop I mean deliver to the other side of the VLL
On 16 Dec 2014 16:20, "Sam Russell"
My thought would be the opposite - if there's a layer 2 loop then you'd see a lot more than 97 of each packet, but a routing loop in the MPLS cloud could drop your packets out until the TTL runs out On 16 Dec 2014 16:15, "Ben Cornish"
wrote: Hi All,
Curious if anyone has seen this issue previously or could offer any advice on the issue:
We have a VLL/EOMPLS with Pipe from Sydney to NZ which then utilises a City Link QinQ service across town in Auckland.
We’ve had an issue ongoing for some time and it only relates to multicast traffic traversing the path.
Essentially this issue started when we couldn’t establish a OSPF session over the link – yet all other traffic flows as expected.
When we broke this link down to basics (laptop either end) we send one multicast packet in the Sydney end – we then receive that same packet 97 times out the Auckland end. (Over the Pipe and City link path)
We see this issue when we ping 224.0.0.5 over the link and get that same exact packet 97 times on the other end.
We’ve seen this when doing WireShark captures on the port. The captures show the exact same packet 97 times (ie same payload/sequence number).
Pipe seem to be running out of ideas to track the issue. Being a EoMPLS circuit I doubt the issue is on the Pipe side but more on the City Link side.
When CityLink were engaged in the issue they did what they referred to as an RFC test – as I understand it, unicast testing only and never engaged further.
This issue only seems to effect link local multicast traffic – ie 224.0.0.x/24
All other traffic appears to flow normally/as expected.
Obviously we are chasing pipe to test each side individually to work out who is doing the packet replication – however I am curious if anyone has had issues forming OSPF sessions over a City link service ??
Or if anyone can offer advice or seen a similar issue ?
My brain boggles thinking how this is even possible or what would cause it.
Ben Cornish General Manager
[image: Description: Description: cid:image001.png(a)01CCB4C2.B40A97F0]
Over the Wire Pty Ltd – An Impirical Company Level 3, 24 Little Edward St, Spring Hill QLD 4000 t +61 7 3847 9292
m +61 417 617 204 e ben.cornish(a)overthewire.com.au www.overthewire.com.au
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
We use Citybridge as a wellington-auckland transit provider and at one point we had an issue with the OSPF that underpins our BGP and LSPs. The issue wasn't immediately apparent when we commissioned the service but appeared abruptly a few weeks later. The incident report stated that they had to increase the Multicast limit on the transit. Not particularly clear on what limits were in place or why they suddenly became an issue but its pretty clear that there is some multicast manipulation in place. I would suggest taking this back to Citylink again, i'm guessing that getting a man in the middle to test each leg of the transit is not going to happen?
Hi All, Thanks for all the responses. Thankfully City Link saw this message - opened a fault - saw what we were seeing and committed resources to fixing it. Cheers. From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Ben Cornish Sent: Tuesday, 16 December 2014 1:15 PM To: nznog(a)list.waikato.ac.nz Subject: [nznog] Multicast traffic replication issue Hi All, Curious if anyone has seen this issue previously or could offer any advice on the issue: We have a VLL/EOMPLS with Pipe from Sydney to NZ which then utilises a City Link QinQ service across town in Auckland. We've had an issue ongoing for some time and it only relates to multicast traffic traversing the path. Essentially this issue started when we couldn't establish a OSPF session over the link - yet all other traffic flows as expected. When we broke this link down to basics (laptop either end) we send one multicast packet in the Sydney end - we then receive that same packet 97 times out the Auckland end. (Over the Pipe and City link path) We see this issue when we ping 224.0.0.5 over the link and get that same exact packet 97 times on the other end. We've seen this when doing WireShark captures on the port. The captures show the exact same packet 97 times (ie same payload/sequence number). Pipe seem to be running out of ideas to track the issue. Being a EoMPLS circuit I doubt the issue is on the Pipe side but more on the City Link side. When CityLink were engaged in the issue they did what they referred to as an RFC test - as I understand it, unicast testing only and never engaged further. This issue only seems to effect link local multicast traffic - ie 224.0.0.x/24 All other traffic appears to flow normally/as expected. Obviously we are chasing pipe to test each side individually to work out who is doing the packet replication - however I am curious if anyone has had issues forming OSPF sessions over a City link service ?? Or if anyone can offer advice or seen a similar issue ? My brain boggles thinking how this is even possible or what would cause it. Ben Cornish General Manager [Description: Description: cid:image001.png(a)01CCB4C2.B40A97F0] Over the Wire Pty Ltd - An Impirical Company Level 3, 24 Little Edward St, Spring Hill QLD 4000 t +61 7 3847 9292 m +61 417 617 204 e ben.cornish(a)overthewire.com.aumailto:ben.cornish(a)overthewire.com.au www.overthewire.com.auhttp://www.overthewire.com.au/
participants (3)
-
Ben Cornish
-
Sam Russell
-
Tim Price