Juniper EX4600 "process switching"?
Does anyone have any suggestions for how to investigate what looks like certain paths of inter-virtual-router layer 3 switching (routing, via routes leaked into the VR) on a Juniper EX4600 (and I'd guess QFX5100) being sent via the supervisor CPU? Searching online turns up lots of advice for investigating, eg, "process switching" on Cisco (eg, http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-software-re...), but I'm not finding much documented for Juniper :-( From the customer point of view the most obvious symptom is approximately 10-20 ms additional latency on a local routed path (VLAN A on VR 1 to VLAN B on VR 2), but apparently only in one direction (ie, if we bypass the EX4600 in the problematic direction then it "seems to be okay". It happens even when there's basically no traffic (eg, under 1Mbps on a 1Gbps link), and only a couple of dozen devices on each VLAN. (At times we've seen the latency as high as 1000+ ms -- so over 1s, for local traffic -- but that was at a busier site.) From an internal point of view, the most obvious symptom is that the forwarding-options analzyer shows ingress/egress on one of the VLANs, but only ingress on the other VLAN; and "monitor traffic ..." shows only the missing egress traffic. Since I'd understood "monitor traffic" to only see traffic that goes via the supervisor CPU, I'm starting to wonder if the latency is coming due to the packet flow in that direction going through the processor CPU -- for some unknown reason. This is happening at two sites, but they're basically identical EX4600s facing basically identical switches, and equivalently configured routing instances in each. The two VLANs are sub-interfaces on the same aggregation (ae1) to the same switch, so it's sort of hairpinned, but via leaked static routes in the two routing instances. The on-VLAN hardware at each site is pretty similar too. So we're guessing the cause is the same, but don't know what it is. We've already tried testing with the aggregation down to a single link, and it appears to make no difference. We've also forced ethernet flow control off at both ends, without making a difference. Any thoughts, particularly to see what path through the EX4600 packets are taking, welcomed. Or documentation for how the EX4600/QFX5100 packet forwarding engine architecture works (I can find M/T/MX documents, but not the one I want...). Thanks, Ewen
On 21/10/16 10:23, Ewen McNeill wrote:
Does anyone have any suggestions for how to investigate what looks like certain paths of inter-virtual-router layer 3 switching (routing, via routes leaked into the VR) on a Juniper EX4600 (and I'd guess QFX5100) being sent via the supervisor CPU?
For those playing along at home, it turns out that the Juniper EX platform (apparently including the EX4600) does not support route leaking of direct (connected) routes between virtual router instances: https://kb.juniper.net/InfoCenter/index?page=content&id=KB23027&smlogin=true This appears to manifest for us as: - the direct routes are visible in the RIB (ie, from the supervisor CPU) and "look normal" (eg, "show route table ..."). - the direct routes are not installed into the FIB (ie, for the packet forwarding engine) - normally in this case the traffic would follow some other (shorter prefix) covering route (eg, default) instead, and perhaps take a round about route; but in our case these VRs deliberately have no covering route or default route - the default behaviour if there is no route in the FIB is to reject the packets - which is done by queueing the packets to the supervisor CPU... - and the supervisor CPU looks at its routing table (RIB) before generating an ICMP Unreachable, finds a route in the RIB, so forwards the traffic itself -- ie "process" switching, in this case supervisor switching As the KB article dryly observes "this software forwarding results in heavy overhead on the CPU which causes high CPU utilization and can cause unintended consequences". (I can't help thinking that maybe in this case generating the ICMP Destination Unreachable anyway would have made it more obvious that something was wrong, than software forwarding the packet...) The main work around is to ensure that all routes you leak on an EX platform have a gateway router to send traffic to (rather than relying on an ARP lookup) -- and if necessary to construct such an inter-VR route with an additional linknet hop. (The KB article suggests a firewall filter alternative, which presumably leverages other ASIC features.) Thanks to the very helpful Juniper NZ staff and Juniper JTAC staff for assistance with investigating this, including JTAC for pointing us at that KB article. Since this configuration on my client's EX4600 is a transitional one (separating one routing table into several, in advance of a new device being installed which will route between the VRs eliminating the need for leaked routes), we're going to redesign the transition plan to ensure that there is a routed path with a gateway in both directions, now as well rather than just later. (In our case the performance hit was unidirectional as one direction happened to have a linknet to an existing router, and the other direction had an interface that was the default gateway for the connected subnet.) Ewen PS: I still do not know any way to recognise this "process switching" happening other than observing the high latency (which grows rapidly under any traffic load) symptoms, spotting the traffic in "monitor traffic..." (which hints the supervisor CPU was involved) and then looking for directly connected subnets that have been leaked between VRs.
On 21/10/16 10:23, Ewen McNeill wrote:
For those playing along at home, it turns out that the Juniper EX platform (apparently including the EX4600) does not support route leaking of direct (connected) routes between virtual router instances:
https://kb.juniper.net/InfoCenter/index?page=content&id=KB23027&smlogin=true
Ewen, Thanks for posting that. I take it the same limitation is not present on QFX5x00's? - Donald Neal Donald Neal | Network Architect D: +64 9 929 2958E: Donald.Neal(a)vocusgroup.co.nz M: +64 27 808 1023W: vocus.co.nz A: Level 4, 191 Queen Street, Auckland 1010
On 24/10/16 21:20, Donald Neal wrote:
On 21/10/16 10:23, Ewen McNeill wrote:
For those playing along at home, it turns out that the Juniper EX platform (apparently including the EX4600) does not support route leaking of direct (connected) routes between virtual router instances:
https://kb.juniper.net/InfoCenter/index?page=content&id=KB23027&smlogin=true
Thanks for posting that. I take it the same limitation is not present on QFX5x00's?
I've been told that the same issue has been replicated on a Juniper QFX5100 (which I think has a fairly similar set of ASICs to the EX4600, so presumably the same hardware options for the packet forwarding engine). I don't know about the rest of the QFX5x00 line, but I'd at least keep an eye out for it. "show route forwarding-table destination ... table ..." has also been mentioned as a possible diagnostic option. Apparently all the affected routes have a type of "rtbl", which is not a type that appears in the "show route forwarding-table" documentation: http://www.juniper.net/techpubs/en_US/junos14.1/topics/reference/command-sum... (nor can I find documented elsewhere, but it does appear in a few "why is this happening" discussions online). Assuming the VRs on the EX have only linknets to another ultimate router/gateway for the VLAN/subnets with the actual hosts on them, the main impact would seem to be additional latency on, eg, monitoring to the far end of the linknet. Eg on: v201 -> router -> v10 -> VR-10 EX VR-20 -> v20 -> router -> v202 with routes leaked between VR-10 and VR-20, it seems like a monitoring system in v201 might see the extra latency on pinging the router between v20 and v202 on the v20 address. But traffic from v201 to v202 would be handled via a gateway route on the EX, and thus hardware accelerated. In which case monitoring the v202 address of the router might be more representative in terms of latency. (This monitoring case may also be why someone decided to implement the process switching work around....) Now that we know about it, I think it's a limitation that it's possible to design around, by either ensuring there's an intermediate router between the EX and the final hosts, or avoiding leaked routes. Where it isn't possible to design around, you probably want something based on a more capable router chipset... Ewen
participants (2)
-
Donald Neal -
Ewen McNeill