On 21/10/16 10:23, Ewen McNeill wrote:
Does anyone have any suggestions for how to investigate what looks like certain paths of inter-virtual-router layer 3 switching (routing, via routes leaked into the VR) on a Juniper EX4600 (and I'd guess QFX5100) being sent via the supervisor CPU?
For those playing along at home, it turns out that the Juniper EX platform (apparently including the EX4600) does not support route leaking of direct (connected) routes between virtual router instances: https://kb.juniper.net/InfoCenter/index?page=content&id=KB23027&smlogin=true This appears to manifest for us as: - the direct routes are visible in the RIB (ie, from the supervisor CPU) and "look normal" (eg, "show route table ..."). - the direct routes are not installed into the FIB (ie, for the packet forwarding engine) - normally in this case the traffic would follow some other (shorter prefix) covering route (eg, default) instead, and perhaps take a round about route; but in our case these VRs deliberately have no covering route or default route - the default behaviour if there is no route in the FIB is to reject the packets - which is done by queueing the packets to the supervisor CPU... - and the supervisor CPU looks at its routing table (RIB) before generating an ICMP Unreachable, finds a route in the RIB, so forwards the traffic itself -- ie "process" switching, in this case supervisor switching As the KB article dryly observes "this software forwarding results in heavy overhead on the CPU which causes high CPU utilization and can cause unintended consequences". (I can't help thinking that maybe in this case generating the ICMP Destination Unreachable anyway would have made it more obvious that something was wrong, than software forwarding the packet...) The main work around is to ensure that all routes you leak on an EX platform have a gateway router to send traffic to (rather than relying on an ARP lookup) -- and if necessary to construct such an inter-VR route with an additional linknet hop. (The KB article suggests a firewall filter alternative, which presumably leverages other ASIC features.) Thanks to the very helpful Juniper NZ staff and Juniper JTAC staff for assistance with investigating this, including JTAC for pointing us at that KB article. Since this configuration on my client's EX4600 is a transitional one (separating one routing table into several, in advance of a new device being installed which will route between the VRs eliminating the need for leaked routes), we're going to redesign the transition plan to ensure that there is a routed path with a gateway in both directions, now as well rather than just later. (In our case the performance hit was unidirectional as one direction happened to have a linknet to an existing router, and the other direction had an interface that was the default gateway for the connected subnet.) Ewen PS: I still do not know any way to recognise this "process switching" happening other than observing the high latency (which grows rapidly under any traffic load) symptoms, spotting the traffic in "monitor traffic..." (which hints the supervisor CPU was involved) and then looking for directly connected subnets that have been leaked between VRs.