Hi, I was wondering if anyone has seen a general trend to supporting >1500B MTUs on Internet transit and peering sessions? I recall from years ago that AGC/ANC/PacNet supported interface-default MTUs, meaning that ATM connected customers had 4470B MTU. Obviously these days with most people running GE or XE in their networks with MPLS and 802.1Q you are looking at networks capable internally of jumbograms, but is anyone running this across network boundaries? Are there any plans for the NZ IXs to be jumbo-capable? I'd be interested in hearing from anyone who is jumbo-enabled on their transit connections, and particularly interested in hearing if they are seeing any significant number of packets that are >1500B on the Internet. aj
An ivory tower comment on that: The current discussions in the IRTF Routing Research Group, and in particular the LISP proposal, strongly suggest a future in which a lot of inter-ISP traffic will be encapsulated, in order to avoid unbounded growth in the BGP4 table. So it would be wise for IXs to allow for a reasonable encapsulation header on top of 1500B. LISP currently needs a 56B header for IPv6 encapsulation. See section 5 of http://tools.ietf.org/id/draft-farinacci-lisp Among other things, it says: Based on informal surveys of large ISP traffic patterns, it appears that most transit paths can accommodate a path MTU of at least 4470 bytes. Brian On 2008-09-22 10:47, Alastair Johnson wrote:
Hi,
I was wondering if anyone has seen a general trend to supporting >1500B MTUs on Internet transit and peering sessions?
I recall from years ago that AGC/ANC/PacNet supported interface-default MTUs, meaning that ATM connected customers had 4470B MTU.
Obviously these days with most people running GE or XE in their networks with MPLS and 802.1Q you are looking at networks capable internally of jumbograms, but is anyone running this across network boundaries?
Are there any plans for the NZ IXs to be jumbo-capable?
I'd be interested in hearing from anyone who is jumbo-enabled on their transit connections, and particularly interested in hearing if they are seeing any significant number of packets that are >1500B on the Internet.
aj _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On Mon, Sep 22, 2008 at 06:47:39AM +0800, Alastair Johnson said:
Are there any plans for the NZ IXs to be jumbo-capable?
Yup. In the list of design aims for the IX networks, it's not at the top of the list, but it's not at the bottom either. As most modern gear supports large frames, it is largely happening as a function of normal switch replacement. All switches deployed in the last couple of years on the APE and WIX fabrics support 1998 bytes (at 10/100Mb) and (at least) 9000 bytes for 1000Mb. That doesn't necessarily mean, though, that you'll get those MTU's across the exchanges - sometimes we do things that steal bytes, and there are still some older switches around with lower limits (particularly Cisco 2950's, which are 1500 bytes for all ports, and 3550's, which are ~2000 bytes on all ports). Right now, I think you'll get 9000 bytes on customer facing GE ports on APE, 2000 on WIX, and 1500 bytes on 100Mb ports on both IX's. I haven't tested, that, though, so YMMV. In Auckland, I believe the core is 9000 byte safe, but there are a bunch of 2950's strewn around the edge that will limit to 1500 bytes. There is no fixed plan to pull them out - they'll be replaced as demand dictates. In Wellington, there are less edge 2950's, but there are a dozen switches in the core which need to be replaced before we can carry
2000 bytes. I hope we'll have the majority of that work done by the end of the year, but it'll depend on what other work the outside plant guys have on.
So right now, if >1500 byte frames are a requirement, it's always good to mention that when you are ordering a new port, just to remind the provisioning guys that the 2950's really do need to be thrown in the harbour.
I'd be interested in hearing from anyone who is jumbo-enabled on their transit connections, and particularly interested in hearing if they are seeing any significant number of packets that are >1500B on the Internet.
From a random port on APE:
Transmit TenGigabitEthernet1/0/2 Receive 842676042645 Bytes 30787514711 Bytes 4126605040 Unicast frames 2466873557 Unicast frames 320790489 Multicast frames 14490379 Multicast frames 112278583 Broadcast frames 16399379 Broadcast frames 0 7 collision frames 1674685453 Minimum size frames 0 8 collision frames 3808990723 65 to 127 byte frames 0 9 collision frames 3681511885 128 to 255 byte frames 0 10 collision frames 1316050645 256 to 511 byte frames 0 11 collision frames 2871906426 512 to 1023 byte frames 0 12 collision frames 2013979673 1024 to 1518 byte frames 0 VLAN discard frames 15540398 Valid frames, too large 610595605 64 byte frames 0 Valid frames, too small 2052662643 127 byte frames 2861097914 255 byte frames 0 Too old frames 2208233399 511 byte frames 0 Valid oversize frames 23308406 1023 byte frames 0 System FCS error frames 1081047305 1518 byte frames 0 RxPortFifoFull drop frame 11015598 Too large frames I'm not quite sure how to intepret that, as the frame size breakdowns don't seem to sum to the rx/tx totals, but in any case, large frames don't appear to be a significant proportion (but they are non zero, and I'd imagine that proportion is climbing). Cheers Simon
On 22/09/2008, at 4:40 PM, Simon Blake wrote:
I'm not quite sure how to intepret that, as the frame size breakdowns don't seem to sum to the rx/tx totals, but in any case, large frames don't appear to be a significant proportion (but they are non zero, and I'd imagine that proportion is climbing).
I wonder, is it feasible that the > 1518b frames are eBGP, or something else that's relevant only to that link? The difference being that there would be no customer frames > 1518, just network stuff. It's 0.12% of total packets. -- Nathan Ward
On 22/09/2008, at 2:22 PM, Nathan Ward wrote:
I wonder, is it feasible that the > 1518b frames are eBGP, or something else that's relevant only to that link?
If you use large frames then you need to ensure either everyone is consistent or that you have the ability to specifically say which peer is what MTU, otherwise you'll find BGP doesn't work because usually the BGP frames are the same as the router's max segment size for the interface. (I've screwed this up on an occasion with a private peering session - it LOOKS like it should work, but doesn't).
The difference being that there would be no customer frames > 1518, just network stuff.
I guess this is the key - is anyone running >1518 to customers? If not, then the net effect isn't going to be huge.
It's 0.12% of total packets.
Is it really that high? A very quick look at our network shows it's closer to 0.0005% of packets on external links. MMC
-- Nathan Ward
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
-- Matthew Moyle-Croft Internode/Agile Peering and Core Networks Level 4, 150 Grenfell Street, Adelaide, SA 5000 Australia Email: mmc(a)internode.com.au Web: http://www.on.net Direct: +61-8-8228-2909 Mobile: +61-419-900-366 Reception: +61-8-8228-2999 Fax: +61-8-8235-6909
On Mon, Sep 22, 2008 at 04:52:49PM +1200, Nathan Ward said:
On 22/09/2008, at 4:40 PM, Simon Blake wrote:
I wonder, is it feasible that the > 1518b frames are eBGP, or something else that's relevant only to that link?
Most likely people doing some kind of EoIP/MPLS/EtherIP/GRE encapsulation to tunnel traffic from one router to another across the exchange fabric, or doing q-in-q in other VLAN's (this is a trunk port carrying both the exchange VLAN and various private VLAN's).
The difference being that there would be no customer frames > 1518, just network stuff.
More than likely. Tis a shame that the 3750 doesn't keep granulated counters for frames bigger than 1518. I could set up a monitor port from one of the GE links and get an accurate size histogram from the exchange VLAN - that might be interesting, particularly if it was graphed to see the change overtime. Cheers Simon
More than likely. Tis a shame that the 3750 doesn't keep granulated counters for frames bigger than 1518. I could set up a monitor port from one of the GE links and get an accurate size histogram from the exchange VLAN - that might be interesting, particularly if it was graphed to see the change overtime.
http://www.caida.org/research/traffic-analysis/pkt_size_distribution/graphs....
participants (6)
-
Alastair Johnson -
Brian E Carpenter -
Matthew Luckie -
Matthew Moyle-Croft -
Nathan Ward -
Simon Blake