I'm firmly in the PPPoE is the only way to run a fixed broadband network. You can change customers IP addresses easily by just kicking them and they come back in worst case 90 seconds (or 5 mins if it's a EUBA connection for the session to timeout). Can't do that on IPoE unless you have stupid low lease times. RemoteID and CircuitID are inserted in the PPPoE payload so you can auth them without requiring credentials. Much like you use option 82 on IPoE. No support for username and password so if your friendly LFC has screwed up the provisioning so you can't auth them on RID/CID you can't ask your customer to put in "heyfindme" in their username on the router and they magically appear in the radius logs and then tell the LFC to sort it. All LFCs support 1508 MTU so if your router and BNG does it then you can get a full 1500 sized packets which you can on Spark. Routed networks are easy you just allocate a subnet to sit behind the router. You don't need to waste a /31 per customer and thus 2 public IPs per subscriber or have some dirty flat routed networks across all your customers when each customer is a separate layer 2 network so you have a /22 or something and have a dedicated default route per subnet. PPPoE you only need a single real IP allocated to the interface and then only 2 IPs per subnet being broadcast and network rather than a routed IP too thus losing another ip per range. Using VLAN 10 means you can use .1q tagging from the CPE and have real working QoS over the link. If you don't need QoS then no need for VLAN 10. But once you've gone down the tagged route it's hard to move back as then you have a customer config nightmare. Find arguments against the above and I may listen. But until then I am completely sold on PPPoE vs IPoE. Sent from my phone so excuse the brevity. -------- Original Message -------- Subject: Re: [nznog] Dialup over Ethernet From: Don Stokes To: nznog(a)list.waikato.ac.nz CC: I'm firmly in the PPP Must Die camp. As far as I can make out, PPPoE is a thing because ISPs built PPP & Radius infrastructure for dial-up, used it to for ADSL (because that was the only way Telecom would let you do it), and continue to use the same back-end stuff to drive PPPoE for ADSL, VDSL, UFB and anything else they can get to speak it. It has the advantage that IP address management with PPPoE & friends is fairly straightforward, and you don't go wasting IP addresses, given that everything speaking native IP over Ethernet expects to see a subnet with a network, broadcast and gateway address in a /30 or bigger subnet, an issue I addressed herehttp://www.don.nz.net/wordpress/broadcast-interface-addressing-considered-ha.... This is important, because UFB is VLAN-per-circuit; you can't just lump a bunch of clients into a single IP subnet, a la Citylink in the old days. UFB has PPPoE brains that allows the PPPoE handshake to inform the ISP what service a PPPoE packet, independent of the VLAN tags, which in turn can be fed to a Radius server to identify a customer. (There's also DHCP brains to do the same for DHCP requests. Both are service-order-time options.) I don't believe anyone is actually using usernames/passwords to identify customers any more, but I won't be surprised if I'm wrong. It's not like dial-up where the customer could be coming from anywhere. Note that PPPoE in most configurations has an MTU of 1492, not 1500 (because there's an 8-byte PPPoE/PPP header between the MAC header and the IP payload). Which is another reason PPPoE needs to be killed. I'm not sure if 1500 MTU PPPoE (i.e. 1508+ MTU Ethernet) is supported by either ISPs or UFB. I believe UFB circuits are happy to take the larger frames, but since I've not tried to work with it, I can't say for sure. -- don On 09/13/2017 05:53 PM, Ewen McNeill wrote: A client had a (business) customer switch over to UFB and needed assistance reconfiguring the (Mikrotik) router being attached to the ONT -- it turned out whoever first set up the Mikrotik (sensibly) assumed it'd be seeing IP packets over VLAN 10, but actually the ISP required IP packets over PPPoE over VLAN 10 in order for it to work. Looking around it appears this client's customer's ISP isn't the only one that is requiring PPPoE over VLAN tagging over Ethernet for their UFB connections. Is there a reason other than "let's make everything look like 1990s dialup so it works with our legacy equipment" for the bit/CPE CPU overhead of PPPoE on UFB, including imposing the lowered usable MTU and PMTU discovery headaches on the end user? The only one that really comes to mind is "user/password authentication" (rather than needing to collect CPE MAC addresses which seems to happen with, eg, Vodafone cable/FibreX). But it's not clear to me why, eg, 802.1X isn't used for the user/password authentication in that case; or DHCP with some extension to pass a "secret" identifier. 20-bytes-per-packet-forever seems a large overhead to pay for user/password authentication at CPE power on.... (Maybe in the beginning there's "lack of CPE support" -- but we're a few years into the UFB rollout, and lots of ISPs seem to be supplying their own ISP-badged CPEs anyway, which could presumably implement whatever was needed.) Ewen _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nzmailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog -- Don Stokes, don(a)nz.netmailto:don(a)nz.net, 021 796 072 This communication, including any attachments, is confidential. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002.

Customer RGW / CPE is always the issue with MTUs of 1508 on the WAN side. There isn’t the issue on the wholesale customer side as Chorus support MTUs of 1520(?) from memory on all their copper access types even BUBA. I was involved with testing MTUs of 1508 on all LFCs as part of launching UFB so know that works across all LFCs as per the spec. So the constraint especially in the UFB world of Customers RGWs only supporting a ethernet MTU of 1500 and thus PPPoE MTU of 1492 is purely an issue with CPE the customer has chosen. For me the restore time in the event of loss of service to customers is the main reason why PPPoE is superior to DHCP/IPoE. When doing BNG reboots post upgrade I have seen ~%97+ UFB broadband services on a handover with over 5k customers on it restore in sub 30 seconds once the handover port is brought up. Voice ATA services on the same handover can take up to 30 mins to restore even though the DHCP lease is set down to 5 mins. As the DHCP Client has a longer backoff period the longer the DHCP server isn’t responding whereas the PPPoE clients hammer away while the session is down and come up as soon as the link is back up. Another advantage is to find out the customers current sync rate on copper as the vast majority of Chorus EUBA/WVS DLSAMs send you the connect rate information in the PADI you can kick them and they come back in sub 10 seconds and sometimes they have re-synced at a higher or lower rate. Means that you don’t need to go into Chorus tools like SPM to find out what the line is looking like. Haven’t checked to see if the Actual Data Rate comes in over DHCP Option 82 or not over EUBA/WVS but if you have a 5 min lease time you don’t exactly want to go kicking customers just to see what their connect rate is. Even in the UFB world I would always expect some sort of router / Customer RGW between the ONT and the customers devices. I wouldn’t expect a customer to plug in a switch into an ONT then have multiple devices sending DHCP requests to the BNG, perhaps in the full SDN/NFV world but I am still unconvinced you want all that ARP broadcast traffic flowing over your Layer 2 link even if it’s 500Mbit up. So if a routed network / more than a single IPv4 Address was required on the customer side you would bring up the link and tell the customer they also have this subnet range of addresses they can allocate on their side of the link. Since unconvinced to see the compelling reason case why IPoE/DHCP is any better than a PPPoE link, as I can only see downsides to IPoE. From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Hamish McGlinn Sent: Thursday, 14 September 2017 9:06 a.m. To: Brian Gibbons Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Dialup over Ethernet I'm of the same opinion as Peter on this. I'm firmly in the PPPoE camp. Generally speaking the only real case of opposition is the MTU at 1492. However in saying this I feel like it is a non issue with mss adjustment. If the software takes note of the change in mss then there shouldn't be a problem. UDP deployments as far as I have come across are only small packets so once again a non issue. As a service provider we very easily could increase the MTU to allow for 1500. The issue is that we need to do mss adjustment towards the customer for tcp applications they are hosting (web server, rdp, etc.). Not all CPE's support 1500 MTU PPPoE. This is the primary issue with this. If we increase to allow for it how do then determine which CPEs need the mss adjustment to 1452 and which don't? If all CPE's allowed for 1500 MTU PPPoE we would do this. Until that point it is what it is unfortunately. Cheers, Hamish On Thu, Sep 14, 2017 at 6:48 AM, Brian Gibbons mailto:brian(a)outersite.co.nz> wrote: On 13/09/2017 6:38 p.m., Peter Lambrechtsen wrote: I'm firmly in the PPPoE is the only way to run a fixed broadband network. Well said The Internet has done very well using Point to Point links from provider to CPE since the days of SLIP. DHCP is for wireless and keeps us in a job :) Cheers BG You can change customers IP addresses easily by just kicking them and they come back in worst case 90 seconds (or 5 mins if it's a EUBA connection for the session to timeout). Can't do that on IPoE unless you have stupid low lease times. RemoteID and CircuitID are inserted in the PPPoE payload so you can auth them without requiring credentials. Much like you use option 82 on IPoE. No support for username and password so if your friendly LFC has screwed up the provisioning so you can't auth them on RID/CID you can't ask your customer to put in "heyfindme" in their username on the router and they magically appear in the radius logs and then tell the LFC to sort it. All LFCs support 1508 MTU so if your router and BNG does it then you can get a full 1500 sized packets which you can on Spark. Routed networks are easy you just allocate a subnet to sit behind the router. You don't need to waste a /31 per customer and thus 2 public IPs per subscriber or have some dirty flat routed networks across all your customers when each customer is a separate layer 2 network so you have a /22 or something and have a dedicated default route per subnet. PPPoE you only need a single real IP allocated to the interface and then only 2 IPs per subnet being broadcast and network rather than a routed IP too thus losing another ip per range. Using VLAN 10 means you can use .1q tagging from the CPE and have real working QoS over the link. If you don't need QoS then no need for VLAN 10. But once you've gone down the tagged route it's hard to move back as then you have a customer config nightmare. Find arguments against the above and I may listen. But until then I am completely sold on PPPoE vs IPoE. Sent from my phone so excuse the brevity. -------- Original Message -------- Subject: Re: [nznog] Dialup over Ethernet From: Don Stokes To: nznog(a)list.waikato.ac.nzmailto:nznog(a)list.waikato.ac.nz CC: I'm firmly in the PPP Must Die camp. As far as I can make out, PPPoE is a thing because ISPs built PPP & Radius infrastructure for dial-up, used it to for ADSL (because that was the only way Telecom would let you do it), and continue to use the same back-end stuff to drive PPPoE for ADSL, VDSL, UFB and anything else they can get to speak it. It has the advantage that IP address management with PPPoE & friends is fairly straightforward, and you don't go wasting IP addresses, given that everything speaking native IP over Ethernet expects to see a subnet with a network, broadcast and gateway address in a /30 or bigger subnet, an issue I addressed herehttp://www.don.nz.net/wordpress/broadcast-interface-addressing-considered-ha.... This is important, because UFB is VLAN-per-circuit; you can't just lump a bunch of clients into a single IP subnet, a la Citylink in the old days. UFB has PPPoE brains that allows the PPPoE handshake to inform the ISP what service a PPPoE packet, independent of the VLAN tags, which in turn can be fed to a Radius server to identify a customer. (There's also DHCP brains to do the same for DHCP requests. Both are service-order-time options.) I don't believe anyone is actually using usernames/passwords to identify customers any more, but I won't be surprised if I'm wrong. It's not like dial-up where the customer could be coming from anywhere. Note that PPPoE in most configurations has an MTU of 1492, not 1500 (because there's an 8-byte PPPoE/PPP header between the MAC header and the IP payload). Which is another reason PPPoE needs to be killed. I'm not sure if 1500 MTU PPPoE (i.e. 1508+ MTU Ethernet) is supported by either ISPs or UFB. I believe UFB circuits are happy to take the larger frames, but since I've not tried to work with it, I can't say for sure. -- don On 09/13/2017 05:53 PM, Ewen McNeill wrote: A client had a (business) customer switch over to UFB and needed assistance reconfiguring the (Mikrotik) router being attached to the ONT -- it turned out whoever first set up the Mikrotik (sensibly) assumed it'd be seeing IP packets over VLAN 10, but actually the ISP required IP packets over PPPoE over VLAN 10 in order for it to work. Looking around it appears this client's customer's ISP isn't the only one that is requiring PPPoE over VLAN tagging over Ethernet for their UFB connections. Is there a reason other than "let's make everything look like 1990s dialup so it works with our legacy equipment" for the bit/CPE CPU overhead of PPPoE on UFB, including imposing the lowered usable MTU and PMTU discovery headaches on the end user? The only one that really comes to mind is "user/password authentication" (rather than needing to collect CPE MAC addresses which seems to happen with, eg, Vodafone cable/FibreX). But it's not clear to me why, eg, 802.1X isn't used for the user/password authentication in that case; or DHCP with some extension to pass a "secret" identifier. 20-bytes-per-packet-forever seems a large overhead to pay for user/password authentication at CPE power on.... (Maybe in the beginning there's "lack of CPE support" -- but we're a few years into the UFB rollout, and lots of ISPs seem to be supplying their own ISP-badged CPEs anyway, which could presumably implement whatever was needed.) Ewen _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nzmailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog -- Don Stokes, don(a)nz.netmailto:don(a)nz.net, 021 796 072 This communication, including any attachments, is confidential. If you are not the intended recipient, you should not read it - please contact me immediately, destroy it, and do not copy or use any part of this communication or disclose anything about it. Thank you. Please note that this communication does not designate an information system for the purposes of the Electronic Transactions Act 2002. ________________________________ This e-mail message has passed virus scanning by Outersite Technology. _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nzmailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog _____________________________________________________________________ This e-mail message has passed virus scanning by Outersite Technology. ________________________________ This e-mail message has passed virus scanning by Outersite Technology. _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nzmailto:NZNOG(a)list.waikato.ac.nz https://list.waikato.ac.nz/mailman/listinfo/nznog -- Cheers, Hamish

Chiming in here.. On Thu, Sep 14, 2017 at 9:05 AM, Hamish McGlinn wrote:

I'm of the same opinion as Peter on this. I'm firmly in the PPPoE camp.

Generally speaking the only real case of opposition is the MTU at 1492. However in saying this I feel like it is a non issue with mss adjustment. If the software takes note of the change in mss then there shouldn't be a problem. UDP deployments as far as I have come across are only small packets so once again a non issue.

Yup, everyone who doesn't like PPPoE seems to complain about the 1492 packet size. But almost all cheap CPEs tend to do tcp mss adjustment automatically. All the big name firewalls (Juniper, Cisco, Fortigate) have an option for it. Finally, as a last resort at our end we can do something funky : term tcp-helper { from { protocol tcp; tcp-flags syn; } then { forwarding-class best-effort; routing-instance vr_tcp_mss; } } (we just do that in one direction) And then send those packets through a TCP MSS adjuster. This works well for the rare case where the customer can't do it. Really, the 1492 packet size is just a non issue. Cheers Dave

On 09/14/17 10:16, Ewen McNeill wrote:

On 14/09/17 09:38, Dave Mill wrote:

...

And then send those packets through a TCP MSS adjuster. This works well for the rare case where the customer can't do it.

Really, the 1492 packet size is just a non issue.

I guess the Internet really is only TCP these days... and presumably soon only TCP/443 (maybe with a bit of legacy TCP/80).  The other 120-ish IPv4 protocols are unused.... or should that be unusable.

QUIC (transport over UDP) is growing in share of google's egress traffic http://conferences.sigcomm.org/sigcomm/2017/files/program/ts-5-1-QUIC.pdf

[Reordering quotes for readability] On 13/09/17 18:18, Don Stokes wrote:

I don't believe anyone is actually using usernames/passwords to identify customers any more, but I won't be surprised if I'm wrong. It's not like dial-up where the customer could be coming from anywhere.

Part of what prompted my surprise is that this Mikrotik very much did have a username and password on the PPPoE configuration, apparently given to my client (setting it up for their customer) by the ISP in question -- and the password looks "password like" rather than just a placeholder. It's good to hear from you (and Nathan) that at least that part is less common.

[PPPoE] has the advantage that IP address management with PPPoE & friends is fairly straightforward, and you don't go wasting IP addresses, given that everything speaking native IP over Ethernet expects to see a subnet with a network, broadcast and gateway address in a /30 [....] This is important, because UFB is VLAN-per-circuit;

Unless I'm missing something, wouldn't the PVLAN sort of approach work out here? Put each customer into a isolated PVLAN, and put the ISP-end gateway into a community PVLAN, then the customer VLANs can talk to the ISP but not to each other -- and you can use a larger subnet for efficiency (/24, /22, etc). Possibly this would require some fibre provider/ONT assistance... but the whole UFB thing was built from scratch so doing it didn't have to be layers upon layers of abstraction to make it look like the past.

Note that PPPoE in most configurations has an MTU of 1492, not 1500

This particular one seems to have the Mikrotik defaulting the PPPoE interface MTU to 1480. I didn't investigate exactly why, but it didn't seem to be specified in the PPPoE client config on the Mikrotik. (VLAN 10 MTU was 1500.) 1492 MTU seems to be the best case for PPPoE if the parent ethernet interface MTU is 1500; and configuring for mini-jumbo frames on the ONT/UFB side would seem to be an obvious work around, but it's not clear if that's commonly done either. I'd be a little less bitter about forcing PPPoE on end users if it didn't force their user-data-MTU down below the assumed-in-end-user-equipment 1500 byte MTU.... Ewen