I was wondering what people think about this latest story.. http://www.idg.net.nz/webhome.nsf/UNID/4AA2988B4A1835C5CC256BFF0014A6A8!open... (a more technical expanation from Cisco of the problem http://www.cisco.com/warp/public/105/56.html) I've noticed this problem for ages (for example the ASB's site) when viewing their pages via a GRE tunnel (or the inability to). Is blocking _all_ ICMP types the wrong thing to do? (in paticular type 3 (unreacable), subtype 4(needs fragmentation) for PMTU Discovery) and basiclly breaking their website for people who have paths who get fragmented TCP/IP Packets) Thanks Craig Whitmore
The banks (specifically the ASB and National Bank) complete lack of regard for ensuring a consistent delivery of IP packets to end users annoys me greatly. Imagine if every IP provider took this same lack of regard. I take particular issue with the statement that they have only had two complaints in 4 years. I've personally made more complaints on behalf of customers than this. They simply choose not to listen. Below is a list of ICMP packets that I think everyone should at a minimum allow through. access-list 101 permit icmp any any unreachable access-list 101 permit icmp any any packet-too-big access-list 101 permit icmp any any time-exceeded I also think these types have some merit. access-list 101 permit icmp any any echo-reply access-list 101 permit icmp any any traceroute access-list 101 permit icmp any any administratively-prohibited -----Original Message----- From: owner-nznog(a)list.waikato.ac.nz [mailto:owner-nznog(a)list.waikato.ac.nz] On Behalf Of Craig Whitmore Sent: Wednesday, 24 July 2002 9:37 p.m. To: nznog(a)list.waikato.ac.nz Subject: Banking Problems and MTU I was wondering what people think about this latest story.. http://www.idg.net.nz/webhome.nsf/UNID/4AA2988B4A1835C5CC256BFF0014A6A8! opendocument (a more technical expanation from Cisco of the problem http://www.cisco.com/warp/public/105/56.html) I've noticed this problem for ages (for example the ASB's site) when viewing their pages via a GRE tunnel (or the inability to). Is blocking _all_ ICMP types the wrong thing to do? (in paticular type 3 (unreacable), subtype 4(needs fragmentation) for PMTU Discovery) and basiclly breaking their website for people who have paths who get fragmented TCP/IP Packets) Thanks Craig Whitmore - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
On Wed, Jul 24, 2002 at 09:37:07PM +1200, Craig Whitmore wrote:
Is blocking _all_ ICMP types the wrong thing to do?
Yes. Unofrtunately, it's an argument I've lost with clients in the past, who have self-styled network security experts who believe that if some filtering is good, more must be better! -- Rodger Donaldson rodgerd(a)diaspora.gen.nz vi - virtually incomprehensible -- Paul Thomas - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
Is blocking _all_ ICMP types the wrong thing to do?
Yes. Unofrtunately, it's an argument I've lost with clients in the past, who have self-styled network security experts who believe that if some filtering is good, more must be better!
I've found lots of places which say blocking the icmp stuff for PTMU stuff is wrong (causing this issue). Where did the people who do block it get the idea from to actually do this and "break things" for their clients. Maybe they should be a warning up on their web page saying "People who have Fragmented TCP/IP packets will not be able to access this site properly because we are too lazy to fix our firewalls" (well its what it sounds like on the news article) Thanks Craig Talking for Myself - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
On Wed, 24 Jul 2002, Craig Whitmore wrote:
I've found lots of places which say blocking the icmp stuff for PTMU stuff is wrong (causing this issue). Where did the people who do block it get the idea from to actually do this and "break things" for their clients. Maybe they should be a warning up on their web page saying "People who have Fragmented TCP/IP packets will not be able to access this site properly because we are too lazy to fix our firewalls" (well its what it sounds like on the news article)
Firewall issues apart, I believe the problem is that the banks in question have networks with smaller than 1,500 byte MTUs, but not advertising the fact, so it's not a question of "people having fragmented packets" as such. As for the reason, well... "ICMP? Wot dat crap? We don't need no steenken ICMP here." ;-) It's just too complicated for mere mortals, this whole TCP/IP thing. There should be a simpler variant, like, errr... My/IP or something, that doesn't have all those nasty hairy and dangling techie bits what causes all the bother eh? -- Juha Saarinen - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
On Wed, Jul 24, 2002 at 11:03:37PM +1200, Juha Saarinen wrote:
On Wed, 24 Jul 2002, Craig Whitmore wrote:
I've found lots of places which say blocking the icmp stuff for PTMU stuff is wrong (causing this issue). Where did the people who do block it get the idea from to actually do this and "break things" for their clients. Maybe they should be a warning up on their web page saying "People who have Fragmented TCP/IP packets will not be able to access this site properly because we are too lazy to fix our firewalls" (well its what it sounds like on the news article)
Firewall issues apart, I believe the problem is that the banks in question have networks with smaller than 1,500 byte MTUs, but not advertising the fact, so it's not a question of "people having fragmented packets" as such.
No. The problem is that there is a sub-1500-byte MTU interface on a router somewhere between the bank's firewall and the client, and the bank's servers are not being informed of this fact because bank firewalls are dropping the "would fragment" messages. You'd hope that banks would employ people who actually knew something about security, but apparently not. - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
From: "Joe Abley"
No. The problem is that there is a sub-1500-byte MTU interface on a router somewhere between the bank's firewall and the client, and the bank's servers are not being informed of this fact because bank firewalls are dropping the "would fragment" messages.
This issue was discussed a year or two ago on the ADSL mailing list. Users of 3com Home and Alcatel Speed Touch Home ADSL modems require a tunnel (PPPoE and PPTP respectively) from user to modem thus dropping the MTU to below 1500. If the bank is going to block ICMP then they should drop the MTU on the Web Server's Internet interface to something like 1450, this should clamp the Server's TCP MSS to a lower value. Not a perfect solution but .... Having said that, the "I can't get a bank statement" issue currently being discussed on the ADSL list is not the above fault, it is affecting users with full MTU on Jetstart. It is well known that many implementations of HTTPS/SSL do not handle packet loss very well. Telecom use an overdraft in their shaping of Jetstart which causes TCP to start winding up to a full 8megs until the overdraft is hit, suddenly TCP is trying to drive 8Mb/s into a 128k pipe. There are truckloads of packets dropped on the floor before TCP sorts out this magnitude of "congestion", some SSL implementations never sort it out. So it works fine until the user tries a larger download (bank statement), all they get is the evil hourglass. Cheers BG. - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
On Wednesday, July 24, 2002, at 01:38 , Brian Gibbons wrote:
If the bank is going to block ICMP then they should drop the MTU on the Web Server's Internet interface to something like 1450, this should clamp the Server's TCP MSS to a lower value. Not a perfect solution but ....
If the bank is going to block ICMP, then they should take care to do it by sub-code so that they don't break PMTUD. Either that, or they should turn off PMTUD on their servers.
Having said that, the "I can't get a bank statement" issue currently being discussed on the ADSL list is not the above fault, it is affecting users with full MTU on Jetstart.
How sure are you that the *path* between users and the bank firewalls is contains no router interfaces with sub-1500 byte MTUs? Have you verified that 1500-byte datagrams with DF set are consistently carried successfully between Jetstart subscribers and other destinations?
It is well known that many implementations of HTTPS/SSL do not handle packet loss very well. Telecom use an overdraft in their shaping of Jetstart which causes TCP to start winding up to a full 8megs until the overdraft is hit, suddenly TCP is trying to drive 8Mb/s into a 128k pipe. There are truckloads of packets dropped on the floor before TCP sorts out this magnitude of "congestion", some SSL implementations never sort it out.
So it works fine until the user tries a larger download (bank statement), all they get is the evil hourglass.
Overdraft? What peculiar terminology. The worst thing you'd expect in a path throughput constriction from 8M to 128k is surely the loss of a window's worth of data. You'd expect to TCP to wind back to slow start, and you'd expect the in-flight pipeline to drain very rapidly. Performance would certainly be hit, but I don't think I would expect consistently stalled connections. The visible symptoms (large download gives "evil hourglass") sounds much more like a TCP session which is sufficiently long-lived for large segments to be sent by the bank's server, leading the session to trip over a crippled use of PMTUD by the server. Turning off HTTP/1.1 pipelining can sometimes help make the symptoms go away. You see similar symptoms with mail ("small messages get through, but messages longer than a few paragraphs remain in the queue and don't get delivered"). Joe - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
On Wed, 24 Jul 2002, Joe Abley wrote:
Overdraft? What peculiar terminology.
Not for banks it ain't... ;-)
The worst thing you'd expect in a path throughput constriction from 8M to 128k is surely the loss of a window's worth of data. You'd expect to TCP to wind back to slow start, and you'd expect the in-flight pipeline to drain very rapidly. Performance would certainly be hit, but I don't think I would expect consistently stalled connections.
Dunno if it's relevant, but Jetstart is rate-limited at the RAN, which can be anywhere in NZ (my one is in Glenfield, some 25-30km away, but used to be in Wellington). To and from the DSLAM, Jetstart users have a full RADSL connection (which incidentally is also rate-limited, to 4Mbps down). Would this sort of rate-limiting affect the problem? -- Juha Saarinen - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
On Wed, 24 Jul 2002, Craig Whitmore wrote:
Is blocking _all_ ICMP types the wrong thing to do? (in paticular type 3 (unreacable), subtype 4(needs fragmentation) for PMTU Discovery) and basiclly breaking their website for people who have paths who get fragmented TCP/IP Packets)
The answer is of course in the results of the blocking, and if you read the story again, you'll see that the ASB security specialist says: "It says 'this packet is too big' so it can't send it on but it's not allowed to fragment it either, so it says 'I'll generate an ICMP message'." However, the ICMP message doesn't get sent to the banks because the banks filter ICMP and the message eventually times out. "It's one of those things where our standard policy is if we don't need it and it's no great issue, lets leave it off," says Bilby. Bilby says the issue usually only occurs with larger packets, and that typically happens after the user has logged on and is requesting a larger file, like an account statement. "Often a user can log on to a website that is blocking all ICMP but can't retrieve larger web pages such as a large balance screen or statement listings." I guess it depends on how useful you want to make your Web site. If usefulness isn't a priority, block all ICMP. To further lessen usefulness block TCP as well. ;-) -- Juha Saarinen - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
At 21:37 24/07/02 +1200, Craig Whitmore wrote:
I was wondering what people think about this latest story..
http://www.idg.net.nz/webhome.nsf/UNID/4AA2988B4A1835C5CC256BFF0014A6A8!ope ndocument
(a more technical expanation from Cisco of the problem http://www.cisco.com/warp/public/105/56.html)
I've noticed this problem for ages (for example the ASB's site) when viewing their pages via a GRE tunnel (or the inability to).
Is blocking _all_ ICMP types the wrong thing to do? (in paticular type 3 (unreacable), subtype 4(needs fragmentation) for PMTU Discovery) and basiclly breaking their website for people who have paths who get fragmented TCP/IP Packets)
Thanks Craig Whitmore
I don't know about other people, but the level of ignorance shown by the banks "security specialists" astounds me. "We try to keep entry rules [to the network] as tight as possible to what we specifically want in there on the basis that anything else could be bad. We don't need to support ICMP traffic, therefore it is excluded." Don't need ICMP eh ? Perhaps they havn't read RFC 792: "Occasionally a gateway or destination host will communicate with a source host, for example, to report an error in datagram processing. For such purposes this protocol, the Internet Control Message Protocol (ICMP), is used. ICMP, uses the basic support of IP as if it were a higher level protocol, however, ICMP is actually an integral part of IP, and must be implemented by every IP module." Note the word MUST. Certainly, there are some kinds of ICMP that could/should be blocked for some applications, but blocking all ICMP and therefore breaking PMTU Discovery is just plain ignorant and stupid, especially when its so incredibly easy to avoid, with a single ACL in their firewall/router. (Allow ICMP type 3 code 4) "Woolett says WestpacTrust hasn't heard of any problems along these lines but also says if users are capable of tweaking MTU settings they're probably fixing their own problems." Bollocks. First of all, any customers having that problem contacting their bank about it would likely not encounter any frontline helpdesk staff that would have any clue that the problem they're having is related to PMTU Discovery problems, or even know what PMTU was. They'd probably go through all the standard "Have you rebooted Windows?", "Have you installed the Latest version of Internet Explorer?" stuff, and then conclude that there was some unknown problem with the customers computer and that it needed reinstalling. The so called "security specialists" would probably never hear about 90% of the customers having this problem. The second thing thats bollocks about that statment is the implication that there is a "problem" at the customers end that needs tweaking. Going through a GRE tunnel, or anything else that forces you to use a lower MTU is not a "problem", it is just a situation which the IP protocol is designed to handle as a matter of course. The problem lies at the bank where they are breaking the IP protocol, however much they try to evade the issue. "If someone's done some really crazy tweaking at their end then it could potentially cause an issue." Going through a GRE tunnel is "really crazy tweaking" now is it ? Oh yes of course... I forgot. Anyone that doesn't use the internet for only http and pop3 over a dialup connection is "crazy"...... Hopefully with a bit of bad publicity the self styled "security specialists" might get a kick in the bum to go out and actually read up on ICMP a bit.... </rant> :) Regards, Simon - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
On Thu, 25 Jul 2002, Simon Byrnand wrote:
At 21:37 24/07/02 +1200, Craig Whitmore wrote:
http://www.idg.net.nz/webhome.nsf/UNID/4AA2988B4A1835C5CC256BFF0014A6A8!ope ndocument
I don't know about other people, but the level of ignorance shown by the banks "security specialists" astounds me.
What surprises me is that it hasn't been widely mentioned before; it's a widespread problem in the banking industry, and has been for ages. A few years ago, I was interviewing people for a network/security-type role, and made a point of asking which ICMP types, if any, should be blocked. It sure helped the weeding-out process - and told me which banks to avoid ;) Cheers R -- Richard Stevenson - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
On Thu, 2002-07-25 at 11:40, Richard Stevenson wrote:
On Thu, 25 Jul 2002, Simon Byrnand wrote:
At 21:37 24/07/02 +1200, Craig Whitmore wrote:
http://www.idg.net.nz/webhome.nsf/UNID/4AA2988B4A1835C5CC256BFF0014A6A8!ope ndocument
I don't know about other people, but the level of ignorance shown by the banks "security specialists" astounds me.
What surprises me is that it hasn't been widely mentioned before; it's a widespread problem in the banking industry, and has been for ages.
Because the majority of people probably call their ISPs helpdesk, who try it from their 1500 byte MTU ethernet connection, and say "works fine for me, try such and such.". The banks only get reports from "power users" who "tweak stuff", because they're the only people who realise the correct place to complain is the bank, not their ISP, who can't really do anything about it. How about getting a bunch of facts together, talking to IDG, and seeing if they can't be shamed into fixing it? <soundbyte> "Their ignorance of the path MTU discovery issue does not inspire much confidence in their overall security abilities." </soundbyte> It's not like they can't afford decent firewalls that can look at an ICMP message and only allow it if it relates to a known connection. Hell, the Linux ip_conntrack stuff is perfectly capable of that. Regards, Nic. P.S. Sorry to Richard Stevenson for the duplicate; I'm overly-used to lists with reply-to. - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
On Wed, Jul 24, 2002 at 09:37:07PM +1200, Craig Whitmore wrote: I was wondering what people think about this latest story.. http://www.idg.net.nz/webhome.nsf/UNID/4AA2988B4A1835C5CC256BFF0014A6A8!open... When working with these people some time ago (two years ago perhaps?) I mentioned this to the beanie-wearers there as it was affecting *my* traffic. I spent about 10 minutes (maybe more) explaining the problem in detail (the person I spoke to was quite inquisitive and asked lots of useful questions) for them only to then say that blocking ICMP is the right thing to do and they know better. In short, these people *know* about the issue or should know, it's just an abundance of arrogance an ineptitude that gives us the present situation. I should also point out that I have the same conversation with various other companies and got similar responses, I probably have some of that archived away somewhere too. --cw [1] I didn't talk to the helpdesk and anyone like that. I managed at the time to talk to administrators of the firewall(s) who answered various questions of mine which gave me the distinct impression that there was a severe clue money imbalance at the time. - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
Craig Whitmore wrote:
I was wondering what people think about this latest story..
http://www.idg.net.nz/webhome.nsf/UNID/4AA2988B4A1835C5CC256BFF0014A6A8!open...
(a more technical expanation from Cisco of the problem http://www.cisco.com/warp/public/105/56.html)
How incredibly serendipitous. I just ran into this problem at work the yesterday. The service techs. for our digital radiology system came in and applied the latest service update. All of a sudden users weren't able to log in via their browser loaded Java client. Funny thing is, these guys spent 2 days trying to figure out the problem before resorting to asking "the network guy"... one of those pride things I guess. Some quality time (15 minutes) with tcpdump (well, windump) showed the DF bit being set and repeated acknowledgment requests during the "hang". After a modest amount of investigation and a few phone calls the following was found: 1. a remote firewall blocking all ICMP (the admin won't change it) 2. a router somewhere in the path with a lower MTU requirement 3. a software upgrade forcing the DF bit to be set (developer won't change it) Lowering the MTU on the client end fixed the problem; however, I need to get the registry hack distributed across ~500 desktop and all potentially have different interface descriptions in the HKLM/system/current control set/service/ ...what a PITA. Some additional URLs: http://www.spirit.com/Network/net0700.html http://www.cymru.com/Documents/icmp-messages.html http://www.freelabs.com/~whitis/isp_mistakes.html - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
At 12:34 27/07/02 +0900, cfb wrote:
How incredibly serendipitous. I just ran into this problem at work the yesterday. The service techs. for our digital radiology system came in and applied the latest service update. All of a sudden users weren't able to log in via their browser loaded Java client. Funny thing is, these guys spent 2 days trying to figure out the problem before resorting to asking "the network guy"... one of those pride things I guess.
[.........]
1. a remote firewall blocking all ICMP (the admin won't change it)
Perhaps some education with a clue by four ? :)
2. a router somewhere in the path with a lower MTU requirement
[.......] 3. a software upgrade forcing the DF bit to be set (developer won't change it)
They shouldn't have to either. Most OS's these days default to path MTU discovery on, which means always setting the DF bit. Regards, Simon - To unsubscribe from nznog, send email to majordomo(a)list.waikato.ac.nz where the body of your message reads: unsubscribe nznog
participants (11)
-
Brian Gibbons -
cfb -
Chris Wedgwood -
Craig Whitmore -
Joe Abley -
Juha Saarinen -
Nic Bellamy -
Philip D'Ath -
Richard Stevenson -
Rodger Donaldson -
Simon Byrnand