Shocking iptables performance
We currently have a pair of Linux iptables firewall boxes that are being replaced in a month's time with a pair of Cisco ASA firewalls. Recently during testing we noticed transfer rates through the Ciscos (attached to the same Cisco routers on the other side) is over 30x faster than through the iptables firewalls. I know that iptables isn't hugely fast, there's a reason you pay for Ciscos etc etc, but the resources I have read usually indicate that iptables slowness is related to massive rulesets. We have 2415 rules, which I don't consider to be *that* many... The boxes aren't heavily loaded (about 10% system, 90% idle), and have plenty of free memory. They aren't swapping. I've tested turning off logging, which had virtually no effect. Anybody encountered anything similar? Is it likely to simply be related to the number of rules? Cheers Jasper -- _________________________________________ http://www.digiweb.co.nz Webhosting, Dedicated Servers, E-commerce Phone: (64 3) 351 6713 Fax: (64 3) 351 6705 Free: 0800 DIGIWEB (0800 344-493) Email: jasper(a)digiweb.co.nz PLEASE NOTE: The information contained in this email message and any attached files may be confidential and subject to privilege. The views expressed may not necessarily be the official view of Digiweb New Zealand Limited. All technical advice and opinions are offered on a 'no-liability' basis. If you are not the intended recipient, you are notified that any use, disclosure or copying of this email is unauthorised. If you have received this email in error, please notify us immediately by reply email and delete the original. Thank you.
Considering that the vast majority of non-Cisco commercial firewalls are based on some version of IP Tables, I don't think that is where your problem is. You haven't provided much information- 1. What is your hardware? 2. What sort of speed are you aiming for? 3. Which Linux kernel are you using?
We currently have a pair of Linux iptables firewall boxes that are being replaced in a month's time with a pair of Cisco ASA firewalls.
Recently during testing we noticed transfer rates through the Ciscos (attached to the same Cisco routers on the other side) is over 30x faster than through the iptables firewalls.
I know that iptables isn't hugely fast, there's a reason you pay for Ciscos etc etc, but the resources I have read usually indicate that iptables slowness is related to massive rulesets. We have 2415 rules, which I don't consider to be *that* many...
The boxes aren't heavily loaded (about 10% system, 90% idle), and have plenty of free memory. They aren't swapping. I've tested turning off logging, which had virtually no effect.
Anybody encountered anything similar? Is it likely to simply be related to the number of rules?
Cheers Jasper
Thanks to everyone for their fast replies. The firewalls are Pentium 4 2.8GHz boxes, only 256MB of RAM, but they're not swapping. We're getting about 256Kbit/s absolute maximum through these boxes, whereas on the Ciscos we can sustain 8-10Mbit/s no problem in some tests, and burst much higher. There are RELATED, ESTABLISHED rules around, but one important thing I seem to have neglected to mention is that these are managed by Shorewall, so I have only indirect control over the rules. The connection state tracking rules aren't quite as close to the start of the tables as I would like, but I'm not sure if I can control this easily with Shorewall... Jasper (looking forward to a beer after all this) John @ netTRUST wrote:
Considering that the vast majority of non-Cisco commercial firewalls are based on some version of IP Tables, I don't think that is where your problem is.
You haven't provided much information- 1. What is your hardware? 2. What sort of speed are you aiming for? 3. Which Linux kernel are you using?
We currently have a pair of Linux iptables firewall boxes that are being replaced in a month's time with a pair of Cisco ASA firewalls.
Recently during testing we noticed transfer rates through the Ciscos (attached to the same Cisco routers on the other side) is over 30x faster than through the iptables firewalls.
I know that iptables isn't hugely fast, there's a reason you pay for Ciscos etc etc, but the resources I have read usually indicate that iptables slowness is related to massive rulesets. We have 2415 rules, which I don't consider to be *that* many...
The boxes aren't heavily loaded (about 10% system, 90% idle), and have plenty of free memory. They aren't swapping. I've tested turning off logging, which had virtually no effect.
Anybody encountered anything similar? Is it likely to simply be related to the number of rules?
Cheers Jasper
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
-- Jasper Bryant-Greene Director Album Limited jasper(a)albumltd.co.nz +64 21 708 334 / 0800 425 286 http://www.albumltd.co.nz/
This sort of thing keeps the 'pretty box' manufacturers in business. When you factor in this, and the cost of the ongoing licensing / updates for the 'pretty boxes' plus the use of money value, isn't your time better spent learning how to use the free alternatives? In this paticular case there is nothing wrong with the free options. All the pretty box manufacturer is doing is sticking it in a (often colourful) rack mount case and putting a nice front end GUI to it. 9.5+ times out of 10 it's exactly the same core software.
The firewalls are Pentium 4 2.8GHz boxes, only 256MB of RAM, but they're not swapping.
We're getting about 256Kbit/s absolute maximum through these boxes, whereas on the Ciscos we can sustain 8-10Mbit/s no problem in some tests, and burst much higher.
There are RELATED, ESTABLISHED rules around, but one important thing I seem to have neglected to mention is that these are managed by Shorewall, so I have only indirect control over the rules.
The connection state tracking rules aren't quite as close to the start of the tables as I would like, but I'm not sure if I can control this easily with Shorewall...
Jasper (looking forward to a beer after all this)
John @ netTRUST wrote:
Considering that the vast majority of non-Cisco commercial firewalls are based on some version of IP Tables, I don't think that is where your problem is.
You haven't provided much information- 1. What is your hardware? 2. What sort of speed are you aiming for? 3. Which Linux kernel are you using?
We currently have a pair of Linux iptables firewall boxes that are being replaced in a month's time with a pair of Cisco ASA firewalls.
Recently during testing we noticed transfer rates through the Ciscos (attached to the same Cisco routers on the other side) is over 30x faster than through the iptables firewalls.
I know that iptables isn't hugely fast, there's a reason you pay for Ciscos etc etc, but the resources I have read usually indicate that iptables slowness is related to massive rulesets. We have 2415 rules, which I don't consider to be *that* many...
The boxes aren't heavily loaded (about 10% system, 90% idle), and have plenty of free memory. They aren't swapping. I've tested turning off logging, which had virtually no effect.
Anybody encountered anything similar? Is it likely to simply be related to the number of rules?
Cheers Jasper
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
We're getting about 256Kbit/s absolute maximum through these boxes, whereas on the Ciscos we can sustain 8-10Mbit/s no problem in some tests, and burst much higher.
We've had multiple Shorewall setups and could get 8-10Mbit/s (link was
10Mbits) on a K6-500 with ~1000 rules.
We get a 100Mbits on a Sempron 2400 with ~1000 rules.
(Dual 100Mbit ethernet)
As your cpu usage is rather low, you could be getting packet loss.
We had an issue in the past with a upstream ethernet interface set at
10Mbit/Full duplex, but as the firewall could not negotiate, it defaulted to
10Mbit/Half duplex. We got about 5% packet loss in that scenario.
--
------------------------------------------------------------------------
Jean-Francois Pirus
On 19/01/07, Jean-Francois Pirus
We're getting about 256Kbit/s absolute maximum through these boxes, whereas on the Ciscos we can sustain 8-10Mbit/s no problem in some tests, and burst much higher. [snippy] As your cpu usage is rather low, you could be getting packet loss.
We had an issue in the past with a upstream ethernet interface set at 10Mbit/Full duplex, but as the firewall could not negotiate, it defaulted to 10Mbit/Half duplex. We got about 5% packet loss in that scenario.
Eergh - I had only been skim reading... 256Kbit max through these boxes?? Something's VERY wrong... JFP's post above made the think to explicitly state that it's worth FIXING eth speed and duplex, not just checking to see what it thinks it's negotiated... Change both ends and fix the ports to the best they will support (Or in this case, anything, since 256Kbit is awful)... Apart from that, if you have the luxury, I'd swap in a different PC with different brand NICs with a ghosted copy of the setup just to convince yourself it's not a peculiarity of the setups you have. Finally, make sure you don't have anything funky going on with the hard drives... I recall a while ago (yes, ok, it was on windows) a HDD dropped into PIO mode and utterly crippled the entire machine... Good luck. Cheers - N
Finally, make sure you don't have anything funky going on with the hard drives... I recall a while ago (yes, ok, it was on windows) a HDD dropped into PIO mode and utterly crippled the entire machine...
That brings back bad memories. :-)
Another issue we've had is a silently failing hard disk. It would just lock
the firewall for seconds at a time with nothing out of the ordinary showing
anywhere. Nothing in the logs, normal top, ps etc. It would just become
unresponsive then come back.
So a Neil's ghost to another box is a good idea, it'll rule out flacky hardware.
--
------------------------------------------------------------------------
Jean-Francois Pirus
Jean-Francois Pirus wrote:
Finally, make sure you don't have anything funky going on with the hard drives... I recall a while ago (yes, ok, it was on windows) a HDD dropped into PIO mode and utterly crippled the entire machine...
That brings back bad memories. :-)
Another issue we've had is a silently failing hard disk. It would just lock the firewall for seconds at a time with nothing out of the ordinary showing anywhere. Nothing in the logs, normal top, ps etc. It would just become unresponsive then come back.
So a Neil's ghost to another box is a good idea, it'll rule out flacky hardware.
[root(a)firewall1 root]# mii-tool eth0: negotiated 100baseTx-FD flow-control, link ok eth1: negotiated 100baseTx-FD flow-control, link ok Looks ok to me. Fixing it with mii-tool -F 100baseTx-FD makes no difference to transfer rates. I will investigate a ghost to another box, could be a possibility. Jasper -- _________________________________________ http://www.digiweb.co.nz Webhosting, Dedicated Servers, E-commerce Phone: (64 3) 351 6713 Fax: (64 3) 351 6705 Free: 0800 DIGIWEB (0800 344-493) Email: jasper(a)digiweb.co.nz PLEASE NOTE: The information contained in this email message and any attached files may be confidential and subject to privilege. The views expressed may not necessarily be the official view of Digiweb New Zealand Limited. All technical advice and opinions are offered on a 'no-liability' basis. If you are not the intended recipient, you are notified that any use, disclosure or copying of this email is unauthorised. If you have received this email in error, please notify us immediately by reply email and delete the original. Thank you.
On Fri, 19 Jan 2007 at 13:42:43 +1300, Jasper Bryant-Greene wrote:
Thanks to everyone for their fast replies.
The firewalls are Pentium 4 2.8GHz boxes, only 256MB of RAM, but they're not swapping.
We're getting about 256Kbit/s absolute maximum through these boxes, whereas on the Ciscos we can sustain 8-10Mbit/s no problem in some tests, and burst much higher.
We can push ~100Mbit at around ~40000 simultaneous connections through our 1.7GHz P4 Celeron beige boxes and top out at about 70% CPU while running a libpcap-based traffic accounting application on the machines, so something's not right with your Shorewall. This is using the standard Centos 4.4 2.6 series kernel with a few patches for ratelimiting and TCP MD5. Our firewall rules are done by hand in a few init scripts. This won't suit everyone but it does allow us to fine tune things quite well. Nigel
John @ netTRUST wrote:
Considering that the vast majority of non-Cisco commercial firewalls are based on some version of IP Tables Careful you'll anger the BSD folk. =)
I'd say that you should also be careful as more and more non-Cisco commercial firewalls are not based on IP Tables at all (Netscreen, Fortinet etc) Dean
Actually strike Fortinet from that list - I was having a senior moment =) I was making the distinction between software and asic assist. Dean Dean Pemberton wrote:
John @ netTRUST wrote:
Considering that the vast majority of non-Cisco commercial firewalls are based on some version of IP Tables
Careful you'll anger the BSD folk. =)
I'd say that you should also be careful as more and more non-Cisco commercial firewalls are not based on IP Tables at all (Netscreen, Fortinet etc)
Dean
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On Fri, 19 Jan 2007 at 12:31:27 +1300, Jasper Bryant-Greene wrote:
Anybody encountered anything similar? Is it likely to simply be related to the number of rules?
I reduced the CPU usage on our iptables based firewalls by a large proportion by making sure there were rules allowing ESTABLISHED and RELATED connections through relatively early in the rule set for each table. This is on a set of some 2000 rules. If every packet has to traverse a large rule set you will see reduced performance. Regards, Nigel
Nigel Roberts wrote:
On Fri, 19 Jan 2007 at 12:31:27 +1300, Jasper Bryant-Greene wrote:
Anybody encountered anything similar? Is it likely to simply be related to the number of rules?
I reduced the CPU usage on our iptables based firewalls by a large proportion by making sure there were rules allowing ESTABLISHED and RELATED connections through relatively early in the rule set for each table.
You can also use additional chains to reduce the number of rules. You could for instance split your INPUT chain into 4 chains based on the source IP (0.0.0.0/2, 64.0.0.0/2, 128.0.0.0/2, 192.0.0.0/2) and reduce the number of rules each packet transverses by 1/4th[1]. Or use any other obvious metric to exclude having to check rules (eg in-interface). And yeah, as you mentioned put the usual cases first. (iptables -nvL will list how many times each rule is being hint, ideally you'd want these numbers to decrease down a chain). ---- [1]: Assuming that your rules are split evenly across the IP address space, which is fairly unlikely to be true.
On Fri, Jan 19, 2007 at 12:31:27PM +1300, Jasper Bryant-Greene said:
I know that iptables isn't hugely fast, there's a reason you pay for Ciscos etc etc, but the resources I have read usually indicate that iptables slowness is related to massive rulesets. We have 2415 rules, which I don't consider to be *that* many...
It's getting up there. Depending on the CPU of the box, that could severely overtax a firewall. I've a VIA C3 600Mhz box with 2550 rules that can pass 10.8MByte/s through it for sustained periods (minutes) - I suspect that's being capped by 100Mb interfaces and other traffic in/out of the boxes.
Anybody encountered anything similar? Is it likely to simply be related to the number of rules?
I've a couple of boxes that have over 4000 rules, and are suffering - if I take them down to under 2000 rules, they perform vastly better. The actual cutoff point seems to be primarily CPU performance related - slower boxes will fail at 1000 rules, fast boxes might cope with 5000. I'd try reordering the rules to get the most often hit rules at the top of the list, turning on NAPI if you haven't already, or moving to nf-hipac (if you don't need NAT/MANGLE). The PDF's at http://www.hipac.org/documentation/presentations.html are interesting - if I recall correctly one of them has a graph of throughput versus no of iptables rules which said that on their particular hardware with up to 1000 rules they got 100Mb/s, above that throughput started to tail off and at 5000 rules they got essentially 0Mb/s. Performance to the hipac website blows at the moment, which might not be the best advertisement for the software :-). Cheers Si
Simon Blake wrote:
The PDF's at http://www.hipac.org/documentation/presentations.html are interesting - if I recall correctly one of them has a graph of throughput versus no of iptables rules which said that on their particular hardware with up to 1000 rules they got 100Mb/s, above that throughput started to tail off and at 5000 rules they got essentially 0Mb/s.
Here http://www.hipac.org/performance_tests/results.html
Performance to the hipac website blows at the moment, which might not be the best advertisement for the software :-).
If you can get to it Richard
Cheers Si
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
The machines aren't connected to a Telstra-supplied media converter are they? Telstra probably recently changed something, as we ran into significant problems of a similar nature yesterday, and had to fiddle with the duplexing... Erin Salmon Managing Director Unleash Technology Solutions Phone: +64 3 365 1273 Mobile: +64 275 877 913 -----Original Message----- From: Jasper Bryant-Greene [mailto:jasper(a)digiweb.net.nz] Sent: Friday, 19 January 2007 12:31 p.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] Shocking iptables performance We currently have a pair of Linux iptables firewall boxes that are being replaced in a month's time with a pair of Cisco ASA firewalls. Recently during testing we noticed transfer rates through the Ciscos (attached to the same Cisco routers on the other side) is over 30x faster than through the iptables firewalls. I know that iptables isn't hugely fast, there's a reason you pay for Ciscos etc etc, but the resources I have read usually indicate that iptables slowness is related to massive rulesets. We have 2415 rules, which I don't consider to be *that* many... The boxes aren't heavily loaded (about 10% system, 90% idle), and have plenty of free memory. They aren't swapping. I've tested turning off logging, which had virtually no effect. Anybody encountered anything similar? Is it likely to simply be related to the number of rules? Cheers Jasper -- _________________________________________ http://www.digiweb.co.nz Webhosting, Dedicated Servers, E-commerce Phone: (64 3) 351 6713 Fax: (64 3) 351 6705 Free: 0800 DIGIWEB (0800 344-493) Email: jasper(a)digiweb.co.nz PLEASE NOTE: The information contained in this email message and any attached files may be confidential and subject to privilege. The views expressed may not necessarily be the official view of Digiweb New Zealand Limited. All technical advice and opinions are offered on a 'no-liability' basis. If you are not the intended recipient, you are notified that any use, disclosure or copying of this email is unauthorised. If you have received this email in error, please notify us immediately by reply email and delete the original. Thank you. _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog -- No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.432 / Virus Database: 268.16.13/634 - Release Date: 17/01/2007 4:45 p.m. -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.432 / Virus Database: 268.16.13/634 - Release Date: 17/01/2007 4:45 p.m.
participants (12)
-
Dean Pemberton -
Erin Salmon - Unleash -
Jasper Bryant-Greene -
Jasper Bryant-Greene -
Jean-Francois Pirus -
Jeremy Brooking -
John ＠ netTRUST -
Neil Gardner -
Nigel Roberts -
Perry Lorier -
Richard Nelson -
Simon Blake