Hi all, I've been working on a paper describing some measurements we did to determine how many TCP and UDP sessions residential broadband users consume for the purpose of evaluating the likely impact of Service Provider (or Carrier Grade) NAT. After submitting the paper to a journal I received an interesting comment from a reviewer: "It is not uncommon to see transparent but stateful firewalls in ISPs (without NATs) today - to avoid DoS attacks. These firewalls do a job similar to SPNATs. What is the state maintenance and processing overhead in these firewall deployments? Can we reuse any lessons from them?" The questions I have: Is that initial statement correct? Is there anyone out there who is using (or knows anyone who is using) a stateful firewall in such a fashion? Any responses off-list would be more than welcome. Thanks, Shane Alcock WAND
On Feb 25, 2010, at 10:09 AM, Shane Alcock wrote:
The questions I have: Is that initial statement correct? Is there anyone out there who is using (or knows anyone who is using) a stateful firewall in such a fashion?
It's utter nonsense, of course, as you rightly suspect.
;>
No SP in his right mind does this; you mainly see this sort of thing in legacy mobile data networks, where the folks building the networks didn't have a lot of TCP/IP experience at the time and were (understandably) bamboozled by the firewall-as-silver-bullet snake-oil. Many of these mobile networks, as they're becoming full-fledged wireless broadband providers, are ripping out these stateful DDoS chokepoints as they re-engineer their networks utilizing BCPs.
Marketing claims aside, firewalls do not provide any protection against DDoS; they actually are far more susceptible to DDoS themselves than are end-hosts, and they go down all the time under even low-scale DDoS attacks. Firewalls should not be wedged into the middle of SP networks, nor should they be placed in front of servers. They do make sense in front of workstations on end-customer access LANs, but that's about it.
So, wedging stateful boxes of any kind into the network (firewalls, 'IPS', et. al.), or wedging any sort of device permanently inline is generally a Bad Thing, and is to be avoided whenever possible.
There's some discussion of this topic in this preso:
http://files.me.com/roland.dobbins/k54qkv
and in this preso from the current NANOG meeting:
http://www.nanog.org/meetings/nanog48/presentations/Monday/Kaeo_FilterTrend_...
-----------------------------------------------------------------------
Roland Dobbins
Hi Roland, Just to pick a single statement and possibly interpret it out of context... On 25/02/2010 5:29 p.m., Dobbins, Roland wrote:
Firewalls should not be wedged into the middle of SP networks, nor should they be placed in front of servers.
Just looking at the last part of that (agree with the first bit) - are you suggesting that we put Windows servers bare on the intertubes? Is this for 'crack-that-box-wide-open' time trials or something? We also tend to firewall collocation servers unless specifically asked not to as often those running them aren't as paranoid as they should be about removing / turning off services. I mean, I'm all for a bit of a laugh, and there's nothing quite like a rampant fox in the corner of your hen house to add excitement to an otherwise boring weekend, but leaving boxes open for exploit feels a bit too much. Has some major thing happened and I missed it in terms of server security, or am I reading your statement incorrectly? Cheers, Gerard
On Thu, 25 Feb 2010, Gerard Creamer wrote:
Hi Roland,
Just to pick a single statement and possibly interpret it out of context...
On 25/02/2010 5:29 p.m., Dobbins, Roland wrote:
Firewalls should not be wedged into the middle of SP networks, nor should they be placed in front of servers.
Just looking at the last part of that (agree with the first bit) - are you suggesting that we put Windows servers bare on the intertubes? Is this for 'crack-that-box-wide-open' time trials or something?
We also tend to firewall collocation servers unless specifically asked not to as often those running them aren't as paranoid as they should be about removing / turning off services. I mean, I'm all for a bit of a laugh, and there's nothing quite like a rampant fox in the corner of your hen house to add excitement to an otherwise boring weekend, but leaving boxes open for exploit feels a bit too much.
Has some major thing happened and I missed it in terms of server security, or am I reading your statement incorrectly?
Those interested may like to check out the NANOG thread whch started with "I dont need no stinking firewall" on 5 January 2010... shades of that discussion and the merits of stateful firewalls infront of services come to mind... http://seclists.org/nanog/2010/Jan/126 for reference... Mark.
On Feb 25, 2010, at 1:06 PM, Gerard Creamer wrote:
Has some major thing happened and I missed it in terms of server security, or am I reading your statement incorrectly?
Stateful firewalls make no sense whatsoever in front of servers, since every incoming packet is unsolicited. Instead, the OS should be locked down, as should the apps/services, and policy should be enforced via stateless ACLs in hardware-based routers.
-----------------------------------------------------------------------
Roland Dobbins
Then is not DDOS on a firewall not a form/implementation of stateful inspection and management of protocols ??? They track traffic/processes/protocols looking for incomplete stateful session setups and perform termination when the state has not been setup/completed correctly thus relieving pressure/resource demands from the server behind them? Regards Robert Cotter Personal opinion and not one of any other organisation or person. -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Dobbins, Roland Sent: Thursday, 25 February 2010 7:04 p.m. To: nznog Subject: Re: [nznog] Stateful firewalls On Feb 25, 2010, at 1:06 PM, Gerard Creamer wrote:
Has some major thing happened and I missed it in terms of server security, or am I reading your statement incorrectly?
Stateful firewalls make no sense whatsoever in front of servers, since every incoming packet is unsolicited. Instead, the OS should be locked down, as should the apps/services, and policy should be enforced via stateless ACLs in hardware-based routers.
-----------------------------------------------------------------------
Roland Dobbins
On Feb 25, 2010, at 2:14 PM, Robert Cotter wrote:
They track traffic/processes/protocols looking for incomplete stateful session setups and perform termination when the state has not been setup/completed correctly thus relieving pressure/resource demands from the server behind them?
The capacity of even the largest firewalls is considerably smaller than what the naked servers themselves can handle.
This is all discussed in the linked presos and the relevant NANOG thread; it might be a good idea to read those first, so that we don't rehash previous discussions.
;>
-----------------------------------------------------------------------
Roland Dobbins
"Instead, the OS should be locked down, as should the apps/services, and policy should be enforced via stateless ACLs in hardware-based routers." It still amazes me how many people don't do this, but instead rely on their firewall to protect them. Also in my experience as someone mentioned you can run Windows Servers without a firewall on the internet, but locking them down is 100 times harder than locking down linux boxes. The NANOG thread is certainly worth reading if you have the time. -----Original Message----- From: nznog-bounces(a)list.waikato.ac.nz [mailto:nznog-bounces(a)list.waikato.ac.nz] On Behalf Of Dobbins, Roland Sent: Thursday, 25 February 2010 7:04 p.m. To: nznog Subject: Re: [nznog] Stateful firewalls On Feb 25, 2010, at 1:06 PM, Gerard Creamer wrote:
Has some major thing happened and I missed it in terms of server security, or am I reading your statement incorrectly?
Stateful firewalls make no sense whatsoever in front of servers, since every incoming packet is unsolicited. Instead, the OS should be locked down, as should the apps/services, and policy should be enforced via stateless ACLs in hardware-based routers.
-----------------------------------------------------------------------
Roland Dobbins
Dobbins, Roland wrote:
On Feb 25, 2010, at 10:09 AM, Shane Alcock wrote:
The questions I have: Is that initial statement correct? Is there anyone out there who is using (or knows anyone who is using) a stateful firewall in such a fashion?
It's utter nonsense, of course, as you rightly suspect.
;>
Thanks Roland, Thought provoking information (that's timely for me). Is protecting against vulnerabilities in the network stacks of the hosts a non issue these days? In some scenarios I have a concern because the devices that need to be protected are appliance boxes, i.e. patches for the OS (while typically derived from Linux or Windows) typically lag. So stateful firewalls or ALGs, assuming you build it big enough, merely force the attacker to be a little more intelligent and craft requests that are indistinguishable from legitimate requests. Since I'm not up to speed with what DDoS attacks consist of these days, do you have any good links to information, papers, presentations, etc., on what these attacks consist of? I'm particularly interested in what they're up to in terms of crafting legit requests, are they primarily trying to flood links by downloading large files, or use resources by running scripts, etc? Questions to you and the list: One of the advantages of firewalls is they generally come with good management systems, i.e. the ability to manage ACLs without writing them by hand. Any thoughts with regards to the best practise, or state of the art, these days in terms of managing ACLs on routers? If I have servers that need to make outbound requests as well, any thoughts for how this is best managed, i.e. stateless inbound, stateful outbound? Cheers Kris
On Feb 28, 2010, at 10:09 AM, Kris Price wrote:
Is protecting against vulnerabilities in the network stacks of the hosts a non issue these days?
Not typically. And note that exploits against the network stacks, 'inspectors', and so forth of stateful firewalls abound.
Since I'm not up to speed with what DDoS attacks consist of these days, do you have any good links to information, papers, presentations, etc., on what these attacks consist of? I'm particularly interested in what they're up to in terms of crafting legit requests, are they primarily trying to flood links by downloading large files, or use resources by running scripts, etc?
These may be of interest: http://www.arbornetworks.com/report http://files.me.com/roland.dobbins/y4ykq0 http://files.me.com/roland.dobbins/k54qkv
One of the advantages of firewalls is they generally come with good management systems, i.e. the ability to manage ACLs without writing them by hand.
Not in my experience, and I spent a decade working for the largest vendor of firewalls in the world. All the commercial ACL-management systems I've seen are junk. Matasano are supposed to be working on something actually useful, though I've yet to see it.
Any thoughts with regards to the best practise, or state of the art, these days in terms of managing ACLs on routers?
Most folks use some sort of versioning system combined with custom scripts. A rational IP addressing scheme helps a great deal, as well.
If I have servers that need to make outbound requests as well, any thoughts for how this is best managed, i.e. stateless inbound, stateful outbound?
Do it via a proxy; communicate with the proxy via a separate interface (doesn't even have to be globally reachable).
-----------------------------------------------------------------------
Roland Dobbins
| | > One of the advantages of firewalls is they generally come with good management systems, | i.e. the ability to manage ACLs without writing them by hand. | | Not in my experience, and I spent a decade working for the largest vendor of firewalls in | the world. All the commercial ACL-management systems I've seen are junk. I think it's more than worth mentioning that the "largest vendor of firewalls" you're talking about is not the largest vendor of firewalls in the word and has never been. I don't want to mention any commercial name here because this is not the point but this specific vendor is very well know to have very bad management systems for their firewalls. This is probably the only reason why they are not the largest vendor of firewalls in the world considering how much penetration they've got in the corporate routing and switching market. If you restrict a "firewall" to a firewall from a specific vendor, then you're only looking at the problem from a very specific angle and this can't allow you to draw real conclusions. Multiple other firewall vendors have very good management systems, this is a very well know fact. They might not meet what a Telco is expecting as firewall management, but this is a different question. cheers Florent
On Mar 1, 2010, at 7:22 AM, Florent Bouron wrote:
Multiple other firewall vendors have very good management systems, this is a very well know fact.
Checkpoint do in fact have a very good rule-management system, comparatively speaking, but that's not saying much.
;>
-----------------------------------------------------------------------
Roland Dobbins
I don't think it is common, but I have seen it done where the cost of receiving traffic for the customer is very high, such as mobile barriers, satellite providers, some less well developed countries, etc. Consider the case that you have are using a mobile carrier and are roaming, and paying $10/Mb of traffic. The kind person who had the dynamic IP address before you was using P2P software, and having a public IP address, was being used to seed a lot of connections. There now gone, but your now paying for all the incoming traffic (which your machine promptly drops since it is not listening for it). So in cases like this you might opt to use an APN that is firewalled, so you don't pay to receive traffic you have no interest in. There are also content transformation devices, which do things like downsize images inline that are in web pages before you receive them. Once again, to save the customer money paying for expensive bandwidth. Lots of state involved there. There are some large scale DDOS boxes, but they don't tend to be firewalls per-see (in that you don't create access rules and the like for them). Other statefull inline devices I can think of are traffic shapers and inline transparent proxy servers. If you divert your attention away from service providers to large enterprises, then I think you'll find some statefull firewalls handling volumes of traffic probably bigger than the largest service provider in NZ. I don't know what people like Microsoft and Google use, but chances are they have statefull firewalls doing tens of Gigabit's of throughput. -----Original Message----- "It is not uncommon to see transparent but stateful firewalls in ISPs (without NATs) today - to avoid DoS attacks. These firewalls do a job similar to SPNATs. What is the state maintenance and processing overhead in these firewall deployments? Can we reuse any lessons from them?"
On Feb 25, 2010, at 3:42 PM, Philip D'Ath wrote:
I don't think it is common, but I have seen it done where the cost of receiving traffic for the customer is very high, such as mobile barriers, satellite providers, some less well developed countries, etc.
The place do to this is at the edges via routing policies and stateless ACLs, not in firewalls.
And, no, the largest enterprise firewalls make all these marketing claims about the numbers they can handle, but the reality is quite different - having spent the better part of a decade working for the largest vendor of firewalls in the world, I can assure you of that.
;>
Again, you may wish to read the relevant presos and NANOG threads, all this has already been covered in copious details.
-----------------------------------------------------------------------
Roland Dobbins
Thinking about this further, let me say one word, "China". NAT upon NAT already being done, and firewalling to the extreme. -----Original Message----- "It is not uncommon to see transparent but stateful firewalls in ISPs (without NATs) today - to avoid DoS attacks. These firewalls do a job similar to SPNATs. What is the state maintenance and processing overhead in these firewall deployments? Can we reuse any lessons from them?"
On 25/02/2010 02:09, Shane Alcock wrote:
"It is not uncommon to see transparent but stateful firewalls in ISPs (without NATs) today - to avoid DoS attacks. [...] "
LOL.
The questions I have: Is that initial statement correct?
No. We help people scale beyond the limits that doing exactly this will put in their network, and the project normally kicks off doing what Roland tells you to do for free. Andy
participants (10)
-
Andy Davidson -
Bill Walker -
Dobbins, Roland -
Florent Bouron -
Gerard Creamer -
Kris Price -
Mark Foster -
Philip D'Ath -
Robert Cotter -
Shane Alcock