One of our user domains has just been heavily hit by *huge* quantities of
Non-Delivery Receipts addressed to random_string(a)domain_name. At it's peak
we were seeing around 50,000 NDR's coming in per hr. The worst has passed
(<fingers crossed>), but this does raise a couple of issues with the
current Internet mail infrastructure. A good discussion of the attack and
surrounding issues can be found here at
http://www.gossamer-threads.com/lists/spf/discuss/17041. In a nutshell we
need Sender Policy Framework (http://spf.pobox.com/) - or something
similar ASAP. There is nothing that could stop this happening to us again
tomorrow.
I don't think I've ever felt so powerless. RBL lists proved useless in the
line of defense, as these messages were coming from legitimate sources.
We were able to temporarily remove a few IP's here and there where the
remote MTA were going overkill throwing these NDR's at us. It would seem
that the original spambots that delivered the original message sent it
just about everywhere. An interesting stat is that we received NDR's for
atleast 352,423 unique recipients (<352,423 unique strings>@domain)!!!
We only had a number of things at our disposal to do to limit the damage:
Stop generating any 'unknown user' NDR responses ourselves (ignoring RFC876).
Split the affected domain away from the rest of our incoming mail stream
and then to start removing obvious NDR's from the isolated queues.
Remove more NDR's from the queue.
<repeat last step again>
<and again>...
-----Original Message----- From: Spencer Stapleton [mailto:sstapleton(a)compass.net.nz] Sent: Wednesday, 26 January 2005 6:18 p.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] backscatter attack
One of our user domains has just been heavily hit by *huge* quantities of Non-Delivery Receipts addressed to random_string(a)domain_name.
So, you were receiving NDNs targeted at random_string(a)domain, domain being the one you host? If that is the case, your server should just reject anything for non existant users. The problem with most setups today is that they just do a blind relay, which means you accept all those e-mails, then find out that the user doesn't exist and then you discard it (as it's a double bounce NDN). Usually this is solved by implementing virtual domains and/or relay access rules.
At it's peak we were seeing around 50,000 NDR's coming in per hr. The worst has passed
We had similar things at the UoA. After our system was re-architectured only directly targeted spam attack (with existing users) can affect it.
We only had a number of things at our disposal to do to limit the damage:
Stop generating any 'unknown user' NDR responses ourselves (ignoring RFC876).
Your server should reject it immediately (without accepting/queueing the e-mail). This means that the burden of NDN is on the remote server.
Split the affected domain away from the rest of our incoming mail stream and then to start removing obvious NDR's from the isolated queues. Remove more NDR's from the queue. <repeat last step again> <and again>...
Has anyone seen something similar? Did you manage to locate a better solution? I can't say I've enjoyed the last couple of days one bit!
Your solution is to implement virtual domains and reject e-mail immediately. Cheers, Bojan -- Bojan Zdrnja, CISSP, RHCE Security Implementation Specialist Information Technology Systems and Services (ITSS) Ph 09 373-7599 x82035 The University of Auckland, New Zealand
One of our user domains has just been heavily hit by *huge* quantities of Non-Delivery Receipts addressed to random_string(a)domain_name.
Something like this can be stopped using SRS with SPF. If the messages are bounced back and it doesn't have SRS then it will be rejected. People who have SPF will reject in the 1st instance (as the messages won't be coming from the corect place) and your server will reject in the 2nd instance if bounced back with invalid/no SRS. Thanks Craig http://www.spam.co.nz
Aha. Yes, it looks like this is the way forward! After a quick google I came across http://www.libsrs2.org/ which provides some additional details. Cheers! Spencer. Craig Whitmore said:
One of our user domains has just been heavily hit by *huge* quantities of Non-Delivery Receipts addressed to random_string(a)domain_name.
Something like this can be stopped using SRS with SPF. If the messages are bounced back and it doesn't have SRS then it will be rejected. People who have SPF will reject in the 1st instance (as the messages won't be coming from the corect place) and your server will reject in the 2nd instance if bounced back with invalid/no SRS.
Thanks Craig http://www.spam.co.nz
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
-- Systems Engineer Compass Communications http://www.compass.net.nz
How does your receiving MTA determine what is a valid local user at point of delivery? LDAP lookups? Do you mind providing technical details? What MTA are you using? Your right, it would have stopped most of the issues, but the sheer volume of mail we were receiving would have meant that the lookup themselves would have possibly been an issue. Definitely less of one than actually accepting it, even if we were >/dev/null'ing them though. cheers, Spencer. Bojan Zdrnja said:
Your server should reject it immediately (without accepting/queueing the e-mail). This means that the burden of NDN is on the remote server.
On Thu, 27 Jan 2005, Spencer Stapleton wrote:
How does your receiving MTA determine what is a valid local user at point of delivery? LDAP lookups?
LDAP is probably a good way to go since once it's setup you can use it for Mail, Radius, Web or whatever else.
Do you mind providing technical details? What MTA are you using?
Postfix or exim4 should will handle what you want pretty easily. You just do a few lookups when each email come in to determine if the domain and username are within your system. You can obviously store other stuff in LDAP like aliases, spam and virus settings, quotas, vacation etc. If you want to get some idea on exim4 you could attend Phillip Hazel's talk at nznog next week. Pick up the book too.
Your right, it would have stopped most of the issues, but the sheer volume of mail we were receiving would have meant that the lookup themselves would have possibly been an issue.
LDAP lookups are pretty low overhead compared to a SMTP connection. A reasonably modern PC can handle a thousand lookups per second without a problem. Even doing a few lookups for each email isn't usually a problem. -- Simon J. Lyall. | Very Busy | Mail: simon(a)darkmere.gen.nz "To stay awake all night adds a day to your life" - Stilgar | eMT.
-----Original Message----- From: Spencer Stapleton [mailto:sstapleton(a)compass.net.nz] Sent: Thursday, 27 January 2005 12:51 a.m. To: b.zdrnja(a)auckland.ac.nz Cc: nznog(a)list.waikato.ac.nz Subject: RE: [nznog] backscatter attack
How does your receiving MTA determine what is a valid local user at point of delivery? LDAP lookups? Do you mind providing technical details? What MTA are you using?
All our e-mail systems use postfix as MTA. At the moment we use local hashed files which are built either in real time (when new account is added, a central machine pushes files to front end gateways and rebuilds hash files) or daily, when that suites (ie. for student e-mails which are added or removed once per day). This was done to speed up transition from the old system. We will soon move to LDAP based repository - it's very lightweight and, as was written in another post here as well, presents an ideal means of storing data.
Your right, it would have stopped most of the issues, but the sheer volume of mail we were receiving would have meant that the lookup themselves would have possibly been an issue. Definitely less of one than actually accepting it, even if we were >/dev/null'ing them though.
Lookups are usually totally inexpensive and MTA can cache results for a while. In emergency cases you can go to local hashed files (ie, if you have the data in LDAP you can export it to a text file and generate a hash), but I believe that, on a properly configured system, this won't be a problem. Cheers, Bojan -- Bojan Zdrnja, CISSP, RHCE Security Implementation Specialist Information Technology Systems and Services (ITSS) Ph 09 373-7599 x82035 The University of Auckland, New Zealand
Many thanks to everyone who got back to me on this topic. The way forward appears to be a two-pronged approach: Ensure that we refuse 'unknown_users' at our receiving MTA - probably best achieved via LDAP lookups. Implement SPF/SRS. With the benefit of hindsight I can thoroughly recommend performing the above *before* being hit:) Neither will be trivial for us to implement (other than exporting our own SPF TXT records IN DNS), and will require the installation of OpenLDAP (which will admitedly be useful for 101 other things) as well as upgrading all our MTAs. I @#$@#$S^@# hate spammers!! Bojan Zdrnja said:
-----Original Message----- From: Spencer Stapleton [mailto:sstapleton(a)compass.net.nz] Sent: Wednesday, 26 January 2005 6:18 p.m. To: nznog(a)list.waikato.ac.nz Subject: [nznog] backscatter attack
One of our user domains has just been heavily hit by *huge* quantities of Non-Delivery Receipts addressed to random_string(a)domain_name.
So, you were receiving NDNs targeted at random_string(a)domain, domain being the one you host? If that is the case, your server should just reject anything for non existant users. The problem with most setups today is that they just do a blind relay, which means you accept all those e-mails, then find out that the user doesn't exist and then you discard it (as it's a double bounce NDN). Usually this is solved by implementing virtual domains and/or relay access rules.
At it's peak we were seeing around 50,000 NDR's coming in per hr. The worst has passed
We had similar things at the UoA. After our system was re-architectured only directly targeted spam attack (with existing users) can affect it.
We only had a number of things at our disposal to do to limit the damage:
Stop generating any 'unknown user' NDR responses ourselves (ignoring RFC876).
Your server should reject it immediately (without accepting/queueing the e-mail). This means that the burden of NDN is on the remote server.
Split the affected domain away from the rest of our incoming mail stream and then to start removing obvious NDR's from the isolated queues. Remove more NDR's from the queue. <repeat last step again> <and again>...
Has anyone seen something similar? Did you manage to locate a better solution? I can't say I've enjoyed the last couple of days one bit!
Your solution is to implement virtual domains and reject e-mail immediately.
Cheers,
Bojan
-- Bojan Zdrnja, CISSP, RHCE Security Implementation Specialist Information Technology Systems and Services (ITSS) Ph 09 373-7599 x82035 The University of Auckland, New Zealand
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
-- Systems Engineer Compass Communications http://www.compass.net.nz
Quoting Spencer Stapleton
Neither will be trivial for us to implement (other than exporting our own SPF TXT records IN DNS), and will require the installation of OpenLDAP (which will admitedly be useful for 101 other things) as well as upgrading all our MTAs.
I think you'll find taking the time to learn how LDAP works and the implementation of it to be rather useful in the long run. As you (and Simon, earlier) say, it can be used for a range of other things. Flat files be gone! There are some great resources on the interweb as to where to start. Cheers Michael
On Wed, Jan 26, 2005 at 06:18:09PM +1300, Spencer Stapleton said:
Stop generating any 'unknown user' NDR responses ourselves (ignoring RFC876).
Since 6am this morning, for a domain nearby, we've seen: 1613 received 1077 delivered 8 forwarded 1 deferred (1 deferrals) 2 bounced 25611 rejected (95%) 0 reject warnings 0 held 0 discarded (0%)
Has anyone seen something similar? Did you manage to locate a better solution? I can't say I've enjoyed the last couple of days one bit!
When we ran the domain on qmail, we just accepted all users(a)domain, and wrote the unknown users to /dev/null. This at least stopped the outbound ndr creation, and kept the outbound mail queue at a sensible level. I figured sending NDR's was just not a friendly thing to do, and so decided not to do so. We were still receiving something like 15GB of mail/month to unknown users, which sucked, so we moved from qmail to postfix a few weeks ago, and setup postfix to reject unknownusers(a)domain, and to run amavisd/clamav, both at SMTP receive time. Of those rejections above, about 500 are virii, and the majority of the rest (25000 messages) seem to be incoming NDR's. (qmail defenders, yes, I know you can configure qmail to do all the above, but it involved patches and recompilation from scratch, and, well, I just couldn't be arsed). I suspect that an SMTP reject is probably the only realistic balance between DOS'ing somebody else with NDR's, and not warning legit senders of their typos. I figure the dictionary attack the above config allows is probably the least of my problems, and could probably be mitigated with some kind of IDS. Cheers Si
-----Original Message----- From: Simon Blake [mailto:simon(a)citylink.co.nz] Sent: Wednesday, 26 January 2005 10:10 p.m. To: Spencer Stapleton Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] backscatter attack
When we ran the domain on qmail, we just accepted all users(a)domain, and wrote the unknown users to /dev/null. This at least stopped the outbound ndr creation, and kept the outbound mail queue at a sensible level. I figured sending NDR's was just not a friendly thing to do, and so decided not to do so.
Yep - that was good for other people (very nice of you :), however, it was terribly expensive for your server. One of the worst cases I've seen is the default setup that distributions often use with Cyrus IMAP server. Usually, the MTA can't tell what are local users (because they are accounts under Cyrus) so it accepts absolutely everything, tries to deliver to Cyrus and then bounces it back.
I suspect that an SMTP reject is probably the only realistic balance between DOS'ing somebody else with NDR's, and not warning legit senders of their typos. I figure the dictionary attack the above config allows is probably the least of my problems, and could probably be mitigated with some kind of IDS.
As it was mentioned in another post, SPF with SRS is the real solution, however, the first step would be to configure servers properly so they don't generate NDNs which are not needed. Our servers reject ~200.000 e-mails daily just based on incorrect recipients. Cheers, Bojan -- Bojan Zdrnja, CISSP, RHCE Security Implementation Specialist Information Technology Systems and Services (ITSS) Ph 09 373-7599 x82035 The University of Auckland, New Zealand
That's the story. One domain here is bulletin.net which just happens to be one of the pseudo-random domains generated by one particular worm that came out in 2003 (IIRC). Ever since its release, we've received bounces from the worm recipient server; currently averaging 50,000 attempted deliveries per day. All are rejected immediately using postfix with local (virtual_pcre) recipient map. Interestingly, I added an SPF record on 7 Jan 2005. In the week prior the daily average was 57k, in the three weeks since the average has been 46k. Figures may be distorted by the holiday season though.. Zach. Simon Blake wrote:
Since 6am this morning, for a domain nearby, we've seen:
1613 received 1077 delivered 8 forwarded 1 deferred (1 deferrals) 2 bounced 25611 rejected (95%) 0 reject warnings 0 held 0 discarded (0%)
Has anyone seen something similar? Did you manage to locate a better solution? I can't say I've enjoyed the last couple of days one bit!
-- Services & Support, Bulletin Wireless (NZ) http://www.bulletinwireless.com/ Ph +64 9 307 1764 Mob +64 21 115 0269 Fax +64 9 307 1148
Zach Bagnall wrote:
Interestingly, I added an SPF record on 7 Jan 2005. In the week prior the daily average was 57k, in the three weeks since the average has been 46k. Figures may be distorted by the holiday season though..
Hmm.. http://y3m.net/files/bulletin.net.rejected.png -- Services & Support, Bulletin Wireless (NZ) http://www.bulletinwireless.com/ Ph +64 9 307 1764 Mob +64 21 115 0269 Fax +64 9 307 1148
Not wanting to sound like I'm doing 'my stats are bigger that your stats', but Last Thursday we rejected 518451 delivery attempts for 'user_unknown' to our affected domain. I have had better weeks! spence. Zach Bagnall said:
That's the story. One domain here is bulletin.net which just happens to be one of the pseudo-random domains generated by one particular worm that came out in 2003 (IIRC). Ever since its release, we've received bounces from the worm recipient server; currently averaging 50,000 attempted deliveries per day. All are rejected immediately using postfix with local (virtual_pcre) recipient map.
Interestingly, I added an SPF record on 7 Jan 2005. In the week prior the daily average was 57k, in the three weeks since the average has been 46k. Figures may be distorted by the holiday season though..
Zach.
Simon Blake wrote:
Since 6am this morning, for a domain nearby, we've seen:
1613 received 1077 delivered 8 forwarded 1 deferred (1 deferrals) 2 bounced 25611 rejected (95%) 0 reject warnings 0 held 0 discarded (0%)
Has anyone seen something similar? Did you manage to locate a better solution? I can't say I've enjoyed the last couple of days one bit!
-- Services & Support, Bulletin Wireless (NZ) http://www.bulletinwireless.com/ Ph +64 9 307 1764 Mob +64 21 115 0269 Fax +64 9 307 1148 _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
-- Systems Engineer Compass Communications http://www.compass.net.nz
"Spencer Stapleton"
We only had a number of things at our disposal to do to limit the damage:
Stop generating any 'unknown user' NDR responses ourselves (ignoring RFC876).
It's best to do SMTP rejects for unknown users; this will help cut the load on your servers if the forged From addresses don't exist at your domain. This is relatively straightforward for most MTAs - I believe postfix can do LDAP queries against AD, or can talk to a postgresql database. Since most of the forged addresses will be nonexistent, this should help quite a bit. (Bonus - it will also stop you generating this kind of traffic for other sites in the future.) Consider publishing SPF records for the domain if you know where your mail should be coming from. Some people are talking about a RBL for hosts that send backscatter, but I don't know if it's come to anything yet, or if it will be feasible to use; possibly there will be too much legitimate mail rejected for some of us.
Has anyone seen something similar? Did you manage to locate a better solution? I can't say I've enjoyed the last couple of days one bit!
We've seen spikes where our main domain was forged into email, but only around 5-10x the usual load and not for too long. 550'ing mail to nonexistent users definitely helps. cheers, Jamie -- James Riden / j.riden(a)massey.ac.nz / Systems Security Engineer Information Technology Services, Massey University, NZ. GPG public key available at: http://www.massey.ac.nz/~jriden/
participants (8)
-
Bojan Zdrnja -
Craig Whitmore -
James Riden -
Michael Jager -
Simon Blake -
Simon Lyall -
Spencer Stapleton -
Zach Bagnall