Darxus

I made up a new way of blocking spam: MTX records

I made up a new way of blocking spam: MTX records

Previous Entry Add to Memories Share this! Next Entry
2009-09-29
http://www.chaosreigns.com/mtx/

MTX records are just DNS A records on your DNS server stating that an IP is a legitimate mail transmitter.

There is a SpamAssassin plugin for it on that page.
  • So, you basically re-invented SPF records?
    • Yes, it is intended to be the functionality of SPF without the forwarding breakage which is apparently a significant barrier to adoption.

      When a user configures one email account to automatically forward their email to another email account, it is generally done without rewriting the "envelope from" which SPF uses for validation. So the forwarding server's IP doesn't match the SPF record of the original "envelope from". And the "envelope from" rewriting solutions seem to be an even greater barrier to adoption.
      • Oh, heh, I didn't notice that you mentioned SPF and DNSWL at the bottom of the page. Regardless, I was just mildly teasing, and I don't think I was clear that I was.

        I'm curious, though: How does this not break un-munged forwarding? If the envelope sender is unchanged, then the forwarding IP is going to unlikely going to have an MTX record at the originating domain, thus causing the same problem with forwarding accounts that SPF has.

        As an amusing side note, I independently had a similar idea back in 2003, when (apparently) some spammer who was pissed off I was reporting my spam to Spamcop started forging my domain on his spam. I didn't really think through implementation details except, I reasoned, "MX records say where mail is supposed to go to; it would be awesome if we had records that say where it's supposed to come from. Since it's the opposite of an MX, it would be called an 'XM' record."

        I guess, as I thought of it, it would require a new RR type. And it was shortly after that that I heard of SPF records (which co-opt the TXT field and thus don't require a new RR type), and I think I also heard that SPF records weren't the first proposal of their type.

        I do like that yours doesn't have a confusing syntax like SPF records, just uses simple A records (like standard RBLs/DNSWLs), and therefore also doesn't require a new RR type. And it would be pretty easy to use wildcards, at least on octet boundaries, to whitelist or explicitly blacklist, say, a whole /24 (if you're using BIND; it would be even easier with something like rbldnsd, to which you could delegate mtx.yourdoma.in, and which can use CIDR notation).
        • It doesn't break un-munged forwarding because there is no association with a domain name. The only authority is the delegate of the PTR record, they are the ones that point the IP to a domain, and then only to verify that the IP is a legitimate mail transmitter, not that it is specific to that domain.

          It's cool that you basically came up with the idea of SPF before it was implemented. You should have implemented it :)

          SPF did actually end up getting its own RR type. My domain has the records. Could still take a long time for that to become as widely supported as the TXT records.

          Yeah, the ability to delegate the mtx subdomain only occurred to me last night. Along with the wild cards. Good thing I did it that way. It can be useful to follow widely established patterns even if you don't fully grasp them. I finally fully grasp the reversal of IPs in host names. :P
          • Oh! I was laying out a typical mail-forwarding scenario for you to show me where I was misunderstanding, and I think I realized that we're assuming different things about the system. I thought that you were saying that this was a system which is intended to say which servers are authorized to send mail for a domain, while I realized that you are laying out a system in which it is specified which servers are authorized to send mail, period.

            Is that it?

            So, basically, whoever controls reverse DNS for a particular IP gets to set the FQDN of the PTR record that's returned, and whoever controls the domain in that FQDN gets to say whether or not that machine is authorized to send mail at all. (I was thinking that you'd use the domain part of the From: address.)

            I was thinking more along the lines of SPF, where if I'm forwarding mail from myname.com to, say, my Gmail address, you can't (under your system) create a record that Gmail can look up that says that my server is authorized to send mail on behalf of yourdomain.com.

            Oh, heh, I just read the "Conceptual origin" section. I think I understand your proposal better now; I just misunderstood your intent.

            One thing about SPF that could maybe help alleviate the problem is the use of the Resent-From: header (which Mutt, for example, adds when you use the bounce ('b') command), which wouldn't change the From: header, but it would allow for the proper SPF records to be looked up.

            Now that (I think) I understand the purpose of MTX records a bit better, I don't think that delegation would be as useful.
            • Yes.

              People are also very resistant to modifying SMTP to get around SPF's forwarding problem.
              • My response to that is that people were also once opposed to limiting relay access on their mail servers, but we eventually got over that, too. ;-)

                Honestly, I think that MTX records are of limited utility—mainly useful in cases where the owner of the IP's authoritative name server and the person who runs the server (and therefore, would presumably own the domain name that the IP reverses to) disagree on whether or not it should be sending mail. I don't think that domains or zones that publish MTX records for machines which spam would get blacklisted in any meaningful way (certainly not as meaningful as DNSBLs today).

                I am certainly open to being convinced otherwise and/or proven wrong, though.
                • Limited utility? I would say it applies to the vast majority of the spam I receive that has a PTR record (and I reject everything without a PTR record). In fact, all of the first 10 from yesterday's mail log are broadband or dialup:

                  # cat mail.log.0 | grep reject: | awk '{print $10}' | grep -v '^unknown\[' | cut -d'[' -f1 | head
                  cable200-116-4-57.epm.net.co
                  cable200-116-4-57.epm.net.co
                  stgt-5d841c43.pool.mediaWays.net
                  cable200-116-4-57.epm.net.co
                  201-68-227-171.dsl.telesp.net.br
                  ppp-124-120-118-226.revip2.asianet.co.th
                  201-68-211-174.dsl.telesp.net.br
                  33-21-135-95.pool.ukrtel.net
                  109-184-4-179.dynamic.mts-nn.ru
                  cable-94-189-198-185.dynamic.sbb.rs

                  (Just because you're the most annoying about this, I used cat because I've been switching between how many files I'm running through grep.)

                  Also, in cases where an MTX validating domain sends both legit mail and spam, I plan to provide the ability to blacklist so that it only negates the bonus for using MTX, so in that case the net effect is 0. Or whatever you want it to be for that domain.

                  So for:
                  Most spamming hosts: The owner creates no MTX record = SA penalty
                  Most legit mail servers: The owner creates an MTX record = SA bonus
                  Hosts which send both spam and non-spam: The owner creates an MTX record, and a blacklist is used to negate the bonux = SA score 0
                  Spammers using MTX: Major penalty for blacklisting.
                  Legit servers without MTX: With no MTX record = SA penalty, starting at 0 and raising gradually as adoption spreads.

                  So spammers are actually better off not using MTX, because the penalty for getting blacklisted is worse than the penalty for not having the record. Also, MTX records allow the spam to be associated with their domain, and therefore registrar, who can then be subpoened for the spammer's identity.
                  • I know that is not a random sample, but I'll point out that all of those are on the DUL. You could assign a SA penalty for that.

                    DNSWL's big weakness (centralization) is also its big advantage: They vet the entries on the list, at least somewhat, and assign a trustworthiness rating, and if one of IPs on the list starts spamming, it gets removed (and probably placed on at least one of several DNSBLs). (I myself am using my own blacklist, DNSWL, bl.spamcop.net, psbl.surriel.com, and cbl.abuseat.org.)
                    • I'm not saying those IPs weren't easy to block without MTX. They were, in fact, easily rejected before they got to SpamAssassin. Or even the one RBL I use before SA, zen.spamhaus.org. All
                      were user unknown or non-FQDN helo.

                      My point is that a blacklist of MTX abusers should be much easier to keep up to date than, for example, the DUL or zen.spamhaus.org. Less work, more up to date, more accurate.

                      That is certainly an important advantage of DNSWL, one that I mention in the new "Comparisons" section :)

                      But I assure you, that list is a pain in the ass to maintain. Speaking as a long time admin of it, and probably the person who introduced you to it. You might be interested in reading http://www.chaosreigns.com/mtx/background/ if you haven't.
                      • I didn't realize you're an admin of DNSWL. And yes, I think I recall reading something on your site some time ago where you mentioned DNSWL. When I started using various BLs, I decided to use DNSWL, too.

                        Besides spammers creating their own MTX records and every email admin in the world needing to maintain their own MTX blacklist, I don't hold out high hopes for people creating MTX records. If my own electric company and dice.com can't configure their mail servers with a proper hostname (thereby getting caught by Postfix's reject_unknown_helo_hostname, which I've had to temporarily remove from time to time, and that I've removed for the time being due to job hunting), I can't see that many people creating MTX records.

                        Hell, I think the only reason people create MX records to begin with is so that mail doesn't go to what is usually their main web server(s) and so that they can have a backup mail server to receive incoming messages in the event the main server goes down. SPF isn't faring very well, either; between people that haven't heard of it and people that refuse to use it, and people that don't bother using it, it's not getting very far.
                        • Yeah, I found DNSWL as a result of coming up with the idea myself and checking to see if it had already been done. And it was. Fully up and running, in a way that I was entirely happy with. Although Matthias didn't like the idea for penalizing anybody for not being on the list, as I wanted to do. I also host one of the DNSWL servers. Apparently since March 2007.

                          I think SPF is doing pretty well. Many large entities are using it, including gmail, AOL, and hotmail. And that's in spite of the forwarding breakage which a bunch of people are very emotionally opposed to creating SPF records because of. When I went looking for major sites using SPF, I had no difficulty:

                          http://www.chaosreigns.com/spam/#spf_users

                          And honestly, I think MTX is easier to understand and implement than a valid helo. How many people understand the terms "helo" vs. "whitelist"? When I very recently created an SPF record for my helo, since SA is checking those now, it took me a little while to figure out what HELO my server was actually sending. It was the obvious one, and defined in a pretty obvious place, but still not instant to confirm. And dice.com does have an SPF record.

                          And I think simplicity is a significant advantage for MTX over SPF. One mail server, one extra DNS A record. The entire protocol can be specified in one line. Have you seen the spec for SPF, or some of the crazy stuff they come up with in records?

                          I really don't think there will be many spammers who use MTX, because from the start it comes with blacklisting. Hell, I should state that as a rule for implementations (done). And if they do, a public centralized and very small MTX specific blacklist will be created for those who don't wish to maintain their own. I have considered the possibility that such a thing might be necessary for wide spread adoption, even if there are no spammers using MTX. But still I think MTX has the advantage of not depending on centralized authority because I believe a blacklist will be easy enough for one person to maintain.

                          Also, most importantly, I benefit from MTX now. I catch more spams (score MTX_FAIL 2), and if I get a false positive, the sender gets informed, and has the possibility of doing something to fix it.
                          • I think SPF is doing pretty well. Many large entities are using it, including gmail, AOL, and hotmail. And that's in spite of the forwarding breakage which a bunch of people are very emotionally opposed to creating SPF records because of. When I went looking for major sites using SPF, I had no difficulty:
                            Okay, I concede that point, and I also think I didn't convey what I wished, which is that among people who have heard of it and aren't philosophically opposed to it, it's more than likely laziness that's preventing them from creating just one additional DNS record, as opposed to one additional record per server.

                            (BTW, the #spf_users tag on that link didn't work. You have it as an 'id' attribute to an h2 tag, whereas you need an <a name="spf_users"> tag. I'm using Firefox 3.5 on Linux.)

                            I've not looked at the spec for SPF, but the record itself is ugly, I agree. Fortunately, they have a CGI script that will generate the record for you. ;-)

                            Really, if you don't know what a 'helo' is, you shouldn't be running a mail server, IMNSHO. I think that "Hello, my name is _____" is a pretty simple concept for people to understand, anyway. "You have to give me a real name that I can look up" is pretty simple, too, and yet people still get it wrong.

                            And while dice.com does have an SPF record, it doesn't help if you're requiring mail servers to tell you a real name that you can look up, and they're not properly identifying themselves:

                            Feb 6 13:33:08 dr-evil postfix/smtpd[14286]: NOQUEUE: reject: RCPT from mailbox51.dice.com[65.198.147.51]: 450 4.7.1 <colomailbox.dice.com>: Helo command rejected: Host not found; from=<support@dice.com> to=<my@ddre.ss> proto=ESMTP helo=<colomailbox.dice.com>

                            That was my point. If people can't even get the hostnames they're using for their mail servers in DNS properly now, why would they create extra records? (I no longer have the log entries, but my electric company was using srp.gov, which doesn't exist, instead of srpnet.com. I don't think "Set your hostname to something that can be looked up" is too onerous of a rule, either. They (the electric company, not Dice) have since fixed it, probably in part because I was just bouncing their mail after I temporarily allowed their signup confirmation message in.)

                            As for catching more spam, I would be interested to know if it has improved your accuracy rate. I can create a regex that will reliably catch 100% of spam. Here it is: /./

                            I'm still working on the false positives, though. ;-)

                            Anyway, I realize that it hasn't even been a week, but since you've stated that you're catching more spam with it now, I'm interested in the particulars behind that claim. (In my opinion just "catching more spam" isn't enough; "improving accuracy" is the goal, and the false positives you mentioned concern me a little.) Has it actually improved your accuracy rate? Is the rate different than if you had merely penalized every email by the same amount, or if you had lowered the SA threshold by the same amount? Do you know if there are any other domains with MTX records besides yours and mine?

                            Sorry if I come across as overly negative; I just don't see a significant benefit (although there's obviously no harm by creating the record, which is why I went ahead and did it).

                            • When did "id" become a valid attribute of all entities? Maybe that wasn't until HTML5? It works for me with firefox v3.5.3. The WDG validator isn't throwing an error on it, but there are enough other errors on that page that they could be hiding it.

                              I highly doubt the SPF CGI script will generate the ugly SPF records I've seen. And I suspect they're ugly out of necessity.

                              I pointed out that dice.com has an SPF record only because you gave it as an example of someone too incompetent to provide a valid helo, and therefore, I think, unlikely to create an MTX record.

                              Obviously, MTX has increased my accuracy rate for spam, while decreasing accuracy for non-spam. Overall accuracy isn't really worth comparing, I think. But the part that matters to me is that the cases where I'm decreasing accuracy with false positives, I'm notifying the sender and giving them a way to fix it.

                              I'm sure the current effect is statistically very similar to giving all emails an extra +2. The difference, as I said, is the notification, and ability to fix it.

                              I don't mind your negativity, I appreciate the additional analysis.

                              And I've gotten quite a lot more negativity than from you. It is apparently common for people to think that they came up with a new and useful method for dealing with spam. They're almost always wrong. It shows in the responses.
                            • That page is now validated HTML5. I'm curious if the spf_users link works for you now.
        • I just added a "Conceptual Origin" section. All about SPF and DNSWL.
  • a) How is this *functionally* different, as opposed to *syntactically* different, from an SPF record?

    b) I have SPF records for all of my domains. Have had them for more than 5 years now. As of this writing, not enough of the rest of the world has also done it, making it still something i can't really use as a decent wheat/chaff separator. Sigh. Even if you reply with something I'm not seeing in (a), I don't hold much hope.
    • a) See my previous comment for functional difference (forwarding breakage).

      b) I tried to address this in the "Implementation Sequence" section. I think it's useful for whitelisting now. And I am hopeful that people will adopt it for that reason - reducing false positives. Meanwhile I am also using it to increase the spamminess score of email without an MTX record to catch more spam. This requires a willingness to cause a small number of false positives - which I have largely because I have SpamAssassin configured to run during SMTP delivery so I can reject spams, causing un-forged senders to get an error message without causing backscatter.
Powered by LiveJournal.com