Killing comment spam with Bayes
I’m not a fan of comment moderation, nor of CAPTCHAs, registration requirements, or anything else that makes it hard to leave a comment. I don’t use them on this website, and I get a lot of comment spam as a direct result. I needed a solution.
I’ve gone through three phases of spam fighting. In the first phase, I disabled comments on old posts that had been found by the evil robots. That worked fine, until they struck some posts that were still getting genuine comments.
In phase two, I moved to a simple blacklist system. I wrote a little scrubber script in Ruby that runs every few minutes, looks through recent comments, and hides them if they contain one of a small number of spammy features. Spammers generally peddle the same old crap, so this was pretty effective. But I had to keep up the blacklist.
Phase three came about during an idle moment at Railsconf Europe last week. To pass the time, I dug up a bit of code I’d written earlier that attempted to use a Bayesian classifier to partition comments into ham and spam. I got it running and trained it against the old comments (I keep them all, ham or spam). It worked reasonably well, with a few false positives and false negatives. With a bit more work today, I’ve got it working with 100% accuracy. I’m suffering a storm of spam right now, and it’s correctly identifying and hiding all of them.
I’ve found two things particularly effective in improving reliability:
- Use the Robinson-Fisher combiner algorithm.
- Tokenise and include everything.
By everything, I mean:
- Every word and non-word in the comment text
- Every part of a URL supplied
- Every word in the commenter’s name
- Every non-punctuation part of the email address
- The post the comment refers to
- The first two octets of the originating IP address
Things that aren’t in the comment text are tokenised as fake words with a prefix indicating their origin (e.g. x-url-com), so that the classifier can weight them separately from regular words.
By adding these extra features into the mix, it seems to be possible for the classifier to distinguish copy-and-paste spam from genuine comments. I’ll see how well it performs in the weeks ahead.
2007-09-23 01:23 UTC. Comments: 2.
Michael Neumann
Wrote at 2007-09-23 12:40 UTC using Firefox 2.0.0.6 on FreeBSD:
I use recaptcha.net on my blog. It works really fine, and I don’t think it’s annoying for users to fill out a captcha. Another technique I heard about is very easy: In the comment form, just put a input type=”text” name=”email” field, which is not visible to the user. Some spammers enter an email address into such a field. Real users don’t, as they don’t see it!Paul Battley
Wrote at 2007-09-23 13:07 UTC using Firefox 2.0.0.6 on Mac OS X:
The biggest issue I take with normal CAPTCHAs is that I sometimes find them very difficult to read—particularly when they require case or 0/O to be distinguished.That recaptcha.net thing looks like a smart idea, since it’s using words that are by definition hard for computers to read, but should be readily legible to humans, and making something useful out of them.
It strikes me that there’s a weak point in recaptcha.net, in that it’s vulnerable to a distributed attack: if multiple spambots co-operate and give the same answer for the same word, the system will assume that to be correct.