SPAM protection methods N° 1: Honeypot

#block#honeypot#spam#spambot#spamprotection

A tiresome topic since the dawn of the internet is SPAM (no, not the canned meat, that has been around longer than the internet). On a bored weekend I gave some thought to how we can avoid SPAM (alternatively: web crawlers), and from that came the series "SPAM protection methods". Part 1 of this series is about locking annoying SPAM bots out of our website, but it can also be used to keep automated tools like SiteSucker, as well as automated scripts via Chromium or similar, away. As always - and especially here: this post is only meant as inspiration and shouldn't be copied 1:1 without thinking. If used incorrectly, the script can lock regular users out of the site (for example if they use a content preloader or if the browser itself preloads the content).

1. Concept

Basically, in the first step we want to detect the bot and in the second step block it. My original idea was to block the bot via the htaccess file. As soon as our script detects a bot, it should write the line deny {IP} to the end of the htaccess file and thereby lock the bot, or rather its IP address, out of our site. Good idea, problematic implementation: when I realized that this requires making the .htaccess file writable with chmod, my pulse strangely rose to somewhere around 200. An alternative had to be found, and it's "sessions". We simply store the user's status in the session and the blocked IP addresses in a database. Whenever a new user arrives on the site, we check the database to see whether their IP address is blocked. Depending on that, they get a session variable assigned that we can check on every page view, which saves us some SQL queries. Note: sessions alone are of course not enough. Otherwise a simple cookie deletion helps. More problematic, however, is detecting at all whether it's a bot or not. My solution so far was merely to insert a "hidden link" in the form Ultra Fancy Link. A spam bot that can't tell whether the link is really visible or not (since only the href attribute matters to it, not the style tag) calls target.php, which contains our honeypot. However, that led to browsers or plugins that partly precache the content also calling the page, so a "normal" user can fall into our honeypot too. And that must of course be avoided. I could work around the problem fairly easily by writing the path https://example.com/target.php into the robots.txt. Apparently browsers that precache content respect the robots.txt (at least the good ones), so a "false positive" can be avoided. target.php now does nothing but feed the database with the browser's IP address.

2. Implementation

Step 1: Database

First we create the tables in the database:

CREATE TABLE `spamblock` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `ip` text,
  `stamp` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

The table spamblock now contains the obligatory primary ID, a field "ip" into which the IP address will be entered later, and "stamp", which will later contain the timestamp.

Step 2: target.php

Next up is target.php (you can of course name it differently). It inserts the IP address and the timestamp into our table and sets the session variable.

<?php
$dbname = "dbname";
$username = "username";
$password = "password";

$pdo = new PDO("mysql:host=localhost;dbname=$dbname", $username, $password);
$ip = $_SERVER['REMOTE_ADDR'];
$stamp = time();

$stmt = $pdo->prepare("INSERT INTO spamblocked (ip, stamp) VALUES (:ip, :stamp)");
$stmt->bindParam(':ip', $ip);
$stmt->bindParam(':stamp', $stamp);
$stmt->execute();
?>

Of course dbname, username and password have to be replaced with your own data here.

Step 3: Checking the variable

On every page view we now have to check whether $_SESSION['spamblock'] is true or false. Either we auto-prepend the following script via htaccess: php_value auto_prepend_file "/var/www/virtual/david/check.php" (note: absolute path!) or we add the following lines at the start of every page:

<?php
session_start();
if (isset($_SESSION['spamblock']) && $_SESSION['spamblock'] === true) die();
?>

With this we check whether the variable $_SESSION['spamblock'] is set and has the value true. If so, we end the script.

Step 4: Robots.txt

Next we add target.php to our robots.txt so we don't scare off any "good" crawlers and don't run into problems with precaching.

User-agent: *
Disallow: /target.php

The structure of the robots.txt should be self-explanatory.

Step 5: Hidden link or other redirect

As a last step we add the hidden link to our regular page, preferably as far "up front" as possible (index.php is good, /subpage/subpage/subpage/somefile.php is bad). Because if the crawler has already processed the whole page and is only locked out at the end after it has done its work, the whole thing no longer makes sense. Alternatively you can do a redirect via JavaScript. Option one would be the "display:none" attribute on a link mentioned above.

<a href='target.php' style='display:none'>

Option two would be JavaScript, which however requires the crawler to have JavaScript disabled (and note: as soon as a regular user doesn't have JavaScript enabled, they get locked out too. So this is more food for thought than a serious solution).

<noscript>
  <iframe src="target.php"></iframe>
</noscript>

Surely this can be extended further. If you have ideas, feel free to drop them in the comments.

Step 6: Wrapping up

Finally we upload all files, prepare the database and, in the best case, have already replaced the values "Username" and "Password" in insert.php with the credentials of our database ;-)

Addendum: using an ad blocker

If you want to go one step further, you can use the ad blocker that nearly every user has installed these days but hardly any crawler does. We simply rename the file target.php to subtarget.php and write the following iframe into target.php:

<iframe src="subtarget.php" id="advert"></iframe>

For a "normal" user the ad blocker should now block the resource, while for a bot the iframe is loaded and the IP address is written to the database. Instead of setting the id "advert", we can also simply name subtarget.php advertising.php - that should already activate Adblock.