Blocking Referrer Spam

If you are tired of referrer spam filling up your logs with complete crap here is a very simple way to isolate the spam bots so you can later remove the information from your log files and then only pull out the correct information.

Somthing that I realized while reading another post is a spam bot trying to post garbage on your web site must have to ignore redirects. Otherwise people could do all sorts of horrible tricks like redirecting the spam bot to a tar pit or very large files or particullary nasty html pages aimed to crash them. Its probably worth pointing out that this only works for some of the spam bots out there. Not all of them.

This works in a really simple way. The majority of spam bot's since they are making request to post comments also attempt to post referrer spam along with their first get request to the web server. This can be exploited since it will not be followed.

The method to exploit this is simple. A browser with a real user will follow the redirect. So why not use the referrer url and the ip address the request is coming from to redirect the request to the same url again that the browser will follow. This will cause the spam bot to stop processing but allow normal users to continue on the path. Once a single user has made it past this barrier you have then identifiyed a valid referrer url and can skip the checking on other users. The browsers help with this somewhat because they will carry the correct referrer url across the redirect onto the 2nd request (the request after the redirect)

When processing the log files on the web server all that is now required to to remove all references to the redirects and the log files now have the spammer information removed.

I have also considered the impact on a few other thigns while writing this.

• Search engines don't provide a referrer references when making requests.
• People who disable the browser referrer also will not have a problem.

I put the following asp.net httpmodule together to exploit this weakness in the spammers bots. You will of course have to modify it to ignore your own site url

public class RefAntiSpam : IHttpModule
{
	//Use Cache To Form AN IP + Refferer to perform a redirect
	private Dictionary<string, DateTime> Cache = new Dictionary<string, DateTime>();

	public void Init(HttpApplication App)
	{
		App.BeginRequest += new EventHandler(App_BeginRequest);
	}

	public void App_BeginRequest(object sender, EventArgs e)
	{
		HttpRequest Request = HttpContext.Current.Request;
		HttpResponse Response = HttpContext.Current.Response;

		if (Request.HttpMethod == "GET")
		{
			if (Request.UrlReferrer != null)
			{
				if (Cache.ContainsKey(Request.UserHostAddress + "-" + Request.UrlReferrer.OriginalString) == false &&
					Request.UrlReferrer.OriginalString.Contains("stev.org") == false &&
					Request.UrlReferrer.OriginalString.Contains("localhost") == false)
				{
					Cache[Request.UserHostAddress + "-" + Request.UrlReferrer.OriginalString] = DateTime.Now;
					Response.Redirect(Request.Url.OriginalString, true);
				}
			}
		}
	}

	public void Dispose()
	{
		
	}
}