Referer Spammers

The Internets are full of spam. Maybe you’ve noticed?

It’s in your inbox, in your comments and scattered throughout your web forums. Every spammer is a bag of dicks but the worst bottom feeder on the Internets is the referer spammer.

If you’ve never administered a website, then you’ve probably never heard of referer spam. Yeah, what is that?  Glad you asked. These dregs send requests to your web site with a fabricated referer that points to a site they want to advertise. Ideally, they’ll send requests to a site that publishes its traffic reports. When their URL makes the report, they get a free link back to their site.

Sites that publish their usage reports are easy to find. Put this in the Google Machine and see what pops up: “Top * Total Search Strings” This is what we’re looking for: Usage Stats: Top Referers.  Your JoeDog can get himself on that report by doing this:

Bully $ siege -H "Referer: http://www.joedog.org/" -g http://www.pickart.at/
HEAD / HTTP/1.0
Host: www.pickart.at
Accept: */*
User-Agent: Mozilla/5.0 (unknown-x86_64-linux-gnu) Siege/3.0.8
Referer: http://www.joedog.org/
Connection: close
HTTP/1.1 200 OK
Date: Fri, 03 Oct 2014 17:53:38 GMT
Server: Apache
Connection: close
Content-Type: text/html

Now if he’s really intent on making that report, he’ll repeat that request a few hundred times and place himself at number two on the chart. But here’s the thing: Referer Spammers will spam your logs even if you don’t publish your reports. They’ll go to all that trouble just to lure webmasters to their esoteric fetish sites.

So what can you do to prevent this stuff? Mostly you can decrease their incentive.

  1. Put your usage stats inside a password protected area
  2. Add a robots.txt with a bot exclusion rule so search engines don’t index it.
  3. Add a nofollow directive inside every link, again so engines don’t index them

I guarantee you’ll still get the stuff. They’ll send faked referrals just to capture the attention of the site’s administrators but at least you won’t award them with a boost to their Page Rank.

NOTE: Yes, Your JoeDog spelled Referrer with only two r’s. Most humans use three. Phillip Hallam-Baker is not most humans. He was the first guy to miss an ‘r’ in the original HTTP specification. I say, “first guy” because hundreds of eyeballs viewed that document and none of them noticed the misspelling. By the time it became RFC1945, “Referer” was set in stone. It would have been easier to change the world’s English-language dictionaries at that point….

Posted in Apache, Applications, Security | Leave a comment



Invalid command ‘TypesConfig’

Ah but the joys of trying to match the missing module with its obtuse apache error. In this case, we tried to use the TypesConfig directive but the module wasn’t loaded at runtime. Here’s the error:

# service httpd configtest
Syntax error on line 107 of /etc/httpd/conf/httpd.conf:
Invalid command 'TypesConfig', perhaps misspelled or defined by a module
not included in the server configuration

In this case, we were missing the mime module. You can add that module in your httpd.conf file with the following directive:

LoadModule mime_module modules/mod_mime.so

Happy apaching!

Posted in Apache, Applications | Leave a comment



Invalid command ‘order’

It would be nice if apache told you which module you were missing. Fortunately, there’s the Internets! Hey, this site is on the Internets let’s see if we can help. I just ran ‘service httpd checkconfig’ and received the following error:

# service httpd configtest
Syntax error on line 92 of /etc/httpd/conf/httpd.conf:
Invalid command 'Order', perhaps misspelled or defined by a module 
not included in the server configuration

After a brute force attempt at adding modules, it became clear that I was missing the following module: authz_host_module. I added that in httpd.conf with the following directive:

LoadModule authz_host_module modules/mod_authz_host.so

You can also compile that module into the binary with the following flag: –enable-authz-host  (in most cases that’s compiled by default but I’m using RedHate’s binary so it was necessary to add it at run time).

Posted in Apache, Applications | Leave a comment



Recent Comments

  • Tim: For those who enjoy playing at home and are extra OCD … they’ll spot something wrong with this....
  • roshni: Hi jeff, I need your help regarding running urls in a file containing post directives. Could you please send...
  • Alle: In seige, what does the pink result mean?
  • Windows User: Nice collection of Perl modules. Thanks for sharing.
  • Jeff Fulmer: No idea. What do you see in the webserver’s logs?