JoeDog | Applications

Siege 3.0.4 Becomes Part of the Problem

By jdfulmer | Published September 17, 2013 | Leave a comment

Siege 3.0.4 was just released. It contains a feature that I’ve added with a certain amount of reluctance. To understand the feature and the reason for my trepidation, let’s visit RFC 2616 and read what it has to say about Location headers:

For 3xx responses, the location SHOULD indicate the server's
preferred URI for automatic redirection to the resource. The 
field value consists of a single absolute URI.

    Location = "Location" ":" absoluteURI

An example is:

    Location: http://www.w3.org/pub/WWW/People.html

That’s pretty clear, right? The value of a location header must be an absolute URI. Yet a large number of developers ignore that directive. Here’s the response from a server running SquirrelMail, a popular web-based email program:

     HTTP/1.1 302 Found
     Date: Tue, 17 Sep 2013 16:50:52 GMT
     Server: CERN/1.0A
     X-Powered-By: PHP/5.2.5
     Location: src/login.php
     Content-Length: 0
     Connection: close
     Content-Type: text/html; charset=WINDOWS-1251

Although that Location header violates RFC 2616, nearly every web client will follow it to SquirrelMail’s intended destination. I say “nearly every client.” Until version 3.0.4, siege wouldn’t have followed it any where. It would have scratched its head and said, “Fsck it. Next URL.”

It is with some reluctance that I’ve included siege in the community of clients that allow developers to circumvent established standards. This convention has created a slew of bad coding practices on the world wide web. Didn’t close a table with an end tag? That’s okay, M$ will close it for you. Used a relative URI in a Location header? Don’t worry, siege will normalize it for you.

Ironically, version 3.0.4 includes one other feature enhancement. Its default User-agent is now in full compliance with RFC 2616. You win some, you lose some. And so it goes….

Siege 3.0.3 and URL Encoding

By jdfulmer | Published July 23, 2013 | Leave a comment

URL encoding aka URL escaping aka percent encoding is a mechanism for converting URL characters into a format that can be transmitted by HTTP. Reserved characters are replaced by a hexadecimal value preceded with a ‘%’ which is an escape character. If a URL contains a space, for example, it must be encoded for transmission. Your browser takes a space and reformats it as %20.

Siege, on the other hand, does nothing. It expects you to encode your own damn URLs … that is, until now! Percent encoding is available in siege starting with version 3.0.3-beta2. When it emerges from beta, the first stable version to support this feature will be 3.0.3.

Really? Siege has been around since 1999 and you’re only now adding this feature?

Well, you guys never asked and I haven’t had much need for it. Lately, however, I’ve noticed many of you are asking about json. I suspect URL escaping will be helpful to those folks. Consider this:

siege -g ‘http://www.joedog.org/siege/echo.php?q={ “Hello” : “world” }’

 GET /siege/echo.php?q=%7B%20%22Hello%22%20:%20%22world%22%20%7D HTTP/1.0
 Host: www.joedog.org
 Accept: */*
 User-Agent: JoeDog/1.00 [en] (X11; I; Siege 3.0.3-beta2)
 Connection: close

Booya! Just make sure you single quote the URL like in the example above.

Since URL escaping is in its early stages, I’ve provided a mechanism for disabling it. Inside $HOME/.siegerc add the following: url-escaping = false The default value is true.

Anything else in version 3.0.3-beta2 that we should know about?

Why yes! We changed behavior for -g/–get. When you retrieve a page using -g/–get, siege sets its protocol to HTTP/1.0 so the page is human readable. We don’t need to read chunked encodings and neither do you.

H/T: Your JoeDog would like to give a shout out to the folks at wget from whom he completely stole most of the code necessary to implement URL escaping. Cheers.

How To Stop A WordPress Dictionary Attack

By jdfulmer | Published June 18, 2013 | Leave a comment

You guys! Lest we forget, Your Joe Dog was under attack!

Apparently there’s a widespread dictionary attack that uses tens of thousands of malwared computers to attack WordPress sites. Your JoeDog uses WordPress as a CMS. Your JoeDog was attacked!

The extent of the attack was not initially clear. I was alerted by sluggish performance. I noticed a lot of POSTs to wp-login.php. Those POSTs appeared in the access log like this:

92.47.65.37 - - [17/Jun/2013:09:06:42 -0400] "POST /wp-login.php HTTP/1.0" 
200 3444 "-" "Mozilla/5.0 (Windows NT 6.1; rv:19.0) Gecko/2010 Firefox/19.0"

I have a script that allows me to quickly block IP addresses with iptables. So I started harvesting addresses and blocking them. Done and done.

Except the attacker seemed to have an endless supply of IP addresses. The attack persisted no matter how many addresses I blocked.

Take a look at the log entry above. The referer field is empty. A JoeDog Fellow suggested I block all POSTs that don’t include a referer. Afterall, you don’t POST out of the blue – you submit a form in your browser. I blocked those types of requests with a simple mod_rewrite rule:

 RewriteCond %{REQUEST_METHOD} =POST
 RewriteCond %{HTTP_REFERER} ^-?$
 RewriteRule ^/(wp-login.php|wp-admin) - [F,NS,L]

Done and done. Amiright? Sadly, no….

Continue reading How To Stop A WordPress Dictionary Attack

Your Joe Dog Is Under Attack

By jdfulmer | Published June 17, 2013 | Leave a comment

This site has been under attack for several weeks now. The attacker is using an unthrottled brute force dictionary attack. He seems to have an unlimited supply of IP addresses. After examining some of the source addresses, I’ve concluded that we’re on the wrong end of a bot net.

I’ve been harvesting IP addresses and blocking them as fast as I can. I’ve also added Rewrite rules to deny these attempts based on his request signature. Those rules reduce overhead since his requests won’t generate database transactions. Yet no matter how many timeouts he gets and no matter how many Access Denied responses he endures, the attacks persist.

Because this dictionary attack is unthrottled, the affect is, at times, not unlike a DOS attack. Your Joe Dog is a public service with shallow pockets. We simply don’t have the resources to eat these attacks and provide you with snappy service. Bare with us as we deal with this asshole.

UPDATE: While it provides additional inconvenience, I applied an access control to the page he’s attacking. You can protect a single file inside a FilesMatch block like this:

 <FilesMatch "wp-login.php">
   AuthType Basic
   AuthName "Kiss my fscking ass"
   AuthUserFile /path/to/my/file
   Require user franklindelanoroosevelt
 </FilesMatch>

Obviously, some of that information was obfuscated but “Kiss my fscking ass” really is the realm I’m using.

The benefit to this approach is two-fold: 1. Apache doesn’t expend much effort to say, “401 gimme a password!” 2. If this layer is cracked, he still has to bust the next one before I reset the password on the first one….

Siege and the Single Cookie

By jdfulmer | Published May 8, 2013 | Leave a comment

An emailer wondered if siege could be configured to refuse cookies. I had a vague recollection on the matter. My brain cells were all, “Yeah, sure, you can disable cookies.”

The best place to look for more esoteric features is inside $HOME/.siegerc If you don’t have one, you can generate a new one with this command: siege.config That command will place a new one inside your home directory. ‘siege.config’ is designed to build a resource file which is compatible with your version of the program.

I checked the latest version of that file and I found nothing to disable cookies. Really? That seems like a no-brainer. Why wouldn’t we allow users to disable that? “Sorry,” I replied. “Siege can’t disable cookies.”

Yet my own response was bothersome. I was certain you could turn that off so I checked the code with ‘egrep cooki *.c’ Sure enough, siege parses siegerc for ‘cookie’ which accepts true or false. The latter disables cookie support.

Documentation for this feature has been added to siege-3.0.1-beta4. The feature itself is available in just about every contemporary version of siege that’s floating around the Internets. Just add ‘cookies = false’ to your siegerc file to disable their support.

We’re Number One!

By jdfulmer | Published December 12, 2012 | Leave a comment

nixCraft selected siege as the Number One Greatest Open Source Terminal Application of 2012. We’re not sure what that means other than AWESOME! It’s been a long road to 2012 greatness. The project began in late 1999 and was released into the wild in early 2000. Since then, it has been developed shaped by countless people from all corners of the globe.

Even though siege is a very esoteric program, its source code is downloaded around 50,000 times a year. Binaries are distributed by most major Linux vendors. One day a co-worker wanted to run a test from a new server so he asked if I could get him some binaries. Before I could react to the request, he said, “Nevermind.” He ran ‘apt-get install siege’ and got it from the vendor. That was a pretty cool moment for reasons other than the fact that it saved me some work.

Let’s be clear. This isn’t my recognition. It’s our recognition. For twelve years siege has been a community project and its direction will continue to be shaped by the people who use it. “Now let’s eat a god damn snack.” — Rex Ryan to the siege community.

SSH – Disable known_hosts Prompt

By jdfulmer | Published July 13, 2012 | 2 Comments

Do you have scripts that do remote procedures over ssh? Do host key checks occasionally cause them to break? This entry will show you how to avoid that mess and keep your scripts running smoothly.

BACKGROUND

ssh protocol is designed to verify the host key against a local file to ensure the integrity of the remote server. By default, the user is prompted to accept a new key or warned when the host key changes (like after a server upgrade). This is a nice defense against man-in-the-middle attacks, but it plays havoc on scripts. If a prompt occurs, your script stops and waits for input.

FIX

There are two ways you can avoid this problem. You can pass parameters to ssh or you can change the system setting in ssh_config. If you want to turn off host key checks for scripting, then we recommend using command line parameters. You only type them once when you write your script and they only affect that instance of ssh.

By default, StrictHostKeyChecking is set to ‘ask.’ That’s why you’re prompted to accept a key. In order to avoid the prompt, you can change that to ‘no.’ When it’s set to ‘no’ the key is stored with no questions asked.

Unfortunatley, that’s not as clean as it seems. If the host signature changes due to an upgrade, then ssh stores that key, too. Since you have two, it starts throwing warnings such as this:

key_read: uudecode AAAAAAAB3NzaC1yc2EAAAABIwAAAQEAzh1G5NiiEfawhBhly
VLR92Q/+iXZ3Bs56RBLZtso/lEFk9TYZuS+Qp+tKOIv1j5HpuwsoIAZt6A1fJfCHfN3
KYtuWNbdMywuoOUb5Z9S0c/3jyeesy2eTy+ZZjgb0uPdU8cCKg029NF9gQr5tbDlrj+
vW6QvvWJ0KVJFJPWg6u3/Qt/N/xlPXziyHv4HKuzMDoRLQ5ltiC8zk3ZefeRK7ZZKtp
qSneTsHZt7alOGOsKTrPL5PA50QwBiNJFbvrnmJs2Xjk3x6MunXFuRSZCEsGboQWDie
whcOFxDlkYfWjHNbShPYBY3xuq/MnsL8QHUx9AT75wpl2U0/KFbXsMAKw==
 failed
key_read: uudecode AAAAAAAB3NzaC1yc2EAAAABIwAAAQEAzh1G5NiiEfawhBhly
VLR92Q/+iXZ3Bs56RBLZtso/lEFk9TYZuS+Qp+tKOIv1j5HpuwsoIAZt6A1fJfCHfN3
KYtuWNbdMywuoOUb5Z9S0c/3jyeesy2eTy+ZZjgb0uPdU8cCKg029NF9gQr5tbDlrj+
vW6QvvWJ0KVJFJPWg6u3/Qt/N/xlPXziyHv4HKuzMDoRLQ5ltiC8zk3ZefeRK7ZZKtp
qSneTsHZt7alOGOsKTrPL5PA50QwBiNJFbvrnmJs2Xjk3x6MunXFuRSZCEsGboQWDie
whcOFxDlkYfWjHNbShPYBY3xuq/MnsL8QHUx9AT75wpl2U0/KFbXsMAKw==
 failed
Last login: Fri Jul 13 11:09:36 2012 from jdfulmer-lt.joedog.org
RedHat 5Server - LinuxCOE 4.2 Fri May 11 09:48:33 EDT 2012

We can avoid this mess with another setting. Instead of saving host key entries to known_hosts, we can bury them in /dev/null. We can change the file location with the UserKnownHostsFile parameter. If we change it to /dev/null there are no entries for ssh to read. And when it writes a new entry, well it goes to /dev/null

IMPLEMENTATION

There are two ways we can implement this. One is at the script level and the other is at the system level. If we want to continue to prompt for host key checks, then we can add the configuration to our script. This can be done with OpenSSH’s -o option. Here’s an example in which we run the hostname command on a remote server:

ssh -o StrictHostKeyChecking=no 
    -o UserKnownHostsFile=/dev/null 
       user@host /usr/bin/hostname -s

To set this configuration system-wide, place these entries in ssh_config:

StrictHostKeyChecking no 
UserKnownHostsFile /dev/null
LogLevel QUIET

NOTE: This configuration applies only to OUTBOUND ssh connections. This does not affect your system’s inbound ssh traffic.

UPDATE: I added LogLevel QUIET to the ssh_config above. This is the same as running ssh with a “-q”. This suppresses all warning messages which may wreak havoc on your scripts.

Mondoarchive Exclude List Failures

By jdfulmer | Published June 21, 2012 | Leave a comment

To illustrate config files in sh scripts, I published my mondoarchive script. That script dynamically builds an mondo exclude list from a list of directories inside a file.

Since I published that article, many of you have arrived here after Googling mondoarchive exclude lists. It seems they’re failing you. Fear not, faithful Googlers. Your JoeDog has experienced this pain and he can help.

There are two main problems with mondoarchive exclude lists that causes the program to ignore them. One is documentation and the other is a bug.

Older versions of mondoarchive use space separated exclude lists. You construct them like this:

  -E "/usr/src /data/archive /usr/local/src"

Since version 2.2.9.5 the syntax has changed for both -E and -I. Whereas older versions used space separated lists of directories, newer versions use pipe separated directories. If you have a newer version, construct your lists like this:

  -E "/usr/src|/data/archive|/usr/local/src"

The other problem I’ve encountered appears to be a bug. The first directory in my exclude list wasn’t being excluded. To fix that problem, I’ve placed /tmp first in all my exclude lists.

  -E "/tmp|/usr/src|/data/archive|/usr/local/src"

Problem “solved.”

Creating Config Files For sh Scripts

By jdfulmer | Published June 8, 2012 | 1 Comment

Your JoeDog uses mondorescue for bare-metal Linux restoration. We use mondorestore to recover the OS and Net Backup to recover its content. Since we’re only concerned about archiving the OS for bare-metal recovery, it’s necessary to exclude directories when we run mondoarchive.

My exclude requirement varies from server to server so I wanted to build the list dynamically. As a coder, I have religious aversion to altering scripts for the purpose of configuring them. If we set config variables inside the script, then we have a different version on every server. That’s a paddlin’.

For my mondoarchive script, I developed a pretty slick way to read a configuration file and build an exclude list. The list is configured in a conf file that ignores comment lines and superfluous white space. A typical configuration looks like this:

#
# This file is maintained by the Puppet Master 
# 
# This is the exclude list for mondoarchive Directories inside
# this list will not be archived for bare metal recovery.
#
/tmp
/export
/usr/src
/var/mail
/var/cache
/var/log

My mondoarchive script builds a string of pipe separated directories like this:

/tmp|/export|/usr/src|/var/mail|/var/cache|/var/log

Since very few of you will have a similar usecase, I wrote an example that reads the file into a sh array. This version will loop through the array and print each one.

#!/bin/sh
# An example script that reads a list from a config
# file into a sh script array.
CONF="haha.conf"
LIST=""
#
# Read the directory list from $CONF
if [[ -e $CONF ]] ; then
  while read line ; do
    chr=${line:0:1}
    # XXX: Use awk's substr on older systems like
    # HPUX which don't support the above syntax.
    # chr=$(echo $line | awk '{print substr($1,0,1)}')
    case $chr in
     '#')
       # ignore comments
       ;;
     *)
       if [[ ${#line} -gt 2 ]] ; then
         if [[ -z $LIST ]] ; then
           LIST="$line"
         else
         LIST="$LIST $line"
         fi
       fi
       ;;
    esac
  done < $CONF
else
  echo "$0: [error] unable to locate $CONF"
fi

let X=1
for I in $LIST ; do
  echo "$X: $I"
  let X=$X+1
done

Let’s run this bad boy and see what happens:

$ sh haha
1: /tmp
2: /etc
3: /usr/local
4: /data/mrepo

If some of the concepts listed don’t make sense, then you might want to see our sh scripting cheat sheet. It will help you understand things like ‘-e $CONF’ and sh script arrays. Happy hacking.

UPDATE: Given the introduction to this post, it’s likely that many of you have arrived here in search of a mondoarchive backup script. Well, we won’t let you leave empty handed. You can grab my archive script here: Mondo Rescue Archive Script

This script builds both NFS recoverable archives and DVD images to an NFS mounted volume. Here’s its usage banner:

Usage: archiver [-c|-n]
Requires either a '-c' or a '-n' argument
  -c      create a CD Rom archive
  -n      create an NFS archive

Is There An AJP Functional Test?

By jdfulmer | Published June 6, 2012 | 2 Comments

There are plenty of helpful tools to test network services. If you want to check HTTP functionality, you could craft a request with curl, wget or “siege -g” to see if a server is functioning. If you understand the service protocol, you can always telnet to a TCP port and type a transaction.

Unfortunately, there aren’t many tools to help you test AJP protocol. Sure, you can telnet to the port to ensure it’s running, but how many people know how to craft an AJP transaction? I didn’t.

In order to help you test AJP servers like Apache’s tomcat, I wrote ajping. It connects to a user-define port and conducts a simple transaction. ajping validates the server’s response and clocks the length of the transaction. Over the LAN, you should expect times in the hundreds of seconds. This is a command line utility. In order to install it, run the following commands:

 $ wget http://download.joedog.org/AJP/ajping.txt
 $ mv ajping.txt ajping
 $ chmod +x ajping

You can test a server with it like this:

LT $ ajping tommy.joedog.org:8009
Reply from tommy.joedog.org: 7 bytes in 0.019 seconds
Reply from tommy.joedog.org: 7 bytes in 0.004 seconds
Reply from tommy.joedog.org: 7 bytes in 0.004 seconds
Reply from tommy.joedog.org: 7 bytes in 0.011 seconds
Reply from tommy.joedog.org: 7 bytes in 0.004 seconds
Reply from tommy.joedog.org: 7 bytes in 0.016 seconds
Reply from tommy.joedog.org: 7 bytes in 0.009 seconds
Reply from tommy.joedog.org: 7 bytes in 0.021 seconds
Reply from tommy.joedog.org: 7 bytes in 0.011 seconds
Reply from tommy.joedog.org: 7 bytes in 0.025 seconds

I’ve also incorporated this code into a check_ajp script for Zenoss. Remove the .txt extension and install it on Zenoss as you would any other script. Happy hacking.

UPDATE: I fixed the links to point to the new download location. H/T paalfe

Joe Dog Software