Fork me on Github
Fork me on Github

Joe Dog Software

Proudly serving the Internets since 1999

Mondoarchive Exclude List Failures

To illustrate config files in sh scripts, I published my mondoarchive script. That script dynamically builds an mondo exclude list from a list of directories inside a file.

Since I published that article, many of you have arrived here after Googling mondoarchive exclude lists. It seems they’re failing you. Fear not, faithful Googlers. Your JoeDog has experienced this pain and he can help.

There are two main problems with mondoarchive exclude lists that causes the program to ignore them. One is documentation and the other is a bug.

Older versions of mondoarchive use space separated exclude lists. You construct them like this:

  -E "/usr/src /data/archive /usr/local/src"

Since version 2.2.9.5 the syntax has changed for both -E and -I. Whereas older versions used space separated lists of directories, newer versions use pipe separated directories. If you have a newer version, construct your lists like this:

  -E "/usr/src|/data/archive|/usr/local/src"

The other problem I’ve encountered appears to be a bug. The first directory in my exclude list wasn’t being excluded. To fix that problem, I’ve placed /tmp first in all my exclude lists.

  -E "/tmp|/usr/src|/data/archive|/usr/local/src"

Problem “solved.”



Creating Config Files For sh Scripts

Your JoeDog uses mondorescue for bare-metal Linux restoration. We use mondorestore to recover the OS and Net Backup to recover its content. Since we’re only concerned about archiving the OS for bare-metal recovery, it’s necessary to exclude directories when we run mondoarchive.

My exclude requirement varies from server to server so I wanted to build the list dynamically. As a coder, I have religious aversion to altering scripts for the purpose of configuring them. If we set config variables inside the script, then we have a different version on every server. That’s a paddlin’.

For my mondoarchive script, I developed a pretty slick way to read a configuration file and build an exclude list. The list is configured in a conf file that ignores comment lines and superfluous white space. A typical configuration looks like this:

#
# This file is maintained by the Puppet Master 
# 
# This is the exclude list for mondoarchive Directories inside
# this list will not be archived for bare metal recovery.
#
/tmp
/export
/usr/src
/var/mail
/var/cache
/var/log

My mondoarchive script builds a string of pipe separated directories like this:

/tmp|/export|/usr/src|/var/mail|/var/cache|/var/log

Since very few of you will have a similar usecase, I wrote an example that reads the file into a sh array. This version will loop through the array and print each one.

#!/bin/sh
# An example script that reads a list from a config
# file into a sh script array.
CONF="haha.conf"
LIST=""
#
# Read the directory list from $CONF
if [[ -e $CONF ]] ; then
  while read line ; do
    chr=${line:0:1}
    # XXX: Use awk's substr on older systems like
    # HPUX which don't support the above syntax.
    # chr=$(echo $line | awk '{print substr($1,0,1)}')
    case $chr in
     '#')
       # ignore comments
       ;;
     *)
       if [[ ${#line} -gt 2 ]] ; then
         if [[ -z $LIST ]] ; then
           LIST="$line"
         else
         LIST="$LIST $line"
         fi
       fi
       ;;
    esac
  done < $CONF
else
  echo "$0: [error] unable to locate $CONF"
fi
let X=1
for I in $LIST ; do
  echo "$X: $I"
  let X=$X+1
done

Let’s run this bad boy and see what happens:

$ sh haha
1: /tmp
2: /etc
3: /usr/local
4: /data/mrepo

If some of the concepts listed don’t make sense, then you might want to see our sh scripting cheat sheet. It will help you understand things like ‘-e $CONF’ and sh script arrays. Happy hacking.

UPDATE: Given the introduction to this post, it’s likely that many of you have arrived here in search of a mondoarchive backup script. Well, we won’t let you leave empty handed. You can grab my archive script here: Mondo Rescue Archive Script

This script builds both NFS recoverable archives and DVD images to an NFS mounted volume. Here’s its usage banner:

Usage: archiver [-c|-n]
Requires either a '-c' or a '-n' argument
  -c      create a CD Rom archive
  -n      create an NFS archive



Is There An AJP Functional Test?

There are plenty of helpful tools to test network services. If you want to check HTTP functionality, you could craft a request with curl, wget or “siege -g” to see if a server is functioning. If you understand the service protocol, you can always telnet to a TCP port and type a transaction.

Unfortunately, there aren’t many tools to help you test AJP protocol. Sure, you can telnet to the port to ensure it’s running, but how many people know how to craft an AJP transaction? I didn’t.

In order to help you test AJP servers like Apache’s tomcat, I wrote ajping. It connects to a user-define port and conducts a simple transaction. ajping validates the server’s response and clocks the length of the transaction. Over the LAN, you should expect times in the hundreds of seconds. This is a command line utility. In order to install it, run the following commands:

 $ wget http://download.joedog.org/AJP/ajping.txt
 $ mv ajping.txt ajping
 $ chmod +x ajping

You can test a server with it like this:

LT $ ajping tommy.joedog.org:8009
Reply from tommy.joedog.org: 7 bytes in 0.019 seconds
Reply from tommy.joedog.org: 7 bytes in 0.004 seconds
Reply from tommy.joedog.org: 7 bytes in 0.004 seconds
Reply from tommy.joedog.org: 7 bytes in 0.011 seconds
Reply from tommy.joedog.org: 7 bytes in 0.004 seconds
Reply from tommy.joedog.org: 7 bytes in 0.016 seconds
Reply from tommy.joedog.org: 7 bytes in 0.009 seconds
Reply from tommy.joedog.org: 7 bytes in 0.021 seconds
Reply from tommy.joedog.org: 7 bytes in 0.011 seconds
Reply from tommy.joedog.org: 7 bytes in 0.025 seconds

I’ve also incorporated this code into a check_ajp script for Zenoss. Remove the .txt extension and install it on Zenoss as you would any other script.  Happy hacking.

UPDATE: I fixed the links to point to the new download location. H/T paalfe



Use Fido To Process FTP Uploads

Did you ever want to process a file immediately after it was uploaded via FTP? You could have the upload script execute a remote command after the file is uploaded. That requires shell access that you may or may not be able to grant. On the server, you could run a processing script every minute out of cron but that could get messy.

Fido provides alternative method.

Starting with version 1.0.7, Fido has the ability to monitor a file or directory by its modification date. When the date changes, fido launches a script. We can use this feature to process files that are uploaded via ftp.

In this example, we’ll monitor a directory. In fido.conf, we’ll set up a file block that points to a directory. (For more information about configuring fido, see the user’s manual). This is our configuration:

/home/jdfulmer/incoming {
 rules = modified
 action = /home/jdfulmer/bin/process
 log = /home/jdfulmer/var/log/fido.log
}

With this configuration, fido will continuously watch /home/jdfulmer/incoming for a modification change. When a file is upload, the date will change and fido will launch /home/jdfulmer/bin/process. Pretty sweet, huh?

Not quite. The modification date will change the second ftp lays down the first bite. Our script would start to process the file before it’s fully uploaded. How do we get around that? We can make our script smarter.

For the purpose of this exercise, I’m just going to move uploaded files from incoming to my home directory. Here’s a script that will do that:

#!/bin/sh
PREFIX="/home/jdfulmer/incoming"
FILES=$(ls $PREFIX)
for F in $FILES ; do
  while [ -n "$(lsof | grep $F)" ] ; do
    sleep 1
  done
  mv $PREFIX/$F /home/jdfulmer
done

In order to ensure the file is fully uploaded, I check lsof for its name. If there’s an open file handle under that name, then the script will continue to loop until it’s cleared. When the while loop breaks, the script moves the file.

There’s just one more thing to think about. When the script moves the file what happens to the directory fido is watching? Yes. Its modification date changes. In my example, process runs a second time but does nothing since nothing is there. Depending on your situation, you may need to make the script a little smarter.