Fido Learns A New Trick

I use Mondoarchive to create Linux recovery disks. Each server writes ISO images to a shared volume on a weekly basis. If any file inside that directory is older than seven days, then a server failed to create an ISO. In order to monitor this directory for failure, I added a new feature to fido. Exciting!

Starting with version 1.1.0 (click to download), fido can monitor a file or directory to see if it — or any file inside it — is older than a user configurable period of time. If fido discovers a file whose modification date exceeds the configured time, it fires an alert.

The following example illustrates how to configure the use case above:

/export {
  rules = exceeds 8 days
  exclude = ^\.|^lccns178$|^lccns179$|^lccns335$|lccns336$
  throttle = 12 hours
  action = /etc/fido/notify.sh
}

This file block applies to “/export” which is a directory. Since it’s a directory, the rules apply to every file inside it. In this case ‘rules’ is pretty straight forward. We’re looking for files that exceed eight days in age. This rule will always follow this format: exceeds [int] [modifier]. The modifier can be seconds, minutes, hours or days. If you take the long view — if you’re concerned about events far into the future — then you’ll have to do some math. We don’t designate years so you’ll have to use 1825 days if you want to be alerted five years out.

We also find a new feature inside this block. ‘exclude’ takes a regular expression and tells fido which files to ignore. Currently, ‘exclude’ only works inside a file block with an exceeds rule but I plan to make better use of it.

Finally we notice one final feature that we’ve never seen before. The ‘throttle’ directive tells fido how long to wait between alerts. In this scenario, fido will trigger an alert the second it finds a file which exceeds 8 days. If the problem is not addressed within twelve hours, it will fire another alert. Alerts will continue in twelve hour intervals until the problem is corrected.

I hope you enjoy these features. If there are enhancements you’d like to see, feel free to contact me either in the comments or by email.

Posted in Applications, Fido | Leave a comment



Use Fido To Process FTP Uploads

Did you ever want to process a file immediately after it was uploaded via FTP? You could have the upload script execute a remote command after the file is uploaded. That requires shell access that you may or may not be able to grant. On the server, you could run a processing script every minute out of cron but that could get messy.

Fido provides alternative method.

Starting with version 1.0.7, Fido has the ability to monitor a file or directory by its modification date. When the date changes, fido launches a script. We can use this feature to process files that are uploaded via ftp.

In this example, we’ll monitor a directory. In fido.conf, we’ll set up a file block that points to a directory. (For more information about configuring fido, see the user’s manual). This is our configuration:

/home/jdfulmer/incoming {
 rules = modified
 action = /home/jdfulmer/bin/process
 log = /home/jdfulmer/var/log/fido.log
}

With this configuration, fido will continuously watch /home/jdfulmer/incoming for a modification change. When a file is upload, the date will change and fido will launch /home/jdfulmer/bin/process. Pretty sweet, huh?

Not quite. The modification date will change the second ftp lays down the first bite. Our script would start to process the file before it’s fully uploaded. How do we get around that? We can make our script smarter.

For the purpose of this exercise, I’m just going to move uploaded files from incoming to my home directory. Here’s a script that will do that:

#!/bin/sh
PREFIX="/home/jdfulmer/incoming"
FILES=$(ls $PREFIX)
for F in $FILES ; do
  while [ -n "$(lsof | grep $F)" ] ; do
    sleep 1
  done
  mv $PREFIX/$F /home/jdfulmer
done

In order to ensure the file is fully uploaded, I check lsof for its name. If there’s an open file handle under that name, then the script will continue to loop until it’s cleared. When the while loop breaks, the script moves the file.

There’s just one more thing to think about. When the script moves the file what happens to the directory fido is watching? Yes. Its modification date changes. In my example, process runs a second time but does nothing since nothing is there. Depending on your situation, you may need to make the script a little smarter.

Posted in Applications, Fido, sh | Leave a comment



Counting Downloads With Fido

I wanted to illustrate how to use fido with an example. Today we’re going to use it to count software downloads on this site. Exciting! This will be simple since we only have one data source. A few years ago, I move my software from an FTP repository onto this web server. To quantify software downloads, we can simply monitor the http access log.

Here’s our fido configuration for the log file:

/var/log/httpd/access_log {
 rules  = downloads.conf
 action = /usr/local/bin/tally
 log    = syslog
}

This tells fido to monitor the access_log in real time. Its pattern match rules are in a file called downloads.conf When fido finds a match, it will execute a program called tally. Finally, the last directive tells fido to use syslog to log its activity.

In order to understand what we’re looking for, you should take a look at the software repository. It contains multiple versions and helpful links to the latest releases and betas. We want to match them all.

Let’s take a look at our downloads.conf file. Since we didn’t specify a full path to the file, fido knows to look for it under $sysconfdir/etc/fido/rules. If you configured it to use /etc, then the rules are found in /etc/fido/rules/downloads.conf. Here’s the file:

#
# Track and count downloads from the website
SIEGE:  .*siege-.*tar.gz.*
FIDO:   .*fido-.*\([rpm]|[tar\.gz]\).*
WACKY:  .*wackyd-.*tar.gz.*
DICK:   .*dick.*tar.gz.*
SPROXY: .*sproxy-.*tar.gz.*
CONFIG: .*JoeDog-Config.*
STATS:  .*JoeDog-Stats.*
GETOPT: .*php-getopt.*
WACKY:  .*JoeDog-Wacky.*
PBAR:   .*JoeDog-ProgressBar.*

Each line begins with an optional label. If a label is present, fido will pass it (minus the colon) to the action program. In the example above, if the JoeDog-Config perl module is downloaded, then fido will run /usr/local/bin/tally CONFIG. For more on labels, see the fido user’s manual.

Continue reading »

Posted in Applications, Fido | Leave a comment



Recent Comments

  • berrybenkashop.zendesk.com: regalo double bed rail
  • www.youtube.com: Hi! I know this is kind of off-topic but I had to ask. Does building a well-established website like...
  • http://www.amazon.com/: Hello, I think your website might be having browser compatibility issues. When I look at your...
  • like: Hurrah! At last I got a blog from where I can really obtain valuable facts concerning my study and knowledge.
  • faafshclesks: Ecco Anne Hathaway riprese di The Intern a Manhattan, e che