up arrow Counting Downloads With Fido

I wanted to illustrate how to use fido with an example. Today we’re going to use it to count software downloads on this site. Exciting! This will be simple since we only have one data source. A few years ago, I move my software from an FTP repository onto this web server. To quantify software downloads, we can simply monitor the http access log.

Here’s our fido configuration for the log file:

/var/log/httpd/access_log {
 rules  = downloads.conf
 action = /usr/local/bin/tally
 log    = syslog
}

This tells fido to monitor the access_log in real time. Its pattern match rules are in a file called downloads.conf When fido finds a match, it will execute a program called tally. Finally, the last directive tells fido to use syslog to log its activity.

In order to understand what we’re looking for, you should take a look at the software repository. It contains multiple versions and helpful links to the latest releases and betas. We want to match them all.

Let’s take a look at our downloads.conf file. Since we didn’t specify a full path to the file, fido knows to look for it under $sysconfdir/etc/fido/rules. If you configured it to use /etc, then the rules are found in /etc/fido/rules/downloads.conf. Here’s the file:

#
# Track and count downloads from the website
SIEGE:  .*siege-.*tar.gz.*
FIDO:   .*fido-.*\([rpm]|[tar\.gz]\).*
WACKY:  .*wackyd-.*tar.gz.*
DICK:   .*dick.*tar.gz.*
SPROXY: .*sproxy-.*tar.gz.*
CONFIG: .*JoeDog-Config.*
STATS:  .*JoeDog-Stats.*
GETOPT: .*php-getopt.*
WACKY:  .*JoeDog-Wacky.*
PBAR:   .*JoeDog-ProgressBar.*

Each line begins with an optional label. If a label is present, fido will pass it (minus the colon) to the action program. In the example above, if the JoeDog-Config perl module is downloaded, then fido will run /usr/local/bin/tally CONFIG. For more on labels, see the fido user’s manual.

tally is a perl script that does our heavy lifting. It uses Tie::File to map an array with the contents of a file. When it’s invoked, it searches the map for the label at $ARGV[0] then increments its value. Here’s the script:

#!/usr/bin/perl
use strict;
use warnings;
use Tie::File;
my $file = "/tmp/downloads.txt";
my $type = $ARGV[0] or die "usage: $0 <identifier>";
sub trim($);
if (! -e $file) {
  open (FILE, ">$file");
  close(FILE);
}
tie my @array, 'Tie::File', $file or die;
my $found = 0;
for my $line (@array) {
  my ($name,$downloads) = split(/\|/,$line);
  $downloads += 1;
  if ($name =~ /$ARGV[0]/) {
    $name = trim($name);
    $line = sprintf("%-7s | %5s \n", $name, $downloads);
    print "$line";
    $found = 1;
  }
}
if (! $found) {
  push(@array, $ARGV[0] . " | 1")
}
untie @array;
sub trim($) {
  my $thing = shift;
  $thing =~ s/#.*$//; # trim trailing comments
  $thing =~ s/^\s+//; # trim leading whitespace
  $thing =~ s/\s+$//; # trim trailing whitespace
  return $thing;
}

I could have just as easily inserted those values in a database and used that information to publish on this site.  It was designed to be flexible. In some cases, I have fido daemons that send traps to OpenView, in others I have it send pages. What you do on the event of a match is limited to your imagination.

So what was the result of this tabulation? I set this example up five months ago and here’s the result:

$ cat downloads.txt
SIEGE   | 20406 
STATS   |   151 
GETOPT  |    63 
SPROXY  |  2262 
DICK    |   146 
CONFIG  |   165 
PBAR    |   102 
FIDO    |  1177 
WACKY   |   160 
PWACKY  |   172

Siege is the clear “winner” followed by sproxy which is a siege utility that helps build a urls.txt file. I was disappointed that JoeDog::Config received so little love. I use it all the time.

Happy hacking.



Posted in Applications, Fido | Leave a comment

Leave a Reply




Recent Comments

  • Mike Smith: I find that Dunning-Kruger explains a lot. The difficult part is getting help to those that suffer from...
  • Sohan: I am using node-js http server. I created http request to hit the server and log the message. In that case...
  • Jeff Fulmer: I don’t know what “simple http server” means. If you’re using apache out of the...
  • Sohan: MY server is simple http server. Can you tell me what thresol paramter or property is used in siegerc file.
  • Jeff Fulmer: Ninety-five percent of the time, this error occurs because you’ve configured more simulated users...