JoeDog | Programming

Dunning–Kruger Effect

By jdfulmer | Published September 30, 2014 | Leave a comment

Your JoeDog once worked with a programmer who couldn’t program. Now you’re probably thinking, isn’t programming an important qualification for that position? Not in a large corporation. To succeed in that environment, you need buzzwords and cliches. If you have them, managers just assume you know what you’re talking about.

This particular non-programmer — or Ouch! as we liked to call him — was hired to build a Intranet site. It took him a year and a half to construct something that looked like your eight-year old nephew slapped together in a weekend. It was slow, poorly marked-up but at least it had a confusing layout and design. Ouch had a parry for its shortcomings: Microsoft. “IE is a horrible web browser. It violates standards and ActiveX has a mind of its own.”

An appropriate response would have been, “If that’s true, how come all these non-Ouch sites look fine and work well in IE?” Instead, he received an award.

Because Ouch could steal someone else’s files and alter their markup to render the company’s text and images, we concede that he had some skill. Armed with a comprehensive understanding of his craft, Ouch would have also known: 1.) How to work around a browser’s weaknesses by 2.) Stealing the javascript, too, as it probably fixed those weaknesses but then he would have known too much and realized 3.) He was in the wrong profession.

While Ouch was laboring over his Intranet and ankle-deep in Cold Fusion, we were building an enterprise site with J2EE. And while Ouch didn’t know much, he did know this: in nerd hierarchy, Cold Fusion falls way below java.

So Ouch told everyone — and I mean everyone, his peers, his managers, the cleaning crew that he should be programming in java. To prove his point, he got the java logo tattooed on his bicep … which he showed to everyone.

Here’s the thing: Ouch wasn’t smart enough to know he couldn’t program in java. And management wasn’t smart enough to know he couldn’t program in java. The next thing you know, Ouch was stealing O’Reilly code — including the copyright notice — and attempting to implement the usecase. As far as I can tell, in one year in that position he didn’t release a thing that wasn’t immediately rewritten by somebody else.

Eventually Ouch was sacked but not for incompetence, he called his immediate supervisor the c-word. Management never considered him anything but a fine programmer. The buzzwords he used matched the ones they read in trade rags. How could he be anything but brilliant?

I didn’t realize it at the time but Ouch and the managers who considered him competent all suffered from the Dunning–Kruger Effect.

Shellshocked

By jdfulmer | Published September 30, 2014 | Leave a comment

Wired provides an interesting angle on the bash shell bug that has all your panties in a bind

[Brian] Fox drove those tapes to California and went back to work on Bash, other engineers started using the software and even helped build it. And as UNIX gave rise to GNU and Linux—the OS that drives so much of the modern internet—Bash found its way onto tens of thousands of machines. But somewhere along the way, in about 1992, one engineer typed a bug into the code. Last week, more then twenty years later, security researchers finally noticed this flaw in Fox’s ancient program. They called it Shellshock, and they warned it could allow hackers to wreak havoc on the modern internet.

[Wired: The Internet Is Broken]

How To Eliminate Unused Variable Warnings In GCC

By jdfulmer | Published September 23, 2014 | Leave a comment

Your JoeDog uses gcc v4.8.2 on his snazzy System 76 laptop. By default, that version uses this flag: -Wunused-but-set-variable If you’d like to see a lot of compiler warnings, then I suggest you use it, too.

Inside fido’s src directory you can find GNUs regex. The file names were changed to ereg.c, ereg.h and ereg2.h to avoid potential naming collisions. Well, sir, that .c file cries like a whiney two-year-old when it’s compiled with -Wunused-but-set-variable. Here’s what you’ll see in the console:

gcc -DHAVE_CONFIG_H -I. -I. -I../include -W -Wall -D_REENTRANT -D_POSIX_PTHREAD_SEMANTICS -g -O2 -c ereg.c
In file included from ereg.c:640:0:
ereg.c: In function ‘byte_re_match_2_internal’:
ereg.c:7145:27: warning: variable ‘sdummy’ set but not used [-Wunused-but-set-variable]
const CHAR_T *sdummy = NULL;
^
ereg.c:7144:22: warning: variable ‘pdummy’ set but not used [-Wunused-but-set-variable]
UCHAR_T *pdummy = NULL;

Let’s see the offending code, shall we?

UCHAR_T *pdummy = NULL;
const CHAR_T *sdummy = NULL;

POP_FAILURE_POINT (sdummy, pdummy, dummy_low_reg, dummy_high_reg, 
                   reg_dummy, reg_dummy, reg_info_dummy);

I don’t know about you, but Your JoeDog thinks those variables are pretty much used. POP_FAILURE_POINT is a macro but that shouldn’t matter. Since he doesn’t maintain this code, Your Nerdblogger decided to put a bandaid on the wound rather than stop its bleeding.

The Internets are loaded with bandaids! Here’s one from a discussion on this exact topic:

#define UNUSED(expr) do { (void)(expr); } while (0)

To silence that warning, pass the offending variables to the macro and “Presto!” no more bitching….

UNUSED(sdummy);
UNUSED(pdummy)

A Three-Column CSS Layout

By jdfulmer | Published September 6, 2014 | Leave a comment

Your JoeDog has a new look and feel!

The theme remains the same but we’re rocking a better architecture. The previous version was close to six years old and I made a poor design decision. What was that? I used HTML tables instead of CSS. It was a bad decision then but it’s inexcusable now. Don’t use HTML tables …. ever!

Why’s that?

CSS has numerous advantages over HTML.

It eliminates the need for a clipboard. If somebody refers to you as a copy-and-paste programmer, that’s not a compliment. Redundant code is hard to maintain. CSS allows you to manage pages with a single file.
HTML tables contain large amounts of markup. With CSS, you can reduce much of that, lower your site’s overhead and create faster loading pages.
Site redesign is so much more efficient with CSS because it decouples content from its presentation.
CSS is better for SEO because it boosts your content to code ratio and it better enables you to signal which items you want to emphasize.
And the biggest reason CSS is superior to HTML: NO SPACER.GIFs

Now Your JoeDog is no CSS design expert but he knows what he likes: a header and three columns — one for navigation, one for content and one for ads. You’re not blocking his ads, are you? They help pay the bills.

Fortunately, Matthew James Taylor is a CSS expert. To create this new design, Your JoeDog incorporated his Three-column Holy Grail. Taylor achieves cross platform compatibility and SEO friendliness without resorting to hacks, images or javascript. The design uses 2-1-3 column ordering. So while you’re viewing these words in column number two, search bots are viewing them in column number one. Clever!

I highly recommend his templates. If you’re having difficulty implementing them, feel free to steal code and ideas from this website. (In particular, look for the .main #content tag. Your JoeDog used that to prevent Taylor’s floating blocks from crushing his margins and padding.

sh Script: Read lines and omit comments and blanks

By jdfulmer | Published February 26, 2014 | Leave a comment

I perform remote operations on a series of servers. Rather than maintain the server list in several scripts, I consolidated it into a single file called servers.txt Exciting! The second I did that, I raised my own bar. You’d expect a config file parser to omit comments and blank lines, right? I do.

The anticipated time to write that parser was longer than I expected. In order to save you time, dear reader, I decided to post it here. This sh script reads a file into an array while skipping comment lines and blanks.

Here’s a sample config file:

#
# Comment line 
# Wed Feb 25 19:28:54 EST 2014
homer.joedog.org
 marge.joedog.org

bart.joedog.org
 lisa.joedog.org

burns.joedog.org
# EOF

And here’s a script to parse it:

#!/bin/sh

let X=0
while read line ; do
  if [[ ! $line =~ [^[:space:]] ]] ; then
    continue
  fi
  [ -z "`echo $line | grep '^#'`" ] || continue
  SRV[$X]=$line
  let X=$X+1
done < servers.txt

for (( i=0; i<${#SRV[@]}; i++ )); do
  echo ${SRV[$i]}
done

Here’s the output from the script:

$ sh ./haha
homer.joedog.org
marge.joedog.org
bart.joedog.org
lisa.joedog.org
burns.joedog.org

Here’s another way to coax the data out of the $SRV array. You can convert it into a space separated string and loop through it in a traditional manner:

SRV=${SRV[@]}
for S in $SRV ; do
  echo $S
done

After you guys vet this in the comments, I’ll add it to the sh scripting cheat sheet. Happy hacking.

UPDATE: A reader sends me a one-liner which implements similar functionality. If you don’t require an indexed array, then it’s only drawback is its perl dependency.

SRV=$(egrep '[^[:space:]$]' servers.txt|egrep -v '^#'|perl -pe 's/^s+//')
for S in $SRV ; do
  echo $S
done

It Knows Me Better Than I Know Myself….

By jdfulmer | Published January 9, 2014 | Leave a comment

I write a lot of software with which I interact. If it’s easy for me, then it’s easy for you. I try to keep it easy for me. JoeDog’s Pinochle is the first program against which I’ve competed. It’s been a surreal experience.

The program was designed to be competitive against me. Tonight it took two out of three games. The damn thing knows me inside and out. And why not? I wrote it. And while I can exploit some knowledge of its inter-workings, I can’t predict all its behavior. It was designed to learn bidding from experience.

Bidding is the hardest aspect of this game. The team that wins the bid has an incredible opportunity to earn a lot of points. At the same time, overbids come at a large price. A failure to make the bid means the bid is deducted from your score.

When the game was first released, its bids were implemented programmatically. I like to think I’m a pretty good programmer but that version of the game played like a moran. To improve it, I had the game play itself hundreds of thousands of times. It would store those results and use them to generate future bids.

This implementation has resulted in a much more competitive program. Now it bids more aggressively — much more aggressively. It bids like me which is odd because I didn’t tell it to do that. I told it to learn from its experience and as a result of that experience, its personality morphed into mine.

Microsoft’s Bigguns

By jdfulmer | Published July 23, 2012 | Leave a comment

In April Microsoft made news when it became a top-twenty contributor to the Linux kernel. The Redmond giant contributed over 20,000 lines of code in support of Hyper-V. This was a striking turn around. Remember, Microsoft’s CEO once described Linux as a “cancer.”

Although it contributed over twenty thousand lines of code, the Internets are now abuzz over a single line of code in hv/hv.h line 45:

  #define HV_LINUX_GUEST_ID_HI 0xB16B00B5

A Microsoft programmer assigned 0xB16B00B5 to HV_LINUX_GUEST_ID_HI. That’s a hexadecimal number whose value is 2,976,579,765. But that’s not what has the Internets all worked up. Look again. 0xB16B00B5 is 1337-speak for BIG BOOBS.

Creating Config Files For sh Scripts

By jdfulmer | Published June 8, 2012 | 1 Comment

Your JoeDog uses mondorescue for bare-metal Linux restoration. We use mondorestore to recover the OS and Net Backup to recover its content. Since we’re only concerned about archiving the OS for bare-metal recovery, it’s necessary to exclude directories when we run mondoarchive.

My exclude requirement varies from server to server so I wanted to build the list dynamically. As a coder, I have religious aversion to altering scripts for the purpose of configuring them. If we set config variables inside the script, then we have a different version on every server. That’s a paddlin’.

For my mondoarchive script, I developed a pretty slick way to read a configuration file and build an exclude list. The list is configured in a conf file that ignores comment lines and superfluous white space. A typical configuration looks like this:

#
# This file is maintained by the Puppet Master 
# 
# This is the exclude list for mondoarchive Directories inside
# this list will not be archived for bare metal recovery.
#
/tmp
/export
/usr/src
/var/mail
/var/cache
/var/log

My mondoarchive script builds a string of pipe separated directories like this:

/tmp|/export|/usr/src|/var/mail|/var/cache|/var/log

Since very few of you will have a similar usecase, I wrote an example that reads the file into a sh array. This version will loop through the array and print each one.

#!/bin/sh
# An example script that reads a list from a config
# file into a sh script array.
CONF="haha.conf"
LIST=""
#
# Read the directory list from $CONF
if [[ -e $CONF ]] ; then
  while read line ; do
    chr=${line:0:1}
    # XXX: Use awk's substr on older systems like
    # HPUX which don't support the above syntax.
    # chr=$(echo $line | awk '{print substr($1,0,1)}')
    case $chr in
     '#')
       # ignore comments
       ;;
     *)
       if [[ ${#line} -gt 2 ]] ; then
         if [[ -z $LIST ]] ; then
           LIST="$line"
         else
         LIST="$LIST $line"
         fi
       fi
       ;;
    esac
  done < $CONF
else
  echo "$0: [error] unable to locate $CONF"
fi

let X=1
for I in $LIST ; do
  echo "$X: $I"
  let X=$X+1
done

Let’s run this bad boy and see what happens:

$ sh haha
1: /tmp
2: /etc
3: /usr/local
4: /data/mrepo

If some of the concepts listed don’t make sense, then you might want to see our sh scripting cheat sheet. It will help you understand things like ‘-e $CONF’ and sh script arrays. Happy hacking.

UPDATE: Given the introduction to this post, it’s likely that many of you have arrived here in search of a mondoarchive backup script. Well, we won’t let you leave empty handed. You can grab my archive script here: Mondo Rescue Archive Script

This script builds both NFS recoverable archives and DVD images to an NFS mounted volume. Here’s its usage banner:

Usage: archiver [-c|-n]
Requires either a '-c' or a '-n' argument
  -c      create a CD Rom archive
  -n      create an NFS archive

Is There An AJP Functional Test?

By jdfulmer | Published June 6, 2012 | 2 Comments

There are plenty of helpful tools to test network services. If you want to check HTTP functionality, you could craft a request with curl, wget or “siege -g” to see if a server is functioning. If you understand the service protocol, you can always telnet to a TCP port and type a transaction.

Unfortunately, there aren’t many tools to help you test AJP protocol. Sure, you can telnet to the port to ensure it’s running, but how many people know how to craft an AJP transaction? I didn’t.

In order to help you test AJP servers like Apache’s tomcat, I wrote ajping. It connects to a user-define port and conducts a simple transaction. ajping validates the server’s response and clocks the length of the transaction. Over the LAN, you should expect times in the hundreds of seconds. This is a command line utility. In order to install it, run the following commands:

 $ wget http://download.joedog.org/AJP/ajping.txt
 $ mv ajping.txt ajping
 $ chmod +x ajping

You can test a server with it like this:

LT $ ajping tommy.joedog.org:8009
Reply from tommy.joedog.org: 7 bytes in 0.019 seconds
Reply from tommy.joedog.org: 7 bytes in 0.004 seconds
Reply from tommy.joedog.org: 7 bytes in 0.004 seconds
Reply from tommy.joedog.org: 7 bytes in 0.011 seconds
Reply from tommy.joedog.org: 7 bytes in 0.004 seconds
Reply from tommy.joedog.org: 7 bytes in 0.016 seconds
Reply from tommy.joedog.org: 7 bytes in 0.009 seconds
Reply from tommy.joedog.org: 7 bytes in 0.021 seconds
Reply from tommy.joedog.org: 7 bytes in 0.011 seconds
Reply from tommy.joedog.org: 7 bytes in 0.025 seconds

I’ve also incorporated this code into a check_ajp script for Zenoss. Remove the .txt extension and install it on Zenoss as you would any other script. Happy hacking.

UPDATE: I fixed the links to point to the new download location. H/T paalfe

Use Fido To Process FTP Uploads

By jdfulmer | Published June 4, 2012 | Leave a comment

Did you ever want to process a file immediately after it was uploaded via FTP? You could have the upload script execute a remote command after the file is uploaded. That requires shell access that you may or may not be able to grant. On the server, you could run a processing script every minute out of cron but that could get messy.

Fido provides alternative method.

Starting with version 1.0.7, Fido has the ability to monitor a file or directory by its modification date. When the date changes, fido launches a script. We can use this feature to process files that are uploaded via ftp.

In this example, we’ll monitor a directory. In fido.conf, we’ll set up a file block that points to a directory. (For more information about configuring fido, see the user’s manual). This is our configuration:

/home/jdfulmer/incoming {
 rules = modified
 action = /home/jdfulmer/bin/process
 log = /home/jdfulmer/var/log/fido.log
}

With this configuration, fido will continuously watch /home/jdfulmer/incoming for a modification change. When a file is upload, the date will change and fido will launch /home/jdfulmer/bin/process. Pretty sweet, huh?

Not quite. The modification date will change the second ftp lays down the first bite. Our script would start to process the file before it’s fully uploaded. How do we get around that? We can make our script smarter.

For the purpose of this exercise, I’m just going to move uploaded files from incoming to my home directory. Here’s a script that will do that:

#!/bin/sh

PREFIX="/home/jdfulmer/incoming"
FILES=$(ls $PREFIX)

for F in $FILES ; do
  while [ -n "$(lsof | grep $F)" ] ; do
    sleep 1
  done
  mv $PREFIX/$F /home/jdfulmer
done

In order to ensure the file is fully uploaded, I check lsof for its name. If there’s an open file handle under that name, then the script will continue to loop until it’s cleared. When the while loop breaks, the script moves the file.

There’s just one more thing to think about. When the script moves the file what happens to the directory fido is watching? Yes. Its modification date changes. In my example, process runs a second time but does nothing since nothing is there. Depending on your situation, you may need to make the script a little smarter.

Joe Dog Software