frustratedThe Windows team upgraded a server to W2K8 and now its schedules are failing. The jobs kick off but abend and have to be run manually. Your JoeDog has been sucked in because he manages enterprise scheduling. Exciting!

“You tested this before you went live, right?” Um, …

The team can run jobs manually but they fail in the scheduler. Your JoeDog thinks this is a classic case of ENV mismatch, a discrepancy between the logged-in environment and the sheduled runtime environment. He was implored to open a case with IBM. Their conclusion? An ENV mismatch.

Still, the Windows team can’t figure it out. The jobs kick off in the background and they can’t capture any error messages. Those who live by the GUI, die by the GUI. Thus far they’ve tied up more than 100 man hours trying to solve a problem that a subpar Linux admin could solve in a few minutes on his own.

Add this to the hidden cost of Microsoft Windows.

Posted in Environment, OS, System | Leave a comment

So one of your file systems is full and you traced it to a curious phenomena: root’s mailbox is loaded with messages. And you think to yourself, “Who the hell is sending mail to root?” Chances are, it’s you!

This recently happened to a friend who’s new to Linux. Root’s mailbox was full of messages that looked like this:

/bin/sh: ntpdate: not found

It turns out he had a cron job that ran ntpdate to sync his system time with a server at the National Institute of Standards. (Here’s a complete list of NIST time servers.)

The program ‘ntpdate’ was located in /usr/bin which was not in root’s PATH. So here is lesson number one: Always use the full path to a program in cron. In this case, he should have used /usr/bin/ntpdate.

Now about those mail messages. When a cron job fails, it captures the program’s output and mails it to the cron owner. If you set /bin/ls to run every five minutes in cron, you’re going to get a directory listing in your mailbox every five minutes. You can silence cron and stop mail messages by redirecting output to /dev/null. The ‘>’ character means ‘redirect’ Here’s the syntax: /bin/ls > /dev/null

Yet that wouldn’t have helped Your JoeDog’s friend. In his case, the offending message was an error message. Error messages are written to a different file descriptor.

The Linux/UNIX shell has three file descriptors: stdin, stdout and stderr. ‘/bin/sh: ntpdate: not found’ was written to stderr. In the shell, 1 represents stdout and 2 represents stderr. To capture both we need another character. ‘&’ means file descriptor. So 2>&1 means redirect stderr (2) to whatever stdout (1) points to. Here’s the syntax he should have used:

0 */6 * * * /usr/bin/ntpdate > /dev/null 2>&1

Now Your JoeDog has been in this game for quite a while. In the day, to really silence the crontab we’d write the redirect like this:

0 */6 * * * /usr/bin/ntpdate 1 >/dev/null 2>&1

That’s basically the same as the first one but it seemed to work around oddities and keep our mailboxes clean. The crontab man page still contains that syntax. Your JoeDog will continue to use that superfluous first 1 but your mileage may vary.

Posted in Environment, System | Leave a comment

Your JoeDog got a new laptop – happy nice time!

He bought if from System 76 with Ubuntu 13.10 preinstalled. No major work to get this bad boy configured, right? Well….

In a previous life Your JoeDog was a UNIX guy. He earned his chops on AIX before moving to an HPUX shop. On commercial UNIX, AT&T’s korn shell is king. In the 1990s, it showed its flexibility by accomodating both vi and emacs editing. But c’mon. Any UNIX geek worth his salt ran it in vi-mode. Along the way, JoeDog developed a habit of completing filenames by hitting the esc key twice. /usr/lo esc-esc and he was in local. Bash weenies arrive at the same location by hitting the tab key. Only one stroke they say. Perhaps. But Your JoeDog gets emotional satisfation from banging esc twice. [pop, pop]

Now unfortunately, Ubuntu didn’t include ksh on his shiny new laptop. No problem, ‘apt-get install ksh’ Unfortunately that version didn’t support esc-completion. No problem, ‘apt-get install pdksh’ And ah-way we guh …. WTF? That didn’t support it either.

That’s okay. Your JoeDog is pretty adept with a compiler. He snagged pdksh from source and rolled his own. There’s a couple things to remember when you do this: 1. make sure the shell you install is in /etc/shells or you won’t be permitted to login. 2. You need to add ‘set -o vi-esccomplete’ to your $HOME/.profile in order to achieve esc-esc completion.

During the build, the following error was encountered:

gcc -c -DHAVE_CONFIG_H -I. -I. -g -O table.c
./ "gcc -E -DHAVE_CONFIG_H -I. -I." < ./ > tmpsiglist.out
sort: cannot read: +2n: No such file or directory
make: *** [siglist.out] Error 1

Well that sucks. Your JoeDog punched that error into the Google machines and boy did he get an earful. There were plenty of people who encountered that error. They were frustrated and turned to the Internets for help. Hey, anybody know how to correct this? Oh, the Internets had opinions all right. You know what they had to say? “ksh sucks, use bash!” Well, that’s certainly one thought. Here’s another one: fsck you! Flies gotta fly. Bees gotta be. And UNIX grey beards gotta esc-esc.

So the Internets weren’t much help but the sort’s documentation helped shed some light. To fix the error above, you need to go POSIX. So before you run ‘make’ run this command:

export _POSIX2_VERSION=199209

Are you with us so far? If you’re an esc-esc guy, we’re pretty sure you are. With his new pdksh in place, Your JoeDog was esc-esc’ing all over the place … until he tried to log in to a new Xsession. WTF? Basically, he was locked out of his brand new laptop.

To get into single-user mode in Ubuntu, hold down the shift key while it’s booting. That will provide a menu through which you can navigate to the root shell. Once you’re in the root shell, you’ll have to mount file systems read-write in order to fix the errors that locked you out of your system. Here’s how you mount root as read-write:

mount -o remount,rw /

Here’s what Your JoeDog discovered inside his $HOME/.xsession-errors file:

/usr/sbin/lightdm-session: 15: set: Illegal option -o vi-esccomplete


So the directive that provides the feature he wants prevents him from logging in. If that’s not a Catch-22, then what is? Since we rolled our own shell, let’s just alter its code so it does our bidding. Unfortunately, the pdksh source wasn’t the most intuitive read but Your JoeDog slogged through it so you won’t have to.

On line 1142 of vi.c you’ll find this:

 case Ctrl('['): /* some annoying at&t ksh's */
    if (!Flag(FVIESCCOMPLETE))
        return -1; 
 case '\\': /* at&t ksh */

Do you see that? Try searching for the word “annoying” in the code. Okay, to do esc-esc completion by default without resorting to ‘set -o vi-esccomplete’ change that block of code so it looks like this:

 case Ctrl('['): /* some annoying at&t ksh's */
 case '\\': /* at&t ksh */

Now esc-esc falls down to esc-\ and its behavior is the same.



1. Download pdksh-5.2.14

2. Unzip the tarball: gunzip -9cd pdksh-5.2.14.tar.gz | tar -xvf –

3. Edit vi.c

On line 1142 of vi.c you’ll find this:

case Ctrl('['): /* some annoying at&t ksh's */
    if (!Flag(FVIESCCOMPLETE))
        return -1; 
 case '\\': /* at&t ksh */

Make it look like this:

 case Ctrl('['): /* some annoying at&t ksh's */
 case '\\': /* at&t ksh */

4. Run: ./configure

5. Run: export _POSIX2_VERSION=199209

6. Run: make

7. Run: cp ksh /bin/pdksh

8. Run: cp ksh.1 /usr/share/man/man1/pdksh.1

9. VERY IMPORTANT: add /bin/pdksh to /etc/shells

10. In /etc/password change your shell to /bin/pdksh

Happy double-escaping.

Posted in Environment, System | 2 Comments

Do you have man pages with garbled type? I’m working on a multi-threaded file watcher that searches for patterns in files and executes commands on a match. In order to release it into the wild, I need documentation. That means man pages. So I’m viewing my man pages and I see crap like this: ’-f /path/file’

Those are supposed to be single-quotes, i.e., apostrophes.

For this project, I’m building my man pages from perl PODs with Pod::Man. In case you’d like to do the same, here’s a handy utility for making man pages from perl pods. It converts POD data to *roff.

# A Pod::Man example script
use Pod::Man;
my $input = $ARGV[0] or barf();
my $output = $ARGV[1] or barf();

my $parser = Pod::Man->new (release => $VERSION, section => 8);
$parser->parse_from_file ($input, $output);

sub barf() {
  print "usage: $0 <file.pod> <file.1>\n";

When I saw the garbled text above, I suspected a problem with my method. It turns out that wasn’t the case at all. The culprit was my character set. My language was set to en_US.UTF-8 but my terminal didn’t support that character set. If you’re having a similar problem, you can check your character set with this command:

$ set | grep -i lang

The fix is easy:

export LANG=en_US

Add that to your .profile to make it permanent.

Posted in Environment, System | Leave a comment

Recent Comments

  • CC: Many thks for your reply.
  • Jeff Fulmer: You mean your operating environment can only sustain more than 1000 connections for a few minutes....
  • CC: HI BUDDY I met this when the number of concurrent connections > 1000,siege can only sustained for a few mins....
  • Patrick: Hello- Have a simple 3 tier system and wanting to have multiple Siege testers run against multiple...
  • Oleg: Hello. Are the response time is the same as TTFB?