sh Script: Read lines and omit comments and blanks

I perform remote operations on a series of servers. Rather than maintain the server list in several scripts, I consolidated it into a single file called servers.txt  Exciting! The second I did that, I raised my own bar. You’d expect a config file parser to omit comments and blank lines, right? I do.

The anticipated time to write that parser was longer than I expected. In order to save you time, dear reader, I decided to post it here. This sh script reads a file into an array while skipping comment lines and blanks.

Here’s a sample config file:

#
# Comment line 
# Wed Feb 25 19:28:54 EST 2014
homer.joedog.org
 marge.joedog.org
bart.joedog.org
 lisa.joedog.org
burns.joedog.org
# EOF

And here’s a script to parse it:

#!/bin/sh
let X=0
while read line ; do
  if [[ ! $line =~ [^[:space:]] ]] ; then
    continue
  fi
  [ -z "`echo $line | grep '^#'`" ] || continue
  SRV[$X]=$line
  let X=$X+1
done < servers.txt
for (( i=0; i<${#SRV[@]}; i++ )); do
  echo ${SRV[$i]}
done

Here’s the output from the script:

$ sh ./haha
homer.joedog.org
marge.joedog.org
bart.joedog.org
lisa.joedog.org
burns.joedog.org

Here’s another way to coax the data out of the $SRV array. You can convert it into a space separated string and loop through it in a traditional manner:

SRV=${SRV[@]}
for S in $SRV ; do
  echo $S
done

After you guys vet this in the comments, I’ll add it to the sh scripting cheat sheet. Happy hacking.

UPDATE: A reader sends me a one-liner which implements similar functionality.  If you don’t require an indexed array, then it’s only drawback is its perl dependency.

SRV=$(egrep '[^[:space:]$]' servers.txt|egrep -v '^#'|perl -pe 's/^\s+//')
for S in $SRV ; do
  echo $S
done
Posted in Programming, sh | Leave a comment



It Knows Me Better Than I Know Myself….

robotI write a lot of software with which I interact. If it’s easy for me, then it’s easy for you. I try to keep it easy for me. JoeDog’s Pinochle is the first program against which I’ve competed. It’s been a surreal experience.

The program was designed to be competitive against me. Tonight it took two out of three games. The damn thing knows me inside and out. And why not? I wrote it. And while I can exploit some knowledge of its inter-workings, I can’t predict all its behavior. It was designed to learn bidding from experience.

Bidding is the hardest aspect of this game. The team that wins the bid has an incredible opportunity to earn a lot of points. At the same time, overbids come at a large price. A failure to make the bid means the bid is deducted from your score.

When the game was first released, its bids were implemented programmatically. I like to think I’m a pretty good programmer but that version of the game played like a moran. To improve it, I had the game play itself hundreds of thousands of times. It would store those results and use them to generate future bids.

This implementation has resulted in a much more competitive program. Now it bids more aggressively — much more aggressively. It bids like me which is odd because I didn’t tell it to do that. I told it to learn from its experience and as a result of that experience, its personality morphed into mine.

 

Posted in Applications, Java, Pinochle, Programming | Leave a comment



Pinochle

Today I’m pleased to announce the first public release of JoeDog’s Pinochle. It’s a computerized version of the classic card game. It plays a four-player variation in which you are paired with a computer player against two computer players. Exciting!

This project is notable for several reasons: 1.) It’s the first time I’ve released software with a graphical interface and 2.) it’s the first major project I’ve completed in java.

JoeDog’s Pinochle is the culmination of hundreds of hours of work over the past several years. The groundwork was laid on planes and trains. It offered an enjoyable way to pass the time as I sat in traveling tubes. Last September I finally achieved a functioning version of the game. Since then, I’ve honed its ability to play a descent game.

Because pinochle maintains a strict set of rules for governing play, most of the “intelligence” in this game was implemented programmatically. Unfortunately, its original ability to assess and bid a hand was very weak. In order to improve that, I’ve built experience into the game. It played itself for hundreds of hours and stored those outcomes. Now when it bids a hand, it consults past experience to shape future results. A skilled human can still beat it but give me time. It will get better with each ensuing release.

Future

I’m currently honing the game’s ability to beat the pants off you people. That may take some time. Once I’ve built adequate intelligence into the game I’d like to add more variation. A double-deck version will be added at some later date. I’d also like to add three and five player variations.

Licence

This game is currently licensed under the terms of shareware. It contains some code that was published without licensing terms and the author has not answered my inquiries about its license. It’s probably in the public domain but until I get verification I can’t release it under an open source license. If you’d like a copy of the code, send me an email. Until then, enjoy the binaries.

Posted in Applications, Pinochle | Leave a comment



Because I like esc-esc completion, that’s why

Your JoeDog got a new laptop – happy nice time!

He bought if from System 76 with Ubuntu 13.10 preinstalled. No major work to get this bad boy configured, right? Well….

In a previous life Your JoeDog was a UNIX guy. He earned his chops on AIX before moving to an HPUX shop. On commercial UNIX, AT&T’s korn shell is king. In the 1990s, it showed its flexibility by accomodating both vi and emacs editing. But c’mon. Any UNIX geek worth his salt ran it in vi-mode. Along the way, JoeDog developed a habit of completing filenames by hitting the esc key twice. /usr/lo esc-esc and he was in local. Bash weenies arrive at the same location by hitting the tab key. Only one stroke they say. Perhaps. But Your JoeDog gets emotional satisfation from banging esc twice. [pop, pop]

Now unfortunately, Ubuntu didn’t include ksh on his shiny new laptop. No problem, ‘apt-get install ksh’ Unfortunately that version didn’t support esc-completion. No problem, ‘apt-get install pdksh’ And ah-way we guh …. WTF? That didn’t support it either.

That’s okay. Your JoeDog is pretty adept with a compiler. He snagged pdksh from source and rolled his own. There’s a couple things to remember when you do this: 1. make sure the shell you install is in /etc/shells or you won’t be permitted to login. 2. You need to add ‘set -o vi-esccomplete’ to your $HOME/.profile in order to achieve esc-esc completion.

During the build, the following error was encountered:

gcc -c -DHAVE_CONFIG_H -I. -I. -g -O table.c
./siglist.sh "gcc -E -DHAVE_CONFIG_H -I. -I." < ./siglist.in > tmpsiglist.out
sort: cannot read: +2n: No such file or directory
make: *** [siglist.out] Error 1

Well that sucks. Your JoeDog punched that error into the Google machines and boy did he get an earful. There were plenty of people who encountered that error. They were frustrated and turned to the Internets for help. Hey, anybody know how to correct this? Oh, the Internets had opinions all right. You know what they had to say? “ksh sucks, use bash!” Well, that’s certainly one thought. Here’s another one: fsck you! Flies gotta fly. Bees gotta be. And UNIX grey beards gotta esc-esc.

So the Internets weren’t much help but the sort’s documentation helped shed some light. To fix the error above, you need to go POSIX. So before you run ‘make’ run this command:

export _POSIX2_VERSION=199209

Are you with us so far? If you’re an esc-esc guy, we’re pretty sure you are. With his new pdksh in place, Your JoeDog was esc-esc’ing all over the place … until he tried to log in to a new Xsession. WTF? Basically, he was locked out of his brand new laptop.

To get into single-user mode in Ubuntu, hold down the shift key while it’s booting. That will provide a menu through which you can navigate to the root shell. Once you’re in the root shell, you’ll have to mount file systems read-write in order to fix the errors that locked you out of your system. Here’s how you mount root as read-write:

mount -o remount,rw /

Here’s what Your JoeDog discovered inside his $HOME/.xsession-errors file:

/usr/sbin/lightdm-session: 15: set: Illegal option -o vi-esccomplete

JESUS H. CHRISTMAS STOCKING!!1!1!!!

So the directive that provides the feature he wants prevents him from logging in. If that’s not a Catch-22, then what is? Since we rolled our own shell, let’s just alter its code so it does our bidding. Unfortunately, the pdksh source wasn’t the most intuitive read but Your JoeDog slogged through it so you won’t have to.

On line 1142 of vi.c you’ll find this:

 case Ctrl('['): /* some annoying at&t ksh's */
    if (!Flag(FVIESCCOMPLETE))
        return -1; 
 case '\\': /* at&t ksh */

Do you see that? Try searching for the word “annoying” in the code. Okay, to do esc-esc completion by default without resorting to ‘set -o vi-esccomplete’ change that block of code so it looks like this:

 case Ctrl('['): /* some annoying at&t ksh's */
 case '\\': /* at&t ksh */

Now esc-esc falls down to esc-\ and its behavior is the same.

 

THE IMPATIENT PERSON’S GUIDE TO MAKING esc-esc FILENAME COMPLETION THE DEFAULT BEHAVIOR IN PDKSH

1. Download pdksh-5.2.14

2. Unzip the tarball: gunzip -9cd pdksh-5.2.14.tar.gz | tar -xvf –

3. Edit vi.c

On line 1142 of vi.c you’ll find this:

case Ctrl('['): /* some annoying at&t ksh's */
    if (!Flag(FVIESCCOMPLETE))
        return -1; 
 case '\\': /* at&t ksh */

Make it look like this:

 case Ctrl('['): /* some annoying at&t ksh's */
 case '\\': /* at&t ksh */

4. Run: ./configure

5. Run: export _POSIX2_VERSION=199209

6. Run: make

7. Run: cp ksh /bin/pdksh

8. Run: cp ksh.1 /usr/share/man/man1/pdksh.1

9. VERY IMPORTANT: add /bin/pdksh to /etc/shells

10. In /etc/password change your shell to /bin/pdksh

Happy double-escaping.

Posted in Environment, System | 2 Comments



An Old Dog Learns A New Trick

Beginning with version 3.0.6-beta2, siege reacts differently to –reps=once.

In the past, when you invoked –reps=once, each siege user would invoke each URL in the file exactly one time. If urls.txt contained 100 files and you ran -c10 –reps=once, siege would finish its business with 1000 hits.

That was then.

This is now: siege runs each URL in the file exactly once. If you run -c10 –reps=once, then siege will split the file among all 10 users and hit each URL one time. Whereas in the past, you’d finish with 1000 hits, you now finish with 100 hits.

This should give you greater control by making tests more precise.

Posted in Applications, Siege | Leave a comment



Why Do Investors Love Amazon?

What’s happening at Amazon isn’t suppose to happen in modern finance. Shares are rising as profits are falling:

Amazon shares are up around 150 percent since mid-2010, which perhaps not coincidentally was the last time the company had sizable profits. In other words, investors really decided they loved the company only when net income began to slide.

Any fool can run a profitable company but it takes a gutsy person to build the world’s largest retailer….

[New York Times - All Amazon Is Missing Is a Profit]

Posted in Community, Technology | Leave a comment



Fido Learns A New Trick

I use Mondoarchive to create Linux recovery disks. Each server writes ISO images to a shared volume on a weekly basis. If any file inside that directory is older than seven days, then a server failed to create an ISO. In order to monitor this directory for failure, I added a new feature to fido. Exciting!

Starting with version 1.1.0 (click to download), fido can monitor a file or directory to see if it — or any file inside it — is older than a user configurable period of time. If fido discovers a file whose modification date exceeds the configured time, it fires an alert.

The following example illustrates how to configure the use case above:

/export {
  rules = exceeds 8 days
  exclude = ^\.|^lccns178$|^lccns179$|^lccns335$|lccns336$
  throttle = 12 hours
  action = /etc/fido/notify.sh
}

This file block applies to “/export” which is a directory. Since it’s a directory, the rules apply to every file inside it. In this case ‘rules’ is pretty straight forward. We’re looking for files that exceed eight days in age. This rule will always follow this format: exceeds [int] [modifier]. The modifier can be seconds, minutes, hours or days. If you take the long view — if you’re concerned about events far into the future — then you’ll have to do some math. We don’t designate years so you’ll have to use 1825 days if you want to be alerted five years out.

We also find a new feature inside this block. ‘exclude’ takes a regular expression and tells fido which files to ignore. Currently, ‘exclude’ only works inside a file block with an exceeds rule but I plan to make better use of it.

Finally we notice one final feature that we’ve never seen before. The ‘throttle’ directive tells fido how long to wait between alerts. In this scenario, fido will trigger an alert the second it finds a file which exceeds 8 days. If the problem is not addressed within twelve hours, it will fire another alert. Alerts will continue in twelve hour intervals until the problem is corrected.

I hope you enjoy these features. If there are enhancements you’d like to see, feel free to contact me either in the comments or by email.

Posted in Applications, Fido | Leave a comment



Gub’mint (IT) Mule

bureaucratsSean Gallagher has an interesting piece on (ars)technica. He asks, “Why do government IT projects fail so hard and so often?” Gallagher provides several reasons, most of which are symptoms of a large organization. Let’s examine that list.

1. The government uses antiquated technologies. Its bureaucracy is slow to move and slow to adapt. Older technologies remain long after their life cycle expires largely because the approval process for new ones is long and arduous. You can imagine many frustrating meetings that end with, “Fsck it. We’ll put it on XP.”

2. Its user base is really large. Gallagher cits as one example a DOD email rollout that touched 1.5 million users. That’s an astonishing number for an in-house IT department. Certainly there are web companies with more users — there are 425 million GMail accouts, for instance — but Google does web for profit. The army’s IT department is an expense.

3. Flawed metrics. Gallagher notes that many government IT dashboards are filled with nice metrics that contain a lot of nines. Unfortunately, those nines have little bearing on end user experience. If department CIOs measured things that mattered, they’d be filled with zeros.

I’m sure these are valid criticisms but how do they vary from other large organizations? I work for a large corporation and this week my company finally moved my laptop off Windows XP. There’s no way the entire organization will be XP-free by end-of-life-cycle. It took a Great Recession to convince management that Linux was a viable alternative to HP-UX.

Our user base is 15,000 and every internal roll-out contains some glitches or problems. The army rolls out applications to a user base that is two orders of magnitude larger than ours. Where we roll out in a carefully controlled environment, they have to provide service to all corners of the world. Some of those corners are pre-fab barracks on an Afghanistan mountain top.

I’m yet to meet a person in my company who likes our out-sourcing partner. The bean counters like how little they cost, but they’re not happy with their services either. Yet if you look at this partner’s dashboard, you’ll find it’s filled with as many nines as those government CIOs. Faulty metrics aren’t limited to the public sector.

The Affordable Care website famously crashed during its rollout. Yet on the surface it wasn’t subject to many of government IT’s shortcomings. Most of the work was handled by a private partner. They used apache webservers on Linux. The site was fronted by the Akamai CDN which greatly reduces load by moving content close to the users. Most importantly, the site wasn’t tied to antiquated government infrastructure. Yet it failed. Why?

When you examine the site you find it’s simply not optimized for heavy traffic. The pages are too heavy and they contain too many elements. Fifty-six javascript files? Really? Then we learn the system was tested under load that was an order of magnitude less then what they received on October 1st. The site was basically slash-dotted.

Certainly there are good failures and bad failures. Collapsing under the weight of your own popularity is a good one. Still, with better planning and better coding the Affordable Care site could have experienced a more successful roll-out. Those operations require a high level of expertise which brings us to what I suspect is the real reason government IT projects fail: constrained by the tax payer’s dime, government can’t attract the talent necessary to service a very large user base. In other words, we get what we pay for.

Posted in On The Job, Technology | Leave a comment



Siege 3.0.4 Becomes Part of the Problem

Siege 3.0.4 was just released. It contains a feature that I’ve added with a certain amount of reluctance. To understand the feature and the reason for my trepidation, let’s visit RFC 2616 and read what it has to say about Location headers:

For 3xx responses, the location SHOULD indicate the server's
preferred URI for automatic redirection to the resource. The 
field value consists of a single absolute URI.
    Location = "Location" ":" absoluteURI
An example is:
    Location: http://www.w3.org/pub/WWW/People.html

That’s pretty clear, right? The value of a location header must be an absolute URI. Yet a large number of developers ignore that directive. Here’s the response from a server running SquirrelMail, a popular web-based email program:

     HTTP/1.1 302 Found
     Date: Tue, 17 Sep 2013 16:50:52 GMT
     Server: CERN/1.0A
     X-Powered-By: PHP/5.2.5
     Location: src/login.php
     Content-Length: 0
     Connection: close
     Content-Type: text/html; charset=WINDOWS-1251

Although that Location header violates RFC 2616, nearly every web client will follow it to SquirrelMail’s intended destination. I say “nearly every client.” Until version 3.0.4, siege wouldn’t have followed it any where. It would have scratched its head and said, “Fsck it. Next URL.”

It is with some reluctance that I’ve included siege in the community of clients that allow developers to circumvent established standards. This convention has created a slew of bad coding practices on the world wide web. Didn’t close a table with an end tag? That’s okay, M$ will close it for you. Used a relative URI in a Location header? Don’t worry, siege will normalize it for you.

Ironically, version 3.0.4 includes one other feature enhancement. Its default User-agent is now in full compliance with RFC 2616. You win some, you lose some. And so it goes….

Posted in Applications, Siege | 5 Comments



Your JoeDog Loves Coffee, Hates Lines

coffee-houndIt used to be that everybody in America had their coffee cup filled to the same level: full. Now baristas ask if you need “room” for creme. Basically they’ve added a few seconds to each of the more than 100 million daily coffee transactions for the benefit of a few special snowflakes for whom a quarter inch clearance isn’t adequate for all the god damn creme they’re gonna pour into their cups. Just stop it.

Posted in Community | Leave a comment



Recent Comments

  • luxoderm: An impressive share! I’ve just forwarded this onto a coworker who has been doing a little research on...
  • bit.ly: An interesting discussion is worth comment. I do think that you ought to publish more on this subject matter,...
  • Sohan: I am using node-js http server. I created http request to hit the server and log the message. In that case...
  • Jeff Fulmer: I don’t know what “simple http server” means. If you’re using apache out of the...
  • Sohan: MY server is simple http server. Can you tell me what thresol paramter or property is used in siegerc file.