A sh script INI parser

An INI file provides an old-timey way to configure applications. On MSDOS and early versions of Windows, it was the primary configuration mechanism. (Sadly, it’s been mostly replaced by the stupid registry.) Fortunately, INI files are still around because they remain very useful. On Linux, many developers have adapted the format.

Years ago Your JoeDog wrote an INI parser in sh. This enabled him to use one config file across multiple SAP landscapes. Each section of the file, is headed by the landscape name: [D] [Q] [P] etc. A script’s configuration varied depending on which landscape it was launched in. This enabled us to test on one landscape and promote the script without any changes to the next one.

Recently a colleague needed this type of one-with-many configuration. She needed to loop through a list of servers and reference their many attributes. Your JoeDog dusted off this parser for her and now he’s passing it along to you.

Consider this INI file:

#
# The parser supports comments
[WEB]
 srv = www.joedog.org
 usr = jdfulmer
 file = haha.txt
 path = /usr/local/content/www
[FTP]
 srv = ftp.joedog.org
 usr = jeff
 file = papa.txt
 path = /usr/local/content/ftp

Now here’s a script that contains our parser along with an example of how to use it:

#!/bin/sh
##++
## Parses an INI style file:
## [section]
## attr=thing
## key=val
## [header]
## thing=another
## foo=bar
## <p>
## @param file full path to the INI file
## @param section a header that matches the stuff in brackets [section]
## @return void (the variables are made available to your script)
##--
ini_parser() {
 FILE=$1
 SECTION=$2
 eval $(sed -e 's/[[:space:]]*\=[[:space:]]*/=/g' \
 -e 's/[;#].*$//' \
 -e 's/[[:space:]]*$//' \
 -e 's/^[[:space:]]*//' \
 -e "s/^\(.*\)=\([^\"']*\)$/\1=\"\2\"/" \
 < $FILE \
 | sed -n -e "/^\[$SECTION\]/I,/^\s*\[/{/^[^;].*\=.*/p;}")
}
# A sections array that we'll loop through
SECTIONS="WEB FTP"
for SEC in $SECTIONS; do
 ini_parser "papa.conf" $SEC
 echo "scp $file $usr@$srv:$path/$file"
done
exit

Now let’s run this script and see what happens:

Pom $ sh papa
scp haha.txt [email protected]:/usr/local/content/www/haha.txt
scp papa.txt [email protected]:/usr/local/content/ftp/papa.txt



Posted in Programming, sh | Leave a comment



ArrayList.ensureCapacity

Have you ever wanted to initialize a java ArrayList to a certain size? You peruse the javadoc and you see what appear to be two different options, one is a constructor option and the other is a chained method.

Here’s the constructor:

ArrayList (int initialCapacity)

Constructs an empty list with the specified initial capacity.

And here’s the method:

void ensureCapacity(int minCapacity)

Increases the capacity of this ArrayList instance, if necessary, to ensure that it can hold at least the number of elements specified by the minimum capacity argument.

And here’s the thing: Neither one changes the logical size of the array. They each change its capacity. That is they change the size it can reach before it has to start copying its values to resize itself. If you do something like the following, you’ll get an IndexOutOfBounds Exception:

ArrayList<String> list = new ArrayList<String>();
list.ensureCapacity(200);
list.add(190, "JoeDog");

Exception in thread "main" java.lang.IndexOutOfBoundsException

If you want to initialize an ArrayList to a particular size, you’ll have to roll your own method:

 private static void ensureSize(ArrayList<?> list, int size) {
   list.ensureCapacity(size);
   while (list.size() < size) {
     list.add(null);
   }
 }

You can call it like this:

ArrayList<Thing> list = new ArrayList<Thing>();
ensureSize(list, 64);

 


Posted in Java, Programming | Leave a comment



Nerd Splaining Large Numbers

Holy shit — the Economist really outdid itself. What now? In this post, they explained why Gangnam Style will break YouTube’s view counter. They used 3726 characters and 612 words to explain that computer integers don’t go on forever. When the Gangnam Style counter reaches 2,147,483,647 it will stop counting. Why?

Integers are stored in a series of ones and zeroes. On a 32-bit platform, you can only store value in 32 consecutive ones or zeros. Go to this binary to decimal calculator and put 32 ones in the binary field. Press “Calculate” and you’ll get this answer: 4294967295.

But the Gangnam Style counter is maxed at half of that? How come? That’s because computers use positive and negative numbers. The range falls above and below zero, i.e., from -2,147,483,648 to 2,147,483,647. Gangnam Style is approaching the upper bound.

If YouTube switched to 64-bit architecture they could capture up to 9 quintilian views.

Remember kids, there are 10 kinds of people in this world. Those who understand binary numbers and those who don’t.

[Economist: Wordy Word Words on Computer Integers]

 

Posted in Programming, Tech Media, Technology | Leave a comment



In Praise of Default Values

Your JoeDog likes options. He feels that if a program takes a variable value, that value should be configuarable. A programmer can spend a great deal of time selecting the perfect socket timeout, but unless the user works in the same environment it’s not necessarily perfect for them.

On the occasions when Your JoeDog uses Windows, he finds himself struggling to make the software do his bidding. It takes time to add another text field to a Windows GUI, so developers tend to limit the number of configurable options.

At the same time, he hates complicated software. You shouldn’t need a computer science PhD in order to configure scheduling software. Yet it’s impossible to use Tivoli’s workload scheduler and not feel completely overwhelmed. It can take days to set up.

These notions don’t have to be mutually exclusive. Software can be extremely flexible and simple to use. Your JoeDog achieves this notion in his own software with a novel concept known as the “default value.” If you don’t set a value, you get the default. If you require more precision, you can change those settings.

Generally speaking, software users don’t care about every configurable value. They have a subset of values they want to change. If everything has a default that doesn’t need to be set for the software to function, then the documentation becomes less overwhelming. If all you want to do is change one setting, then you can search the docs for just that configuration.

Your JoeDog does enough GUI programming that he can speak to the notion he mentioned above. It takes time to add labels and text fields to a program. Those GUI elements also take valuable screen real estate. As a result, many programmers limit the flexibility of their programs.

Here’s a thought: why not make the program configurable with a combination of a GUI and a configuration file? You can place the frequently changed stuff inside the GUI and the more obscure features inside the file. Trust me, the users who really want to change something will discover how to do that if you let them.

Keep it simple but make it flexible and your users will be appreciative … until you blog about it.

 

 

Posted in Programming | Leave a comment



AI Reporting

Your JoeDog is a Big Fan of artificial intelligence. His pinochle game represents one foray into the field. The computer bids based on its results experience. Your JoeDog can’t predict how it will bid a particular hand. It looks for experiences that resemble its current hand and it bids accordingly. Unfortunately, it still plays programmatically. As such, it can never be better than this nerd-blogger.

CBS Sports and Yahoo are doing interesting things with AI. Their fantasy football sites use artificial intelligence to summarize millions of games each week. The software analyzes lots of data and composes articles much like a human reporter. They only fail the Turing Test due to a contemplation of scale: A rational person soon realizes there aren’t enough humans on earth to produce that many articles by Tuesday morning.

There’s a more personal reason why Your JoeDog likes these cyber reporters: they think highly of his coaching skill:

Tonzie Crushers benefited from smart coaching this week. Coach Fulmer left Chris Johnson and Justin Hunter on the bench in favor of Frank Gore and Robert Woods, who were both expected to score less.

These great decisions boosted Tonzie Crushers’ final score by 22.1 points, which just made the final result that much more embarrassing. Putting Gore in the staring lineup also gained more points than any other coaching move this week, making it the Volkswagen Start of the Week.

Well this week. Last week they thought he was a moran

 

Posted in Programming | Leave a comment



Check Your Inputs: SQL Injection Edition

Here’s a question which tends to make Your JoeDog cringe: “So, what do you do?”

It’s often asked when he has a drink in his hand. And when he has a drink in hand, he doesn’t want to talk about work. Sometimes the inquiring person hears the answer, parses “computers” and wants to know why their laptop is slow. Honestly, Your JoeDog has no idea. Occasionally, he meets another nerd who wants to talk shop.

Recently he met a web nerd, the kind of web nerd who suffers from illusory superiority because he lacks the skill to recognize his ineptitude. These guys often contain a conspiratorial streak. This guy was no exception. The conversation soon shifted to hacking and web security.

Web Nerd puked a word salad of vulnerabilities but his beloved PHP was exonerated. “You can’t inject SQL because the mysql libs don’t allow multiple statements,” he said.

Couple points. 1.) the PHP mysql_ functions are deprecated. Astute JoeDog readers use PDO or MySQLi. 2.) You can still do injection as long as you keep it in a single statement.

Let’s try that after the jump!

Continue reading “Check Your Inputs: SQL Injection Edition” »

Posted in Community, On The Job, PHP, Programming | 2 Comments



Memory Leaks

Your JoeDog loves C but he hates memory management. That’s why he writes objects in C. They make managing memory easier. Still, he’s been know to cause a memory leak or two … okay, maybe a few more than that.

Valgrind is supposed to help identify those fsckers. You run a program inside valgrind and it checks your memory utilization. Awesome! Except when it’s not.

Your JoeDog has had zero point zero luck with valgrind. He’s been working on a snippet that will be incorporated into Your Fido. Your JoeDog loves snippets. By reducing complexity they allow him to focus on the integrity of the new code. In this particular snippet, he placed the new functionality inside a loop to check its long-running performance. The damn thing leaked!

“How did you know it was leaking?”

“I watched its memory usage in top and it continued to expand.”

“Well that sucks.”

“Indeed.”

So Your JoeDog ran the program in valgrind like this:

valgrind -v --leak-check=full --tool=memcheck ./haha

After it completed its iterations, valgrind had this to say:

==21112== HEAP SUMMARY:
==21112== in use at exit: 0 bytes in 0 blocks
==21112== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==21112==
==21112== All heap blocks were freed -- no leaks are possible

What the hell? This thing was leaking all over the place. Now Your JoeDog may have done something stupid. Instead of memory leaks, he might have created memory hogs. “What are memory hogs?” They are allocated resources that are no longer used but with references that are still maintained. Valgrind won’t identify a memory hog as a memory leak. But this was a snippet and an audit was easy. There were no hogs; something was leaking.

After an extensive audit, Your JoeDog identified the leak. It wasn’t in the snippet. He was using a dynamic array from array.c Here’s the offending code:

ARRAY
array_destroy(ARRAY this) 
{
  int i;

  for (i = 0; i < this->length; i++) {
    xfree(this->data[i]);  
  } 
  xfree(this->data);
  this = NULL;
  return this; 
}

Your JoeDog freed everything but the array object itself. Here’s the fix:

ARRAY
array_destroy(ARRAY this) 
{
  int i;

  for (i = 0; i < this->length; i++) {
    xfree(this->data[i]);  
  } 
  xfree(this->data);
  xfree(this);
  this = NULL;
  return this; 
}

It’s not clear why valgrind didn’t identify that. Any thoughts?

 

Posted in C, Programming | Leave a comment



So Are You Vulnerable To Shell-shock?

Here’s a quick command line test to see if you’re vulnerable to shell-shock, the bash vulnerability that everyone — I mean everyone — is talking about:

$ env x='() { :;}; echo 1. env' bash -c "echo 2. bash"

If your bash is vulnerable, it will execute the echo command inside the environment, if it’s not vulnerable, then it will only execute the stuff after -c

A vulnerable system prints this:

$ env x='() { :;}; echo 1. env' bash -c "echo 2. bash"
1. env
2. bash

A non-vulnerable system prints this:

$ env x='() { :;}; echo 1. env' bash -c "echo 2. bash"
2. bash

On the vulnerable system, the echo command that is set in the environment is executed by bash when the shell is invoked:

env x='() { :;}; echo 1. env' bash -c "echo 2. bash"

The stuff in red should NOT be executed. That’s a bug; it needs to be fixed.

NOTE: The second command was run on the server that hosts this blog entry. You guys can quit trying, mmmkay?

 

Posted in Applications, Security, sh | Leave a comment



Dunning–Kruger Effect

the dumbest man on the internetsYour JoeDog once worked with a programmer who couldn’t program. Now you’re probably thinking, isn’t programming an important qualification for that position? Not in a large corporation. To succeed in that environment, you need buzzwords and cliches. If you have them, managers just  assume you know what you’re talking about.

This particular non-programmer — or Ouch! as we liked to call him — was hired to build a Intranet site. It took him a year and a half to construct something that looked like your eight-year old nephew slapped together in a weekend. It was slow, poorly marked-up but at least it had a confusing layout and design.  Ouch had a parry for its shortcomings: Microsoft. “IE is a horrible web browser. It violates standards and ActiveX has a mind of its own.”

An appropriate response would have been, “If that’s true, how come all these non-Ouch sites look fine and work well in IE?” Instead, he received an award.

Because Ouch could steal someone else’s files and alter their markup to render the company’s text and images, we concede that he had some skill.  Armed with a comprehensive understanding of his craft, Ouch would have also known: 1.) How to work around a browser’s weaknesses by 2.) Stealing  the javascript, too, as it probably fixed those weaknesses but then he would have known too much and realized 3.) He was in the wrong profession.

While Ouch was laboring over his Intranet and ankle-deep in Cold Fusion, we were building an enterprise site with J2EE. And while Ouch didn’t know much, he did know this: in nerd hierarchy, Cold Fusion falls way below java.

So Ouch told everyone — and I mean everyone, his peers, his managers, the cleaning crew that he should be programming in java. To prove his point, he got the java logo tattooed on his bicep … which he showed to everyone.

Here’s the thing: Ouch wasn’t smart enough to know he couldn’t program in java. And management wasn’t smart enough to know he couldn’t program in java. The next thing you know, Ouch was stealing O’Reilly code — including the copyright notice — and attempting to implement the usecase. As far as I can tell, in one year in that position he didn’t release a thing that wasn’t immediately rewritten by somebody else.

Eventually Ouch was sacked but not for incompetence, he called his immediate supervisor the c-word. Management never considered him anything but a fine programmer. The buzzwords he used matched the ones they read in trade rags. How could he be anything but brilliant?

I didn’t realize it at the time but Ouch and the managers who considered him competent all suffered from the Dunning–Kruger Effect.

 

Posted in On The Job, Programming | 1 Comment



Shellshocked

Wired provides an interesting angle on the bash shell bug that has all your panties in a bind

[Brian] Fox drove those tapes to California and went back to work on Bash, other engineers started using the software and even helped build it. And as UNIX gave rise to GNU and Linux—the OS that drives so much of the modern internet—Bash found its way onto tens of thousands of machines. But somewhere along the way, in about 1992, one engineer typed a bug into the code. Last week, more then twenty years later, security researchers finally noticed this flaw in Fox’s ancient program. They called it Shellshock, and they warned it could allow hackers to wreak havoc on the modern internet.

[Wired: The Internet Is Broken]

 

Posted in Applications, Programming, sh | Leave a comment



Recent Comments

  • Oleg: Hm, i have the same problem as “Gokul Muralidharan says: July 27, 2012 at 1:00 pm” He says “I...
  • Jeff: Oleg, Because you told it to run just one repetition: -r1 I think what you’re looking for is...
  • Oleg: I think, i have found one bug. The command siege -d10 -r1 -c25 -f url.txt hits ONLY FIRST url. Why?
  • Jeff Fulmer: The hit counts will vary since you’re doing time-based testing. Instead of -t10s try -r10 siege...
  • Oleg: Strange results. For example i do siege -t 10s -v -c 10 https://google.com and count of hits always...