Introducing BCachefs

Well this is interesting:

An ex-Google engineer is developing a new file system for Linux, with the hopes that it can offer a speedier and more advanced way of storing data on servers.

After a number of years of development, the Bcache File System (Bcachefs) “is more or less feature complete — nothing critical should be missing,” wrote project head Kent Overstreet, in an e-mail to the Linux Kernel Mailing List late Thursday.

[PC World: Introducing BCachefs]


The Hidden Cost of Microsoft Windows

The Windows team upgraded a server to W2K8 and now its schedules are failing. The jobs kick off but abend and have to be run manually. Your JoeDog has been sucked in because he manages enterprise scheduling. Exciting!

“You tested this before you went live, right?” Um, …

The team can run jobs manually but they fail in the scheduler. Your JoeDog thinks this is a classic case of ENV mismatch, a discrepancy between the logged-in environment and the sheduled runtime environment. He was implored to open a case with IBM. Their conclusion? An ENV mismatch.

Still, the Windows team can’t figure it out. The jobs kick off in the background and they can’t capture any error messages. Those who live by the GUI, die by the GUI. Thus far they’ve tied up more than 100 man hours trying to solve a problem that a subpar Linux admin could solve in a few minutes on his own.

Add this to the hidden cost of Microsoft Windows.

[Image lifted from Nerdy Perv]

Site Outage Or How To Move Your EC2 Into Another Region

Your JoeDog’s site was down for much of the afternoon. Why? Well, it appears some Amazon employees are having a bad day. This site is hosted with Amazon Web Services and throughout much of the day it was unavailable. When he could get in, Your JoeDog checked his logs for a root cause. As far as he could tell, there was nothing wrong at the server level. Something was wrong with the host.

Hoping the problem was regional, Your JoeDog decided to move his EC2 instance into another region. He did that by following these steps inside the EC2 console:

  1. Shut down the running instance
  2. Hilight the volume, then Actions => Create image
  3. Go to AMIs on the left nav; select your image, then Actions => Launch
  4. As you navigate the settings select a new region to place it in.

Your JoeDog’s server runs in the Western region and he didn’t have an option to move it east. He did, however, have subregions from which to choose. Whereas the old server was running in us-west-2a, he put the new one in us-west-2c.

When the server came up, it did so with new security settings. No problem. It’s easy to associate the new server with your old rules. Hilight it in the dashboard then select Actions => Networking => Change Security Group. You can select your old group and toss the new one that was created when you launched a new image.

NOTE: Your JoeDog now has two volumes and one running instance in his EC2 Dashboard. He’s 99% certain he can toss that old volume but that won’t happen until he grabs copies of all his 1s and 0s. (Just to be safe).


Anonymous At War

anonYour JoeDog blogged about politics for a solid five years. During that time, there was nobody he wouldn’t attack head-on. With his blogging posse, he helped drive several foes from the Internets, he defaced other blogs with titties and Goatse and waged a holy war with the state of Denmark. He even got into a bare-knuckle brawl with Daniel Pearl’s widow. (Hey – she started it.)

But it’s not true to say he’s without fear. Your JoeDog wouldn’t dream of sparring with the “hacker” collective known as Anonymous. They have a lot combined knowledge and a lot of time on their hands. You may consider yourself well guarded but Anonymous will tirelessly prod your defenses until they find an exploitable weakness. They are more relentless than you are thorough.

And so Your JoeDog was greatly amused when he learned that Anonymous declared war on Islamic Jihadis. Because fsck Jihadis. Their values are anathema to hacker values. They try to silence opposing views. They monopolize information and points of view. They destroy rather than build. They are sexist, intolerant douchebags.

Anonymous will start with their social media but they’ll soon tire of that and move onto other things. The fledgling Islamic State surely has some digital infrastructure that could use a good fscking with. And I have no doubt they’ll expose the identities of Western assholes who take up weapons in the name of a sixth century goat herder. Good luck, Jihadis. You’re gonna need it.


How To Prevent root’s mailbox From Filling With Cron Messages?

So one of your file systems is full and you traced it to a curious phenomena: root’s mailbox is loaded with messages. And you think to yourself, “Who the hell is sending mail to root?” Chances are, it’s you!

This recently happened to a friend who’s new to Linux. Root’s mailbox was full of messages that looked like this:

/bin/sh: ntpdate: not found

It turns out he had a cron job that ran ntpdate to sync his system time with a server at the National Institute of Standards. (Here’s a complete list of NIST time servers.)

The program ‘ntpdate’ was located in /usr/bin which was not in root’s PATH. So here is lesson number one: Always use the full path to a program in cron. In this case, he should have used /usr/bin/ntpdate.

Now about those mail messages. When a cron job fails, it captures the program’s output and mails it to the cron owner. If you set /bin/ls to run every five minutes in cron, you’re going to get a directory listing in your mailbox every five minutes. You can silence cron and stop mail messages by redirecting output to /dev/null. The ‘>’ character means ‘redirect’ Here’s the syntax: /bin/ls > /dev/null

Yet that wouldn’t have helped Your JoeDog’s friend. In his case, the offending message was an error message. Error messages are written to a different file descriptor.

The Linux/UNIX shell has three file descriptors: stdin, stdout and stderr. ‘/bin/sh: ntpdate: not found’ was written to stderr. In the shell, 1 represents stdout and 2 represents stderr. To capture both we need another character. ‘&’ means file descriptor. So 2>&1 means redirect stderr (2) to whatever stdout (1) points to. Here’s the syntax he should have used:

0 */6 * * * /usr/bin/ntpdate > /dev/null 2>&1

Now Your JoeDog has been in this game for quite a while. In the day, to really silence the crontab we’d write the redirect like this:

0 */6 * * * /usr/bin/ntpdate 1 >/dev/null 2>&1

That’s basically the same as the first one but it seemed to work around oddities and keep our mailboxes clean. The crontab man page still contains that syntax. Your JoeDog will continue to use that superfluous first 1 but your mileage may vary.

Because I like esc-esc completion, that’s why

Your JoeDog got a new laptop – happy nice time!

He bought if from System 76 with Ubuntu 13.10 preinstalled. No major work to get this bad boy configured, right? Well….

In a previous life Your JoeDog was a UNIX guy. He earned his chops on AIX before moving to an HPUX shop. On commercial UNIX, AT&T’s korn shell is king. In the 1990s, it showed its flexibility by accomodating both vi and emacs editing. But c’mon. Any UNIX geek worth his salt ran it in vi-mode. Along the way, JoeDog developed a habit of completing filenames by hitting the esc key twice. /usr/lo esc-esc and he was in local. Bash weenies arrive at the same location by hitting the tab key. Only one stroke they say. Perhaps. But Your JoeDog gets emotional satisfation from banging esc twice. [pop, pop]

Now unfortunately, Ubuntu didn’t include ksh on his shiny new laptop. No problem, ‘apt-get install ksh’ Unfortunately that version didn’t support esc-completion. No problem, ‘apt-get install pdksh’ And ah-way we guh …. WTF? That didn’t support it either.

That’s okay. Your JoeDog is pretty adept with a compiler. He snagged pdksh from source and rolled his own. There’s a couple things to remember when you do this: 1. make sure the shell you install is in /etc/shells or you won’t be permitted to login. 2. You need to add ‘set -o vi-esccomplete’ to your $HOME/.profile in order to achieve esc-esc completion.

During the build, the following error was encountered:

gcc -c -DHAVE_CONFIG_H -I. -I. -g -O table.c
./ "gcc -E -DHAVE_CONFIG_H -I. -I." < ./ > tmpsiglist.out
sort: cannot read: +2n: No such file or directory
make: *** [siglist.out] Error 1

Well that sucks. Your JoeDog punched that error into the Google machines and boy did he get an earful. There were plenty of people who encountered that error. They were frustrated and turned to the Internets for help. Hey, anybody know how to correct this? Oh, the Internets had opinions all right. You know what they had to say? “ksh sucks, use bash!” Well, that’s certainly one thought. Here’s another one: fsck you! Flies gotta fly. Bees gotta be. And UNIX grey beards gotta esc-esc.

So the Internets weren’t much help but the sort’s documentation helped shed some light. To fix the error above, you need to go POSIX. So before you run ‘make’ run this command:

export _POSIX2_VERSION=199209

Are you with us so far? If you’re an esc-esc guy, we’re pretty sure you are. With his new pdksh in place, Your JoeDog was esc-esc’ing all over the place … until he tried to log in to a new Xsession. WTF? Basically, he was locked out of his brand new laptop.

To get into single-user mode in Ubuntu, hold down the shift key while it’s booting. That will provide a menu through which you can navigate to the root shell. Once you’re in the root shell, you’ll have to mount file systems read-write in order to fix the errors that locked you out of your system. Here’s how you mount root as read-write:

mount -o remount,rw /

Here’s what Your JoeDog discovered inside his $HOME/.xsession-errors file:

/usr/sbin/lightdm-session: 15: set: Illegal option -o vi-esccomplete


So the directive that provides the feature he wants prevents him from logging in. If that’s not a Catch-22, then what is? Since we rolled our own shell, let’s just alter its code so it does our bidding. Unfortunately, the pdksh source wasn’t the most intuitive read but Your JoeDog slogged through it so you won’t have to.

On line 1142 of vi.c you’ll find this:

 case Ctrl('['): /* some annoying at&t ksh's */
    if (!Flag(FVIESCCOMPLETE))
        return -1; 
 case '\': /* at&t ksh */

Do you see that? Try searching for the word “annoying” in the code. Okay, to do esc-esc completion by default without resorting to ‘set -o vi-esccomplete’ change that block of code so it looks like this:

 case Ctrl('['): /* some annoying at&t ksh's */
 case '\': /* at&t ksh */

Now esc-esc falls down to esc- and its behavior is the same.



1. Download pdksh-5.2.14

2. Unzip the tarball: gunzip -9cd pdksh-5.2.14.tar.gz | tar -xvf –

3. Edit vi.c

On line 1142 of vi.c you’ll find this:

case Ctrl('['): /* some annoying at&t ksh's */
    if (!Flag(FVIESCCOMPLETE))
        return -1; 
 case '\': /* at&t ksh */

Make it look like this:

 case Ctrl('['): /* some annoying at&t ksh's */
 case '\': /* at&t ksh */

4. Run: ./configure

5. Run: export _POSIX2_VERSION=199209

6. Run: make

7. Run: cp ksh /bin/pdksh

8. Run: cp ksh.1 /usr/share/man/man1/pdksh.1

9. VERY IMPORTANT: add /bin/pdksh to /etc/shells

10. In /etc/password change your shell to /bin/pdksh

Happy double-escaping.

I’m Speed Racer And I Drive Real Fast

When it comes to supercomputing Linux is king. The top forty-three fastest supercomputers in the world run Linux. The forty-fourth fastest, which does weather forecasting in Europe, runs IBM’s AIX.  How does Windows fare in the world of supercomputing? Not well. Just three of the boxes classified as supercomputers run Microsoft Windows. The fastest of these is in Shanghai. It ranks 187th in the world….

Intel releases updated Linux drivers

Intel has released a new version of the open source Intel graphics package, that can be used on Linux systems. This is good news for users of the software, who will be able to access the updated version, known as 12.07. With the package, users will get new X Server drivers for Intel cards, along with other features that work with these specific drivers.

Version 2.20.0 is the most significant component of this new release, available with the xf86-video-intel driver for the X Server. Those who regularly use this type of technology, or any for that matter, spending time on their computers playing or communicating to their friends via e-mail, will be glad to hear this news. The feature comes with SNA, which is a 2D acceleration method that can be selected at runtime. It’s purpose is to use less CPU than the previous method, UXA, and to be faster. To take advantage of this, the user must specify Option “AccelMethod” “sna” in the Xorg.conf file. It also features modern Intel graphics cards, which the new SNA architecture is designed to make the most of.

This new release from Intel is pre-tested, and therefore geared towards developers who look for this feature rather than Linux end users. It comes with other features that have also been tested, along with the main ones that will be the biggest draw. The other features include the stable 3.4.x version of the Linux kernel. This was made available to users in May and its default setting is to use the RC6 power saving mode. Those who want to get their hands on this new package can download it from Intel’s Linux driver web site and begin enjoying the benefits. Its source code is licensed under MIT and GPLv2 licences.

Linux Marketshare

I’m responsible for around 150 GNU/Linux servers. Not one of them actually shipped with Linux. They were all bare metal installs at the point of delivery. That’s generally how Linux works. You buy hardware from one vendor and OS entitlements from another. If my experience isn’t unusal, then the latest server track numbers from IDC are quite extraordinary.

IDC tracks servers shipped by OEMs to customers and reports on hardware and OS marketshare. It doesn’t track bare metal installations, hardware re-provisions and VM guest installs. Last quarter, according to IDC, factory revenue for Linux grew while it shrank for Windows and UNIX. On top of the provisioning methods I mentioned above, customers are increasingly asking IBM, HP and Dell to ship servers with Linux installed.

According to IDC, demand was driven by the need for high performance and cloud computing. Linux has also earned a reputation as more reliable and more secure than that other Intel OS. And if you want to ruin hardware performance, just add virus protection which is a necessity in the Windows operating environment. Given the risk vs. the reward of increased performance, many Linux administrators simply eschew virus protection. That gives Linux a real world performance boost over its rival from Microsoft.

Once Linux conquers the datacenter, it’s only a matter of time until millions of open source developers really start to focus on the desktop. Proprietary software’s best days are behind it.


Garbled Apostrophes And Other Things

Do you have man pages with garbled type? I’m working on a multi-threaded file watcher that searches for patterns in files and executes commands on a match. In order to release it into the wild, I need documentation. That means man pages. So I’m viewing my man pages and I see crap like this: ’-f /path/file’

Those are supposed to be single-quotes, i.e., apostrophes.

For this project, I’m building my man pages from perl PODs with Pod::Man. In case you’d like to do the same, here’s a handy utility for making man pages from perl pods. It converts POD data to *roff.

# A Pod::Man example script
use Pod::Man;
my $input = $ARGV[0] or barf();
my $output = $ARGV[1] or barf();

my $parser = Pod::Man->new (release => $VERSION, section => 8);
$parser->parse_from_file ($input, $output);

sub barf() {
  print "usage: $0 <file.pod> <file.1>n";

When I saw the garbled text above, I suspected a problem with my method. It turns out that wasn’t the case at all. The culprit was my character set. My language was set to en_US.UTF-8 but my terminal didn’t support that character set. If you’re having a similar problem, you can check your character set with this command:

$ set | grep -i lang

The fix is easy:

export LANG=en_US

Add that to your .profile to make it permanent.