Select and Timeouts and Poll, Oh My!

Hanging ChadsBack in 1999-ish, Your JoeDog made a decision! Once he makes a decision, he doesn’t look back. “That’s over, Doggers. It’s time to move on.” And move on we did. Since 1999 we’ve been using select to check if our socket descriptor is ready for input. This is an important test because siege pounds your web server until it calls him “Daddy.” If the socket doesn’t get ready, then the request must time out or siege will hang.

We don’t want more hanging chads!

There were two mechanisms we could have used to accomplish this: poll and select. We chose select because it was more flexible and more readily available. At the time, we supported a great many platforms: Solaris, HP-UX, AIX, SCO, BSD, Linux, etc. The downside to select was its capacity. The fd_set couldn’t handle numbers higher than 1024. Well back in 1999, Yahoo! might have been doing 1024 concurrent connections but mere mortals didn’t need that capacity.

Recently we told you that siege crashes when large pools of simultaneous users are created. We finally got to the bottom of that problem and it dates back to that 1999 decision. Siege aborts when socket descriptors larger than 1024 are passed to FD_SET. We will fix this problem but there’s currently no ETA. It will require extensive testing and more of those difficult 1999-ish decisions.

For example, should we just stone-cold switch to poll or should we support both mechanisms? The latter will create a macro soup; does anyone have a stomach for that? Another option is create an array of fd_sets and place them on each thread. That would allow us to continue using select. (Your JoeDog kinda likes this option). And finally we could just say “Fsck it.” If you really have that many users, then you can afford Load Runner.

NOTE: To give you some perspective on traffic, this site is ranked 155,295 in the US. We almost never exceed 50 hits a second. Your JoeDog professionally webmasters a site ranked under 8000 and it almost never exceeds 100 hits a second. You must have a really large audience to generate 1024 hits a second. The vast majority of you should be fine with 255 threads or less.

Siege 3.1.0 Release Candidate 1

We received messages about core dumps that occur when siege is run with more than 700 threads. That’s a lot of threads, you guys! First things first — and this can’t be stressed enough — if you run siege with -c700, make sure your web server is configured with a pool of 700 threads. If you hit a pool of 256 threads with 700 users, all you’re gonna do is make a mess, mmmkay?

Now back to those core dumps. Here’s the thing: we’ve been unable to reproduce the problem but some diligent siegers have worked with us to get stack traces. As a result, we’ve added improved error handling and integrity checks in the problematic region. Your JoeDog would greatly appreciate if you could test version 3.1.0 release candidate 1. You can provide feedback here or by email, whichever you prefer.

Millions of Cookies, Cookies For Me

cookiemonsterA couple weeks ago, a reader notified Your JoeDog of a problem with siege’s cookie handling. When the server sets a cookie that’s already stored, siege won’t update its expiration time. A JoeDog Fellow told him, “That’s a paddlin’.” The last cookie in, is the next cookie out.

The investigation took Your JoeDog into cookie.c, a file he hasn’t touched in years. “This code is awful,” he said. “Who wrote this shit? … Hmm, some guy named Jeff Fulmer, et al. If I ever meet that guy, I’ll tell him a thing or two.”

As an ethnic German with a tendency to over engineer — well, um — everything,  Your JoeDog decided to completely overhaul cookie handling. Why change two lines when you can rewrite everything, amirite? The file cookie.c now contains a COOKIE object with getters and setters, cookies.c is a cookie list object which houses all cookies in memory. Cookies will persist in $HOME/.cookies but the details are still being considered.

Here’s the thing. Siege isn’t a browser, it’s many browsers. We store cookies at the thread level. Each thread gets its own list of cookies. To distinguish them, we used pthread_self(), which is a long long int like 140018384393984 or something. Technically it’s not a long long int, it’s a pthread_t, but you get the point. The next time you fire up siege, its threads will have different IDs so we need a convenient way to associate stored cookies with new thread IDs.

Furthermore, there’s no guarantee that your next run will have the same number of simulated users. We need to consider how to handle that situation. If you have 50 stored cookies and you run 60 new users, should we stop assigning cookies at 50 or do we start repeating them starting with cookie 1 to user 51? Inquiring minds want to know. I want to know.

The new code is not yet in version control. If you want a copy, let us know. We can create a beta branch or send you a tar ball. Happy hacking.

NOTE: Your JoeDog refers to cookie and cookies as objects because they are. Yes, siege is written in C rather than C++ but you can write objects in C just fine thankyouverymuch. An object is simply a thing which references itself. To achieve this in C, you just pass a reference to the object as an argument.

Hey! There’s No configure Script!!1!1!!!1!

Your JoeDog gets this a lot lately: “The INSTALL file says run ./configure but there’s no configure file.” It took a second but it’s now clear what’s happening. You guys are grabbing the code directly from

When Your JoeDog redesigned this site, he put that snazzy “Fork me on Github” banner up in the corner. Exciting! So now you guys are forking me! Here’s the thing. The stuff on github is THE source code, it’s not a source distribution. You can find the source distribution on the downloads site.

So what’s the difference? The source distribution contains helper scripts which are generated by autotools. You know, like that configure script. Since configure is built from other files, it’s not a source file and we don’t maintain it in version control.

That doesn’t mean you can’t fork me on github! If you take that route, you’ll need to build your own configure script. We’ll learn how to do that after the jump….

Continue reading Hey! There’s No configure Script!!1!1!!!1!

Recursion Level 21 Exceeded

recursionYour JoeDog’s long national mail nightmare is over.

Here at JoeDog Industries we forward our mail to Gmail. Google de-spams them and places them snuggly into our mailboxes. When we send mail, Google masquerades as us, Your JoeDog. To play this game, they want secure authentication. It logs into our sendmail server which relays our precious messages.

Since our server was configured so that only authenticated users could relay and since Google was de-spamming our messages, Your JoeDog was all, “Why should we even bother with the bloated pig known as spam assassin?” So we shut it down and removed it from the system. Done and done, amirite?

Life was good! We were running a mail server on the public internets without paying a heavy price in terms of administration or computing costs.

Now Your JoeDog studied economics and one thing he learned was this: There’s so such thing as a free lunch. Well sum bitch if that ain’t true. A couple weeks ago, Gmail decided it would start bouncing obvious spam. Guess what happened?

They bounced the message and our mailer sent it back. So they bounced it again and we promptly sent it back. While this was going on, Amazon was all “Why are these fsckers crushing our infrastructure?” So they throttled our asses at the firewall. So then the mail queue started backing up.

At that point Your JoeDog was all, “UGH! We have to run spam assassin again.” The idea was this: We’ll catch spam before it gets to Google then they won’t bounce it back.”

Seems like a good idea, right? WRONG!

Google is better at this shit than Your Spam Assassin. Even though we caught a bunch of stuff, they found plenty more to bounce at us. With Amazon throttling us and Google bouncing us, JoeDog Industries became The United States of Suck.

The solution to this cluster bump turned out to be pretty simple. We’ll learn about that after the jump! Continue reading Recursion Level 21 Exceeded

How To Switch Your Site To HTTPS

I told you that Your JoeDog would eventually complete that task. There was no need to remind us every couple of months. It’s done, now. Cross it off the list.

Congratulations, I guess?? What task is now complete?

Last August Your Google announced that it would give all sites running https a slight rankings bounce. Your JoeDog thought, “Yeah, yeah, yeah. We should do that.” Then August became October and October became January and January became whatever comes after that. Now you’re able to read this blog over a secure connection.

Let’s set-up https after the jump! Continue reading How To Switch Your Site To HTTPS

It Knows When You Are Sleeping

lurchMohammad Sabah is a former Netflix data scientist who worked on that company’s movie recommendation engine. He brought some of that technology to Workday, an HR software manufacturer. So what’s the result of a marriage between movie recommendations and human resource management? It’s an application called Workday Talent Insights.

What’s that do? It’s kind of cool, actually. The program predicts which employee is likely to quit and what can be done to retain them. It’s not clear how many inputs it evaluates but Your JoeDog can envision an applicable scenario. Let’s say — and why not? — a really tall employee dislikes being known as “Lurch.” The application’s corrective action could be “Stop calling him that!”


Block IP Addresses With, um, block

Last night we told you that ISIS assholes were attacking America and its WordPress blogs. Have no fear, kids. In the War Against Internet Dickbags, Your JoeDog is here to help. Whenever he’s attacked by these people, i.e., all the time, he grabs their IP addresses from the logs and blocks their asses. He feeds those addresses to a script called block which, um, blocks them.

The script is just a convenience which makes your life easier. iptables does the heavy lifting. If you’re running a linux system — and why aren’t you? — then you probably have iptables.

Let’s block some dickwads now! (Continued after the jump)

Continue reading Block IP Addresses With, um, block

Who’s Honoring Me Now?

Your JoeDog’s Fido was selected by Profit Bricks as its 24th Best Free Sysadmin Tool

About Fido:

A multi-threaded file watch utility, Fido can monitor a file or directory to see if its modification time changed, and it’s intuitive enough to recognize a change even if the daemon was down at the time of the change. Written for RedHat Enterprise Linux, Fido should run in all Linux flavors, and the source bundle contains RedHat init.d scripts for users’ convenience.

Key Features:

  • Content changes can be recognized with regular expression pattern matches
  • Monitors files for changes in content or modification times
  • It will kick off a user-defined script when it notices a change
  • Coded to POSIX 1003.1 standards

Pinochle 1.0.7

Yesterday was Christmas and you know what? You can’t get a stinkin’ slice of pizza on Christmas. Hardly anything is open on December 25th. Christmas is basically house arrest.

To kill the time between Christmas morning and the grand re-opening of his coffee shop, Your JoeDog played a little pinochle. It’s a pretty good game. The computer bids well and plays a reasonably strong hand but — WTF? — the bid dialog box moves all over the place.

Each time you submit a bid, the dialog moves so you can’t just click a second time without moving your mouse. Well that’s annoying. Indeed. Your JoeDog fixed that yesterday. House arrest ended today….

Life is good.

[JoeDog: Pinochle-1.0.7]