Using Siege to Tune Apache on GNU/Linux

UPDATE: This article is now slightly dated. In the comments below, Null4Ever provides more contemporary kernel parameter settings. 

Siege is a great tool to measure web site performance and establish benchmark metrics. What do you do if siege shows your performance stinks? This document illustrates how to tune your server and validate those tunings with siege.

For the purpose of this exercise we will tune a very old Linux server to effectively serve static content at a rate of 200 requests per second. We’ll start with an out-of-box apache configuration and build from there. By the way, when I say “old” I mean old:

Ben: $ uname -a 
Linux ben 2.4.10-4GB #1 Fri Sep 28 17:20:21 GMT 2001 i586 unknown
Ben: $ bin/httpd -V
Server version: Apache/2.0.47
Server built:   Jul 18 2003 18:16:04

Damn, that’s old! Let’s see how this Model-T performs. The first thing we need to do is establish a benchmark:

Bully $ siege -d1 -c200 -t1m http://ben.home.joedog.org/apache_pb.gif
Lifting the server siege...      done.
Transactions:                10230 hits
Availability:               100.00 %
Elapsed time:                59.90 secs
Data transferred:            22.69 MB
Response time:                0.15 secs
Transaction rate:           170.78 trans/sec
Throughput:                   0.38 MB/sec
Concurrency:                 26.18
Successful transactions:     10230
Failed transactions:             0
Longest transaction:          2.34
Shortest transaction:         0.00

There’s no two ways about it: that’s crappy. What steps can we take to improve that? An obvious place to look is the apache configuration. Is the server configured to handle 200 simulataneous connections? If apache is configured with less than 200 workers, then requests are queued until workers are available to handle them. Let’s check our config:

<IfModule prefork.c>
  StartServers         5
  MinSpareServers      5
  MaxSpareServers     10
  MaxClients         150
  MaxRequestsPerChild  0
</IfModule>

Sure enough, we have fewer workers than requests. Let’s bump those numbers in order to meet our requirement. While we’re at it, we’ll increase the initial pool so we pay less of a penalty to fork new workers:

<IfModule prefork.c>
  StartServers        50
  MinSpareServers     15
  MaxSpareServers     25
  MaxClients         225
  MaxRequestsPerChild  0
</IfModule>

After apache restart we find thirty-three httpd processes in memory. With a bigger pool in memory, it’s easier to accomodate a large pool of incoming requests. With the expansion of our workers pool, we have the capacity to meet our load requirement. Let’s see how these changes affected our performance:

Lifting the server siege...      done.
Transactions:                10290 hits
Availability:                99.44 %
Elapsed time:                59.07 secs
Data transferred:            22.83 MB
Response time:                0.05 secs
Transaction rate:           174.20 trans/sec
Throughput:                   0.39 MB/sec
Concurrency:                  9.55
Successful transactions:     10290
Failed transactions:            58
Longest transaction:          3.07
Shortest transaction:         0.00

Our transaction rate improved by nearly 3.5 per second. This is because requests spent less time waiting for a worker to handle them. If you watch the web server’s socket table during a siege, one thing stands out. The number of sockets in TIME_WAIT. This what I saw during the last run:

Ben: $ netstat -a | grep TIME_WAIT | wc -l   
8081

If we could recycle those connections, we could improve our performance. The kernel parameter that controls TIME_WAIT reuse is ‘net.ipv4.tcp_tw_recycle’. The sysctl command will enable you to view and set kernel parameters. Let’s check the value on our web server:

Ben: $ sysctl net.ipv4.tcp_tw_recycle
net.ipv4.tcp_tw_recycle = 0

We’re not reusing sockets in TIME_WAIT. Let’s change it and view our results:

Ben: $ sysctl -w net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_tw_recycle = 1

For readers with newer systems, we recommend this additional setting:

sysctl -w net.ipv4.tcp_tw_reuse=1

I set those parameters on both the computer running siege and the server running apache. To make the changes permanent, add them to /etc/sysctl.conf Now let’s run another test:

Lifting the server siege...      done.
Transactions:                10231 hits
Availability:               100.00 %
Elapsed time:                31.25 secs
Data transferred:            22.69 MB
Response time:                0.05 secs
Transaction rate:           327.39 trans/sec
Throughput:                   0.73 MB/sec
Concurrency:                 16.68
Successful transactions:     10231
Failed transactions:             0
Longest transaction:          3.12
Shortest transaction:         0.00

Our transactions per second sky-rocketed to 327.39! That’s a significant improvement but we still have a problem. Notice the elapsed time. I stopped the siege short of a minute because it was hung. In run after run, siege was hung at about 10,200 transactions. In fact, every time I run the siege it hangs at that number. We’re exhausting something, but what? Let’s check our kernel parameters and see what we find:

Ben: $ /sbin/sysctl -a  | grep 102
net.ipv6.neigh.default.gc_thresh3 = 1024
net.ipv6.route.gc_thresh = 1024
net.ipv4.ip_conntrack_max = 10232
net.ipv4.neigh.default.gc_thresh3 = 1024
net.ipv4.route.gc_thresh = 1024
net.ipv4.tcp_max_syn_backlog = 1024
net.core.optmem_max = 10240
kernel.sem = 250    32000    32    1024
kernel.rtsig-max = 1024

It looks like we hit the ip_conntrack_max limit. If kernel dropped packets it will let us know. Let’s check the system logs:

Apr  2 15:00:58 ben kernel: ip_conntrack: table full, dropping packet.

Sure enough. Let’s double it.

Ben: $ sysctl -w net.ipv4.ip_conntrack_max=`echo 10232*2 | bc`
net.ipv4.ip_conntrack_max = 20464

One more run, let’s see how we do:

Bully # siege -d1 -c200 -t1m http://ben.home.joedog.org/apache_pb.gif
Lifting the server siege...      done.
Transactions:                20462 hits
Availability:               100.00 %
Elapsed time:                59.65 secs
Data transferred:            45.39 MB
Response time:                0.04 secs
Transaction rate:           343.03 trans/sec
Throughput:                   0.76 MB/sec
Concurrency:                 15.31
Successful transactions:     20462
Failed transactions:             0
Longest transaction:          1.26
Shortest transaction:         0.00

Much better. With a little bit of tuning, we were able to double our server’s transaction rate.