Using Siege to Tune Apache on GNU/Linux
By Jeffrey Fulmer (April 2010)
Siege is a great tool to measure web site performance and establish
benchmark metrics. What do you do if siege shows your performance
stinks? This document illustrates how to tune your server and validate
those tunings with siege.
For the purpose of this exercise we will tune a very old Linux server
to effectively serve static content at a rate of 200 requests per
second. We'll start with an out-of-box apache configuration and build
from there. By the way, when I say "old" I mean old:
Ben: $ bin/httpd -V Server version: Apache/2.0.47 Server built: Jul 18 2003 18:16:04
That's old! Let's see how this Model-T performs. The first thing we need to do is establish a benchmark:
Bully $ siege -d1 -c200 -t1m http://ben.home.joedog.org/apache_pb.gifLifting the server siege... done.Transactions: 10230 hitsAvailability: 100.00 %Elapsed time: 59.90 secsData transferred: 22.69 MBResponse time: 0.15 secsTransaction rate: 170.78 trans/secThroughput: 0.38 MB/secConcurrency: 26.18Successful transactions: 10230Failed transactions: 0Longest transaction: 2.34Shortest transaction: 0.00
That's pretty crappy. What steps can we take to improve that. An obvious place to look is the apache configuration. Is the server configured to handle 200 simulataneous connections? If apache is configured with less than 200 workers, then requests are queued until workers are available to handle them. Let's check our config:
<IfModule prefork.c> StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 150 MaxRequestsPerChild 0</IfModule>
Sure enough, we have fewer workers than requests. Let's bump those numbers in order to meet our requirement. While we're at it, we'll increase the initial pool so we pay less of a penalty to fork new workers:
<IfModule prefork.c> StartServers 50 MinSpareServers 15 MaxSpareServers 25 MaxClients 225 MaxRequestsPerChild 0</IfModule>
After apache restart we find thirty-three httpd processes in memory. With a bigger pool in memory, it's easier to accomodate a large pool of incoming requests. With the expansion of our workers pool, we have the capacity to meet our load requirement. Let's see how these changes affected our performance:
Lifting the server siege... done.Transactions: 10290 hitsAvailability: 99.44 %Elapsed time: 59.07 secsData transferred: 22.83 MBResponse time: 0.05 secsTransaction rate: 174.20 trans/secThroughput: 0.39 MB/secConcurrency: 9.55Successful transactions: 10290Failed transactions: 58Longest transaction: 3.07Shortest transaction: 0.00
Our transaction rate improved by nearly 3.5 per second. This is because requests spent less time waiting for a worker to handle them. If you watch the web server's socket table during a siege, one thing stands out. The number of sockets in TIME_WAIT. This what I saw during the last run:
Ben: $ netstat -a | grep TIME_WAIT | wc -l 8081
If we could recycle those connections, we could improve our performance. The kernel parameter that controls TIME_WAIT reuse is 'net.ipv4.tcp_tw_recycle'. The sysctl command will enable you to view and set kernel parameters. Let's check the value on our web server:
Ben: $ sysctl net.ipv4.tcp_tw_recycle net.ipv4.tcp_tw_recycle = 0
We're not reusing sockets in TIME_WAIT. Let's change it and view our results:
Ben: $ sysctl -w net.ipv4.tcp_tw_recycle=1 net.ipv4.tcp_tw_recycle = 1
For readers with newer systems, we recommend this additional setting:
sysctl -w net.ipv4.tcp_tw_reuse=1
I set those parameters on both the computer running siege and the server running apache. To make the changes permanent, add them to /etc/sysctl.conf Now let's run another test:
Lifting the server siege... done.Transactions: 10231 hitsAvailability: 100.00 %Elapsed time: 31.25 secsData transferred: 22.69 MBResponse time: 0.05 secsTransaction rate: 327.39 trans/secThroughput: 0.73 MB/secConcurrency: 16.68Successful transactions: 10231Failed transactions: 0Longest transaction: 3.12Shortest transaction: 0.00
Our transactions per second sky-rocketed to 327.39! That's a significant improvement but we still have a problem. Notice the elapsed time. I stopped the siege short of a minute because it was hung. In run after run, siege was hung at about 10,200 transactions. In fact, every time I run the siege it hangs at that number. We're exhausting something, but what? Let's check our kernel parameters and see what we find:
Ben: $ /sbin/sysctl -a | grep 102 net.ipv6.neigh.default.gc_thresh3 = 1024 net.ipv6.route.gc_thresh = 1024 net.ipv4.ip_conntrack_max = 10232 net.ipv4.neigh.default.gc_thresh3 = 1024 net.ipv4.route.gc_thresh = 1024 net.ipv4.tcp_max_syn_backlog = 1024 net.core.optmem_max = 10240 kernel.sem = 250 32000 32 1024 kernel.rtsig-max = 1024
It looks like we hit the ip_conntrack_max limit. If kernel dropped packets it will let us know. Let's check the system logs:
Apr 2 15:00:58 ben kernel: ip_conntrack: table full, dropping packet.
Sure enough. Let's double it.
Ben: $ sysctl -w net.ipv4.ip_conntrack_max=`echo 10232*2 | bc` net.ipv4.ip_conntrack_max = 20464
One more run, let's see how we do:
Bully # siege -d1 -c200 -t1m http://ben.home.joedog.org/apache_pb.gifLifting the server siege... done.Transactions: 20462 hitsAvailability: 100.00 %Elapsed time: 59.65 secsData transferred: 45.39 MBResponse time: 0.04 secsTransaction rate: 343.03 trans/secThroughput: 0.76 MB/secConcurrency: 15.31Successful transactions: 20462Failed transactions: 0Longest transaction: 1.26Shortest transaction: 0.00
Much better. With a little bit of tuning, we were able to double our server's transaction rate.
Joe Dog's Sponsors pay the bills. Please take the time to visit them...
