FREQUENTLY ASKED QUESTIONS
Sproxy is the Siege PROXY server. It is designed to harvest URLs into a file that can be read by siege. To create a siege urls.txt file, just configure your browser to use sproxy then browse the website you want to test.
Sproxy is the Siege PROXY server. This program is a perl daemon that sits between your browser and a web server. After you start the sproxy server, you configure your browser to use it as a proxy. With sproxy as your proxy, all you need to do is click on the URLs you want to siege. Sproxy will harvest the URL along with all the page elements, style sheets, images, javascript, etc. BE CAREFUL. Sproxy will harvest external elements as well. Before you feed the URL list to siege, make sure you edit it. You don’t want to siege your partners.
Unfortunately, no. In order to support HTTPS, sproxy needs to establish a secure tunnel between itself and the web server. Thus far, we haven’t had the need to incorporate this feature. If you hack HTTPS support, please send us the code and we’ll get it into the base.
Where's Pom?
Jeff
Great product ! My question is how do you configure the browser to use the sproxy when its running locally ? Thanks
When sproxy comes up, it should tell you which host it’s bound to. That will either be the hostname or localhost. If no hostname is mentioned, that means its set to “any”. In that case, you should be able to use localhost.
NOTE: At some point I’ll delete these comments and make this part of the FAQ.
Jeff
Thanks for reply. But I am still confused … Lets say my apache server runs on localhost port 80. I will run >sproxy localhost…. Now I open up the browser and go to http://localhost and I see the apache home page. How would I route the http get through from browser through the sproxy ?
are you saying I select ‘sproxy’ and port 9001 in the PROXY tab in browser before I access the site?
Thanks
You have to configure your browser to use sproxy as your PROXY server. On Chrome, for example, it’s Wrench => Preferences => Under the hood => Change Proxy Settings…
Jeff
I have a load balancer which has my dev box as a node in a pool. Lets say the IP of the VIP is 10.10.10.10 . I tried running the sproxy against the host 10.10.10.10 but it gave an error in line 56…and I have no clue what it stands for….My question is does sproxy work with a VIP ?
Jeff
I started my sproxy on my mac as follows listening on default port to my local host (127.0.0.1)
GC-MNAIR-ML1:Documents mnair$ sproxy
SPROXY v1.02 listening on port 9001
…appending HTTP requests to: /Users/mnair/urls.txt
…default connection timeout: 120 seconds
In Chrome I change my web proxy(http) setting to SPROXY PORT 9001.
Then I go to Chrome and try http://127.0.0.1 and i get this proxy connection error.
Unable to connect to the proxy server
A proxy server is a server that acts as an intermediary between your computer and other servers. Right now, your system is configured to use a proxy, but Google Chrome can’t connect to it.
Here are some suggestions:
If you use a proxy server, check your proxy settings or contact your network administrator to make sure the proxy server is working. If you don’t believe you should be using a proxy server, adjust your proxy settings: Go to Applications > System Preferences > Network > Advanced > Proxies and deselect any proxies that have been selected.
Error 130 (net::ERR_PROXY_CONNECTION_FAILED): Proxy server connection failed.
Hello,
First of, you’ve done a great job with Sproxy. Thanks for this software.
We have added some patches to it in order to fix some issues ( get rid of dead children, setting custom output file, set the process run as a daemon – we needed that for our work ).
If you feel the code is right ( I’m still a beginner in Perl ) maybe you will add these patches in upcoming releases.
If you need the patches in another format, just contact me.
Regards,
Ovidiu
# diff -u src/sproxy.in.orig src/sproxy.in
— src/sproxy.in.orig 2012-07-12 23:42:41.000000000 -0700
+++ src/sproxy.in 2012-07-13 00:15:51.000000000 -0700
@@ -13,14 +13,14 @@
use JoeDog::Config;
use Getopt::Std;
use FileHandle;
-use vars qw($opt_V $opt_v $opt_f $opt_h $opt_p $opt_o $opt_t);
+use vars qw($opt_V $opt_v $opt_f $opt_h $opt_p $opt_o $opt_t $opt_P);
use vars qw($VERSION $PROGRAM $DATE);
use strict;
$VERSION = ‘%_VERSION%’;
$PROGRAM = ‘%_PROGRAM%’;
$DATE = ‘%_DATE%’;
-getopts(‘Vvhp:f:o:t:’);
+getopts(‘VvhpP:f:o:t:’);
my($ua, $conf, %conf, $port, $host);
if($opt_h){
@@ -40,17 +40,30 @@
$ua->env_proxy;
$ua->timeout(($conf{‘timeout’} eq int($conf{‘timeout’}))?$conf{‘timeout’}:120);
+ my $pid = fork;
+ defined ($pid) or die “Sproxy daemon running…..[ FAILED ]\n$!\n”;
+
+ if ($pid){
+ print “Started Sproxy daemon. ( $pid )\n”;
+ open (PID, “>”.$conf{‘pid’} ) or die “Can’t open PID file “.$conf{‘pid’}.” : $! \n”;
+ print PID “$pid”;
+ close ( PID );
+ exit 0;
+ }
+
+ $SIG{CHLD} = ‘IGNORE’;
+
my $port = ($conf{‘port’} eq int($conf{‘port’}))?$conf{‘port’}:9001;
my $parent = new HTTP::Daemon(
LocalAddr => $conf{‘host’},
LocalPort => $port,
- );
+ ReuseAddr => 1 ) or die “ERROR! Couldn’t start Sproxy service: $!\n”;
my $child;
my $FILE = open_file();
printf(“%s v%s listening on port %d\n”, uc($PROGRAM), $VERSION, $port);
- printf(“…appending HTTP requests to: %s\n”, $conf{‘file’});
+ printf(“…appending HTTP requests to: %s\n”, $conf{‘output’});
printf(“…default connection timeout: %s seconds\n”, $conf{‘timeout’});
while($child = $parent->accept){
connection_handler($FILE, $child);
@@ -63,6 +76,7 @@
parse_config()
{
my $file = $ENV{‘HOME’} . “/urls.txt”;
+ my $pid_file = $ENV{‘HOME’} . “/sproxy.pid”;
if($opt_f){
$conf = new JoeDog::Config($opt_f);
@@ -74,7 +88,8 @@
%conf = $conf->getHash(“=”);
$conf{‘host’} = ($ARGV[0])?$ARGV[0]:(($conf{‘host’})?$conf{‘host’}:”localhost”);
$conf{‘port’} = ($opt_p)?$opt_p:(($conf{‘port’})?$conf{‘port’}:9001);
- $conf{‘file’} = ($opt_o)?$opt_o:(($conf{‘file’})?$conf{‘file’}:$file);
+ $conf{‘output’} = ($opt_o)?$opt_o:(($conf{‘output’})?$conf{‘output’}:$file);
+ $conf{‘pid’} = ($opt_o)?$opt_P:(($conf{‘pid’})?$conf{‘pid’}:$pid_file);
$conf{‘timeout’} = ($opt_t)?$opt_t:(($conf{‘timeout’})?$conf{‘timeout’}:120);
$conf{‘verbose’} = ($opt_v)?”true”:(($conf{‘verbose’})?$conf{‘verbose’}:”false”);
}
@@ -137,10 +152,10 @@
my $FILE;
$FILE = new FileHandle();
- if($FILE->open(“>>”.$conf{‘file’})){
+ if($FILE->open(“>>”.$conf{‘output’})){
flock($FILE, $LOCK_EX);
} else {
- die $PROGRAM.” error: cannot open “.$conf{‘file’};
+ die $PROGRAM.” error: cannot open “.$conf{‘output’};
}
return $FILE;
}
# diff -u doc/sproxy.conf.in.orig doc/sproxy.conf.in
— doc/sproxy.conf.in.orig 2012-07-13 01:32:22.000000000 -0700
+++ doc/sproxy.conf.in 2012-07-13 00:12:22.000000000 -0700
@@ -48,3 +48,12 @@
# timeout = 90
# timeout =
+# this directive tells %_PROGRAM% to write daemon pid
+# to this alternative file. The default
+# file is $HOME/sproxy.pid You may also specify an
+# alternative pid file with the -P option on the
+# command line.
+# example: pid = /var/run/sproxy.pid
+#
+# pid =
+
# diff -u doc/sproxy.1.in.orig doc/sproxy.1.in
— doc/sproxy.1.in.orig 2012-07-13 01:33:43.000000000 -0700
+++ doc/sproxy.1.in 2012-07-13 01:37:46.000000000 -0700
@@ -48,6 +48,9 @@
\fB\-o FILE\fR
OUTPUT FILE, change the default output file from $HOME/urls.txt to FILE. This is the siege-friendly file of URLs.
.TP
+\fB\-P PID FILE\fR
+PID FILE, change the default pid file from $HOME/sproxy.pid to PID FILE.
+.TP
If %_PROGRAM% is invoked without a hostname, it will bind to the address given by the “host = ” directive in the %_PROGRAM%.conf file. If a host directive is not present in that file, then %_PROGRAM% will bind to localhost. To use %_PROGRAM% to create a urls.txt file, configure your web browser to use a proxy server at the address and port that %_PROGRAM% is listening for requests.
Also created a init/start-up script for this (tested only on CentOS):
# cat /etc/init.d/sproxy
#!/bin/bash
#
# Sproxy as daemon
#
# chkconfig: 2345 99 10
# description: Sproxy replaces Scout as the preferred method of URL harvest for Siege. It is an HTTP proxy server written in perl and designed to collect all URL information in a siege-friendly format. All necessary modules are bundled with the source. Sproxy is built with GNU autotools.
# processname: sproxy
#
#
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
RETVAL=0
PROG_BIN=”/usr/local/bin/sproxy”
PID_FILE=”/var/run/sproxy.pid”
PID=$( cat $PID_FILE 2>/dev/null)
status -p $PID_FILE >/dev/null 2>&1
running=$?
conf_check() {
[ -x $PROG_BIN ] || exit 5
}
start() {
[ "$EUID" != "0" ] && exit 4
# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 1
conf_check
# Start daemons.
echo -n $”Starting Sproxy daemon: ”
$PROG_BIN 2>/dev/null 1>&2
RETVAL=$?
if [ $RETVAL -eq 0 ]; then
success
else
failure $”$PROG_BIN start”
fi
echo
return $RETVAL
}
stop() {
[ "$EUID" != "0" ] && exit 4
conf_check
# Stop daemons.
echo -n $”Shutting down Sproxy daemon: ”
kill $PID
RETVAL=$?
[ $RETVAL -eq 0 ] && rm -f $PID_FILE
success
echo
return $RETVAL
}
# See how we were called.
case “$1″ in
start)
if [ $running -eq 0 ]; then
echo “Script already running ($PID) !”
failure
echo
exit 0
fi
start
;;
stop)
[ $running -eq 0 ] || exit 0 && failure
stop
;;
restart)
stop
start
;;
status)
status -p $PID_FILE
;;
condrestart)
[ $running -eq 0 ] || exit 0
stop
start
;;
*)
echo $”Usage: $0 {start|stop|restart|status|condrestart}”
exit 2
esac
exit $?
Hi Jeff,
Thanks for your great tools. I have a small patch which allows to not follow redirects inside sproxy. If you’re interested, please contact me. Thanks.