Sproxy FAQ

FREQUENTLY ASKED QUESTIONS

What Is It? ^

Sproxy is the Siege PROXY server. It is designed to harvest URLs into a file that can be read by siege. To create a siege urls.txt file, just configure your browser to use sproxy then browse the website you want to test.

How Does It Work? ^

Sproxy is the Siege PROXY server. This program is a perl daemon that sits between your browser and a web server. After you start the sproxy server, you configure your browser to use it as a proxy. With sproxy as your proxy, all you need to do is click on the URLs you want to siege. Sproxy will harvest the URL along with all the page elements, style sheets, images, javascript, etc. BE CAREFUL. Sproxy will harvest external elements as well. Before you feed the URL list to siege, make sure you edit it. You don’t want to siege your partners.

Does Sproxy Work With HTTPS? ^

Unfortunately, no. In order to support HTTPS, sproxy needs to establish a secure tunnel between itself and the web server. Thus far, we haven’t had the need to incorporate this feature. If you hack HTTPS support, please send us the code and we’ll get it into the base.

 

Posted in | 8 Comments

8 Responses to “Sproxy FAQ”

  1. M Nair says:

    Jeff

    Great product ! My question is how do you configure the browser to use the sproxy when its running locally ? Thanks

    • Jeff Fulmer says:

      When sproxy comes up, it should tell you which host it’s bound to. That will either be the hostname or localhost. If no hostname is mentioned, that means its set to “any”. In that case, you should be able to use localhost.

      NOTE: At some point I’ll delete these comments and make this part of the FAQ.

  2. m nair says:

    Jeff

    Thanks for reply. But I am still confused … Lets say my apache server runs on localhost port 80. I will run >sproxy localhost…. Now I open up the browser and go to http://localhost and I see the apache home page. How would I route the http get through from browser through the sproxy ?

    are you saying I select ‘sproxy’ and port 9001 in the PROXY tab in browser before I access the site?

    Thanks

    • Jeff Fulmer says:

      You have to configure your browser to use sproxy as your PROXY server. On Chrome, for example, it’s Wrench => Preferences => Under the hood => Change Proxy Settings…

  3. M Nair says:

    Jeff

    I have a load balancer which has my dev box as a node in a pool. Lets say the IP of the VIP is 10.10.10.10 . I tried running the sproxy against the host 10.10.10.10 but it gave an error in line 56…and I have no clue what it stands for….My question is does sproxy work with a VIP ?

  4. M Nair says:

    Jeff

    I started my sproxy on my mac as follows listening on default port to my local host (127.0.0.1)

    GC-MNAIR-ML1:Documents mnair$ sproxy
    SPROXY v1.02 listening on port 9001
    …appending HTTP requests to: /Users/mnair/urls.txt
    …default connection timeout: 120 seconds

    In Chrome I change my web proxy(http) setting to SPROXY PORT 9001.

    Then I go to Chrome and try http://127.0.0.1 and i get this proxy connection error.

    Unable to connect to the proxy server
    A proxy server is a server that acts as an intermediary between your computer and other servers. Right now, your system is configured to use a proxy, but Google Chrome can’t connect to it.
    Here are some suggestions:
    If you use a proxy server, check your proxy settings or contact your network administrator to make sure the proxy server is working. If you don’t believe you should be using a proxy server, adjust your proxy settings: Go to Applications > System Preferences > Network > Advanced > Proxies and deselect any proxies that have been selected.
    Error 130 (net::ERR_PROXY_CONNECTION_FAILED): Proxy server connection failed.

  5. Ovidiu says:

    Hello,

    First of, you’ve done a great job with Sproxy. Thanks for this software.

    We have added some patches to it in order to fix some issues ( get rid of dead children, setting custom output file, set the process run as a daemon – we needed that for our work ).
    If you feel the code is right ( I’m still a beginner in Perl ) maybe you will add these patches in upcoming releases.

    If you need the patches in another format, just contact me.

    Regards,
    Ovidiu

    # diff -u src/sproxy.in.orig src/sproxy.in
    — src/sproxy.in.orig 2012-07-12 23:42:41.000000000 -0700
    +++ src/sproxy.in 2012-07-13 00:15:51.000000000 -0700
    @@ -13,14 +13,14 @@
    use JoeDog::Config;
    use Getopt::Std;
    use FileHandle;
    -use vars qw($opt_V $opt_v $opt_f $opt_h $opt_p $opt_o $opt_t);
    +use vars qw($opt_V $opt_v $opt_f $opt_h $opt_p $opt_o $opt_t $opt_P);
    use vars qw($VERSION $PROGRAM $DATE);
    use strict;

    $VERSION = ‘%_VERSION%';
    $PROGRAM = ‘%_PROGRAM%';
    $DATE = ‘%_DATE%';
    -getopts(‘Vvhp:f:o:t:’);
    +getopts(‘VvhpP:f:o:t:’);
    my($ua, $conf, %conf, $port, $host);

    if($opt_h){
    @@ -40,17 +40,30 @@
    $ua->env_proxy;
    $ua->timeout(($conf{‘timeout’} eq int($conf{‘timeout’}))?$conf{‘timeout’}:120);

    + my $pid = fork;
    + defined ($pid) or die “Sproxy daemon running…..[ FAILED ]\n$!\n”;
    +
    + if ($pid){
    + print “Started Sproxy daemon. ( $pid )\n”;
    + open (PID, “>”.$conf{‘pid’} ) or die “Can’t open PID file “.$conf{‘pid’}.” : $! \n”;
    + print PID “$pid”;
    + close ( PID );
    + exit 0;
    + }
    +
    + $SIG{CHLD} = ‘IGNORE';
    +
    my $port = ($conf{‘port’} eq int($conf{‘port’}))?$conf{‘port’}:9001;

    my $parent = new HTTP::Daemon(
    LocalAddr => $conf{‘host’},
    LocalPort => $port,
    – );
    + ReuseAddr => 1 ) or die “ERROR! Couldn’t start Sproxy service: $!\n”;

    my $child;
    my $FILE = open_file();
    printf(“%s v%s listening on port %d\n”, uc($PROGRAM), $VERSION, $port);
    – printf(“…appending HTTP requests to: %s\n”, $conf{‘file’});
    + printf(“…appending HTTP requests to: %s\n”, $conf{‘output’});
    printf(“…default connection timeout: %s seconds\n”, $conf{‘timeout’});
    while($child = $parent->accept){
    connection_handler($FILE, $child);
    @@ -63,6 +76,7 @@
    parse_config()
    {
    my $file = $ENV{‘HOME’} . “/urls.txt”;
    + my $pid_file = $ENV{‘HOME’} . “/sproxy.pid”;

    if($opt_f){
    $conf = new JoeDog::Config($opt_f);
    @@ -74,7 +88,8 @@
    %conf = $conf->getHash(“=”);
    $conf{‘host’} = ($ARGV[0])?$ARGV[0]:(($conf{‘host’})?$conf{‘host’}:”localhost”);
    $conf{‘port’} = ($opt_p)?$opt_p:(($conf{‘port’})?$conf{‘port’}:9001);
    – $conf{‘file’} = ($opt_o)?$opt_o:(($conf{‘file’})?$conf{‘file’}:$file);
    + $conf{‘output’} = ($opt_o)?$opt_o:(($conf{‘output’})?$conf{‘output’}:$file);
    + $conf{‘pid’} = ($opt_o)?$opt_P:(($conf{‘pid’})?$conf{‘pid’}:$pid_file);
    $conf{‘timeout’} = ($opt_t)?$opt_t:(($conf{‘timeout’})?$conf{‘timeout’}:120);
    $conf{‘verbose’} = ($opt_v)?”true”:(($conf{‘verbose’})?$conf{‘verbose’}:”false”);
    }
    @@ -137,10 +152,10 @@
    my $FILE;

    $FILE = new FileHandle();
    – if($FILE->open(“>>”.$conf{‘file’})){
    + if($FILE->open(“>>”.$conf{‘output’})){
    flock($FILE, $LOCK_EX);
    } else {
    – die $PROGRAM.” error: cannot open “.$conf{‘file’};
    + die $PROGRAM.” error: cannot open “.$conf{‘output’};
    }
    return $FILE;
    }

    # diff -u doc/sproxy.conf.in.orig doc/sproxy.conf.in
    — doc/sproxy.conf.in.orig 2012-07-13 01:32:22.000000000 -0700
    +++ doc/sproxy.conf.in 2012-07-13 00:12:22.000000000 -0700
    @@ -48,3 +48,12 @@
    # timeout = 90
    # timeout =

    +# this directive tells %_PROGRAM% to write daemon pid
    +# to this alternative file. The default
    +# file is $HOME/sproxy.pid You may also specify an
    +# alternative pid file with the -P option on the
    +# command line.
    +# example: pid = /var/run/sproxy.pid
    +#
    +# pid =
    +

    # diff -u doc/sproxy.1.in.orig doc/sproxy.1.in
    — doc/sproxy.1.in.orig 2012-07-13 01:33:43.000000000 -0700
    +++ doc/sproxy.1.in 2012-07-13 01:37:46.000000000 -0700
    @@ -48,6 +48,9 @@
    \fB\-o FILE\fR
    OUTPUT FILE, change the default output file from $HOME/urls.txt to FILE. This is the siege-friendly file of URLs.
    .TP
    +\fB\-P PID FILE\fR
    +PID FILE, change the default pid file from $HOME/sproxy.pid to PID FILE.
    +.TP

    If %_PROGRAM% is invoked without a hostname, it will bind to the address given by the “host = ” directive in the %_PROGRAM%.conf file. If a host directive is not present in that file, then %_PROGRAM% will bind to localhost. To use %_PROGRAM% to create a urls.txt file, configure your web browser to use a proxy server at the address and port that %_PROGRAM% is listening for requests.

    Also created a init/start-up script for this (tested only on CentOS):

    # cat /etc/init.d/sproxy

    #!/bin/bash
    #
    # Sproxy as daemon
    #
    # chkconfig: 2345 99 10
    # description: Sproxy replaces Scout as the preferred method of URL harvest for Siege. It is an HTTP proxy server written in perl and designed to collect all URL information in a siege-friendly format. All necessary modules are bundled with the source. Sproxy is built with GNU autotools.
    # processname: sproxy
    #
    #

    # Source function library.
    . /etc/rc.d/init.d/functions

    # Source networking configuration.
    . /etc/sysconfig/network

    RETVAL=0
    PROG_BIN=”/usr/local/bin/sproxy”
    PID_FILE=”/var/run/sproxy.pid”
    PID=$( cat $PID_FILE 2>/dev/null)

    status -p $PID_FILE >/dev/null 2>&1
    running=$?

    conf_check() {
    [ -x $PROG_BIN ] || exit 5
    }

    start() {
    [ "$EUID" != "0" ] && exit 4
    # Check that networking is up.
    [ ${NETWORKING} = "no" ] && exit 1
    conf_check
    # Start daemons.
    echo -n $”Starting Sproxy daemon: ”
    $PROG_BIN 2>/dev/null 1>&2
    RETVAL=$?

    if [ $RETVAL -eq 0 ]; then
    success
    else
    failure $”$PROG_BIN start”
    fi

    echo
    return $RETVAL
    }

    stop() {
    [ "$EUID" != "0" ] && exit 4
    conf_check
    # Stop daemons.
    echo -n $”Shutting down Sproxy daemon: ”
    kill $PID
    RETVAL=$?
    [ $RETVAL -eq 0 ] && rm -f $PID_FILE
    success
    echo
    return $RETVAL
    }

    # See how we were called.
    case “$1″ in
    start)
    if [ $running -eq 0 ]; then
    echo “Script already running ($PID) !”
    failure
    echo
    exit 0
    fi
    start
    ;;
    stop)
    [ $running -eq 0 ] || exit 0 && failure
    stop
    ;;
    restart)
    stop
    start
    ;;
    status)
    status -p $PID_FILE
    ;;
    condrestart)
    [ $running -eq 0 ] || exit 0
    stop
    start
    ;;
    *)
    echo $”Usage: $0 {start|stop|restart|status|condrestart}”
    exit 2
    esac

    exit $?

  6. Magentron says:

    Hi Jeff,

    Thanks for your great tools. I have a small patch which allows to not follow redirects inside sproxy. If you’re interested, please contact me. Thanks.

Leave a Reply




Recent Comments

  • program antyplagiatowy: Definitely consider that which you stated. Your favourite justification seemed to be at the...
  • Mike Smith: I find that Dunning-Kruger explains a lot. The difficult part is getting help to those that suffer from...
  • Sohan: I am using node-js http server. I created http request to hit the server and log the message. In that case...
  • Jeff Fulmer: I don’t know what “simple http server” means. If you’re using apache out of the...
  • Sohan: MY server is simple http server. Can you tell me what thresol paramter or property is used in siegerc file.