SubWeb v1.0

Stephane Aubert <Stephane.Aubert@hsc-labs.com>
kotao <kotao@kotao.org>
HSC security research labs
Hervé Schauer Consultants

Download: http://www.hsc-labs.com/tools/subweb/


Description
===========

SubWeb is a proxy (and also a reverse proxy). It allows to work on
HTTP flows in the line of HTTPush, RFProxy or Achilles.

It becomes possible with SubWeb to handle and visualize on the fly the
HTTP requests, the headers and/or HTML pages.

Main goal of SubWeb is to contribute to the tests of network applications
based on HTTP. HTTPS is not directly managed in SubWeb, it is necessary, 
in order to test a SSL server, to use the stunnel program, for example.

SubWeb has 3 operating modes:

* proxy (classical HTTP proxy)

  /--------\ Request      /--------\               /--------\ 
  | Client |------------->| SubWeb |-------------->| Server |
  |        |<-------------| proxy  |<--------------|        |
  \--------/              \--------/        Answer \--------/
 

* midproxy (HTTP proxy which requires the pages of another proxy)
 
  /--------\ Req.  /--------\     /--------\       /--------\ 
  | Client |------>| SubWeb |---->|  HTTP  |------>| Server |
  |        |<------|midproxy|<----|  proxy |<------|        |
  \--------/       \--------/     \--------/   Ans.\--------/
 

* rproxy (reverse proxy, SubWeb mimic an HTTP server)

  /--------\              /--------\       Request /--------\ 
  | Server |<-------------| SubWeb |<--------------| Client |
  |        |------------->| rproxy |-------------->|        |
  \--------/ Answer       \--------/               \--------/


Virtual Web
-----------
Another functionality, named virtual Web, allow SubWeb to answer 
certain requests (depending on keywords contained in these
requests) without requiering anything to the server.

  /--------\ Request      /--------\               /--------\ 
  | Client |------------->| SubWeb |               | Server |
  |        |<-------------| proxy  |               |        |
  \--------/       Answer \--------/               \--------/

VirtualWeb can be configured with $virtual_web to activate this "module"
and the hash-table %vweb_config to provide actions according to key words.

The actions available for VirtualWeb are

  . send a file from disk,   use file:/tmp/foo
  . send a file from string, use html:<html><head><title> ...
  . send an HTTP redirect,   use  redirect:http://www.hsc.fr/

The example provides:  if SubWeb finds the word 'redirect' in the URL
it will redirect the client on http://www.hsc.fr/


Traffic visualization
---------------------

It is possible to visualize all the traffic between the customers and the servers.
There are several options of visualization (only the headers, to display
binary pages in Hexa, to display only the requests or only the answers...)

  Exemple de visualisation :
  --------------------------

  Connected to the Internet,
    run : ./subweb

  Via a proxy,
    run : ./subweb 8080 midproxy corporate_proxy 3128

  Configure the 'proxy' in your browser on localhost:3128

  Go to: http://www.google.com/

  Results:
  --------
  [Fri Aug 10 16:32:59 2001: connection from localhost [ 127.0.0.1 ] at port 45929]

  GET http://www.google.com/ HTTP/1.0
  User-Agent: Mozilla/5.0 (Linux 2.4.4 i686; U) Opera 5.0  [en]
  Host: www.google.com
  Accept: text/html, image/png, image/jpeg, image/gif, image/x-xbitmap, */*
  Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
  Coockie: NO-COOKIE-PLEASE:NO-COOKIE-PLEASE:NO-COOKIE-PLEASE:NO-COOKIE-PLEASE
  Cookie2: $Version="1"
  Pragma: no-cache
  Cache-Control: no-cache

  --
  HTTP/1.0 200 OK
  Date: Fri, 10 Aug 2001 14:36:13 GMT
  Server: GWS/1.11
  Set-Cookie: PREF=ID=1253841f47eb7ec4:TM=997454173:LM=997454173; \
              domain=.google.com; path=/; expires=Sun, 17-Jan-2038 19:14:07 GMT
  Content-Type: text/html
  Content-Length: 2184
  Cache-Control: private
  X-Cache: MISS from carbone.hsc.fr
  Proxy-Connection: close


Filtering
---------

In the three modes it is possible to apply filtering at all the
levels, ie. in the URL, the headers and the body of the pages, in the
requests and the answers.

Filtering is active if the variable $static_filters is not null.
Another type of filtering, named dynamic, is activable by adding the
string subweb=on in the URL. These dynamic filters are interesting, for 
example, to change fields like a cookie or a session_id in a session 
after passing the authentification.

Static and dynamic filtering is configured in the functions:
FilterIN, FilterOUT, DynamicFilterIN and DynamicFilterOUT.

                          FilterIN(),
                       DynamicFilterIN()
                             ^  |
                             |  v
  /--------\    Request   /--------\               /--------\ 
  |        |------------->|        |-------------->|        | 
  | Client |              | SubWeb |               | Server |
  |        |<-------------|        |<--------------|        |
  \--------/              \--------/      Answer   \--------/
                             ^  |
                             |  v
                          FilterOUT(),
                       DynamicFilterOUT()

  Example of filtering function:
  ------------------------------

  sub FilterIN {
    my $request = shift;
    ## Put your filters here - from client to server ##

    ## Change the user-agent
    $request =~ s/^User-Agent:\s+\S+$/User-Agent: SubWeb/gm;

    # Trivial anti-ads
    $request =~ s/\.doubleclick\.net/xxx/g;

    # Cookie stuff
    # Don't send coockies    
    # $request =~ s/^Cookie:\s+.*\r\n//gm;
    # activism ;) 
    $request=~s/Cookie:\s+.*\r\n/sprintf("Cookie: %s\r\n","NO-COOKIE:"x5)/egm;

    return $request;
  }

The filtering functions are thus implemented to make it possible to
the user to develop his own filters without any restriction.


Modification of requests GET in POST
-------------------------------------

SubWeb makes it possible to transform all requests GET into POST 
automatically and recomputes the Content-length.


Force or change authentification HTTP
-------------------------------------

It is possible with $force_auth and $http_auth to force or replace
HTTP authentification thus in a transparent way several users can use
the proxy which authenticates all the requests on another proxy or on
a Web server.
    
Test intrusion detection systems (IDS)
--------------------------------------

SubWeb has six modes making it possible to make illegible the URL.

With : $anti_ids_modes = '1523' the request GET /fr.yahoo.com/ becomes:

  GET  http://fr.yahoo.com/%2e/%76%7a%73%78%74%69%66%67%64%68%78%6e
  %73%6a%6f%7a%67%77%68%78%6e%6b%79%7a%70%6a%68%66%6b%65%76%77%65%61
  %63%65%68%6e%63%78%75%73%72%68%66%72%70%75%79%66%64%79%70%75%62%65
  %77%66%6f%6c%63/%2e/%2e%2e/%2e/


Encryption of the fields hidden in reverse proxy mode
-----------------------------------------------------

An experimental functionality was added to the reverse relay mode.
It makes it possible to cipher the contents of the hidden fields sent 
by the server to the various customers and to decipher them when the
customers send them back to the server in requests GET or POST requests.

This mechanism forbidden the users to modify the value of the hidden 
fields, which makes it possible to protect them and use them for
example to manage the order of the requests required by a customer.

With such a mechanism it can be very difficult to crack applications 
based on HTTP. It is activated when the variable $cypher_hidden_field 
is nonnull and when the mode is reverse proxy.

The encryption algorithm used is Blowfish provided in the
Crypt::Blowfish module.

The encryption key can be provided in the variable $passphrase_hidden 
and must contains 32 hexa char. If it is not provided SubWeb will use 
a pseudo-random one.


Syntax
======

By defect, SubWeb listens on the port 8080/tcp and is in 'midproxy' mode
on the address localhost and the port 3128/tcp. ie. that SubWeb awaits 
requests of your navigator on port 8080 and tries to send them on your 
local squid relay.

To change the port on which SubWeb is put in listening it is necessary
to launch it with a first parameter, example for port 8081:

% ./subweb 8081

To change the mode it is necessary to add a second parameter:  

% ./subweb 8081 [ proxy | midproxi | rproxy ]

By defect the server and the distant port are localhost and 3128/tcp.

To change this server and this port 2 other parameters should be added:

% ./subweb 8081 rproxy webserver 80

Or:
% ./subweb 8080 proxy

Or:
% ./subweb 8081 midproxy httproxy 8080



Options
=======

my $signature            = 1;   # to activate the SubWeb signature
my $convert_get_to_post  = 1;   # to activate the transformation of the GET into POST
my $force_auth           = 1;   # to force http authenticication to base64($http_auth)
my $http_auth = "aubert:kotao"; # to give an http authentication login:passwd

my $anti_ids_modes  = '1523';   # to activate the anti-ids modes
my $accept_gzipped_pages = 1;   # allows to authorize the server to compress the answers
my $static_filters       = 1;   # to activate static filtering

my $show_only_url        = 1; # to show URL only in requests
my $dump_text_body       = 1; # to show the contain of text pages
my $dump_binary_body     = 1; # to show the contain of binary pages (in hexa)
my $showIN               = 1; # to show requests
my $showOUT              = 1; # to show answers
my $showUNFILTERED       = 1; # to show requests and answers before filtering 
my $showFILTERED         = 1; # to show requests and answers after filtering 


PS: SubWeb is not underground ;)
--------------------------------
