  netrik hacker's manual
>========================<

[This file contains a description of the file loading module. See hacking.txt
or hacking.html for an overview of the manual.]

_load.c_

Every open input resource has a descriptor associated with it. The descriptor
is created when opening the with init_load().

The struct for this descriptor is declared in load.h, and contains:
- The URL of the resource is stored by "url", which points to a "struct Url"
  containing the (split up) effective URL
- "type" is an enum of type "Res_type", describing what kind of resource it is.
  (RES_STDIN for stdin, RES_FILE for local files, RES_HTTP for HTTP connections
  handled by builtin HTTP code, or RES_PIPE for piping data from wget; it may
  also be set to RES_FAIL during the loading process, indicating that the
  loading failed)
- The union "handle" contains data necessary to handle the resource itself,
  e.g. a file handle.
- "buf" is a pointer to a string, where every loaded data block is stored for
  processing.
- "buf_end" contains the current end of the data in the buffer. (Not always the
  whole buffer is filled, escpecially at EOF.)
- "buf_ptr" points to the current read position inside the buffer.
- The "user_break" flag indicates when loading was interrupted by a user break
  and the document wasn't loaded completely.

_init_load()_

init_load() takes two arguments: a string containing the target URL to be
loaded, and a "struct Url" containing a base URL. (Normally the URL of the
file visible in the pager up to now.) Besides of creating the descriptor it
also allocs the "buf[]" and prepares the resource for reading data.

An additional "form" argument can be given to submit the data of an HTML form
along with the HTTP request. See _Submitting_ in hacking-links.* for details
on this.

Calling this function with only a base URL and no target means that a page
is to be reloaded from history.

For all other target URLs, first the effective URL is determined. This is
done by _merge_urls()_ , which first splits the URL up into its components
using _split_url()_ , and then merges it with the base URL components. A
pointer to the resulting merged URL ist stored in "res->url".

If merging the URLs failed for some reason, the resource type is set to
"RES_FAIL", and init_load() returns normally. Thus, there is no special error
handling necessary in the calling function; all necessary handling is done
in _load()_ .

If the URL is "-", meaning load from stdin, "res->url->proto.type" is set to
PT_INTERNAL (meaning no relative links are allowed, and the page is not to
be kept in history).

No URL merging is done if a page from history is reloaded. The split URL
passed as the base is already the destination URL.

Now having the destination URL, the resource is opened appropriately, depending
on what resource type is set in the URL.

If it is a HTTP URL, by default http_init_load() is used to open a connection
to the server and read the HTTP headers. "res->type" is set to HTTP.

Alternatively, "cfg.wget" may be set (using --no-builtin-http), in which case
"wget" is used instead of the builtin HTTP code.  This is done by init_wget(),
which starts wget with the correct filename, and initiates reading its standard
output through a pipe. The popen() function returns our pipe end as a stream;
we use this as the input.  "res->type" is set to "RES_PIPE".

FTP URLs are always fetched using wget.

Local files are simply opened as the input stream, and "res->type" is set to
"RES_FILE".

Internal URLs presently always mean loading from stdin. (Error pages also use
"PT_INTERNAL", but init_load() returns immediataly after setting it, so this
needn't be handled in the switch.) The standard input is reopened as a stream,
and this one is used as the input. As we need some way to read user commands
from the terminal, "stderr" is reopend in place of "stdin". (Some programs
reopen "/dev/tty" instead -- no idea which one is better...) This way the
normal "stdin" descriptor points to a terminal again, while the input file
is read from the pipe that is (hopefully) connected to the original standard
input. "res->type" is set to "RES_STDIN".

Things get a bit more complicated if there was no protocol specification
in the URL, and no base was supplied (meaning the URL has to be treated as
an absolute URL in any case). In this case we have to guess whether it is a
local file or an HTTP URL.

First we try to open a local file with the path returned by merge_urls().
(Should be identical to the given URL.) If this succeeds, we set the protocol
to "PT_FILE" and the resource type to "RES_FILE", and that's it.

If opening the local file fails, and the URL string doesn't start with '/',
the URL is assumed to be an HTTP URL. As these URLs are split in another way,
we have to prepend the "http://" to the URL string and call merge_urls() again.
Afterwards, we proceed just as with any other HTTP URL.

After opening an HTTP connection (regardless whether it was PT_HTTP or
PT_UNKNOWN recognized as HTTP), we need to check for HTTP redirects. If a
redirect is set (a "Location:" header exists), the loading has to be repeated
with the redirect URL. The new "main_url" is taken from the "Location:" value;
the "base_url" is set to "res->url", i.e. the original absolute target URL,
so the redirect will be relative to that; the "form" is also cleared, as
redirects always use a simple GET. With this new URL(s), the whole process
is repeated.  (Except for allocation of the resource descriptor, which is
reused.) In this reiteration the protocol type always has to be PT_HTTP;
redirections to other resource types are not allowed. All this can happen up
to five times; when more, it's probably a redirection loop, and we abort.

If opening a file fails, init_load() returns immediately, only setting
"res->type" to "RES_FAIL" and "res->url->proto" to "PT_INTERNAL". "RES_FAIL"
is then handled appropriately in load(); this way, the caller needn't
bother about it. (Except an additional error message and setting an
error code in parse_syntax(); see _Warning Mesages_ in hacking-layout.*
for that.) "PT_INTERNAL" means that the page isn't to be kept in history,
that following relative URLs isn't allowed etc. Thus, the link/page history
handling functions also do not need special handling for the error pages.

HTTP loading errors inside http_init_load() are handled the same way.

_load()_

Data is read by calling load() with the resource descriptor returned by
init_load().

Every call to load() reads one data block (of size BUF_SIZE) into a buffer. The
reading function (_parse_syntax()_ or _parse_header()_ ) then processes
the data, keeping track of the current read position inside the buffer by
"res->buf_ptr"; when it reaches the end of the data block, load() is called
again to read the next block.

If "res->type" is "RES_HTTP", read() is used to read a data block from the
socket; otherwise, fread() is used to read data from the input stream. (It
doesn't matter if this stream is a normal file (RES_FILE), stdin (RES_STDIN),
or a pipe (RES_PIPE).)

"RES_FAIL" means that opening the input resource failed for some reason,
or an error emerged in a previous load() call. In this case, no data is
read; an empty buffer is returned, which normally would mean EOF. This causes
parse_syntax() to generate an empty page, or stops parsing at this point if some
data already has been read before the error occured. For the latter case, some
(little) additional handling is necessary in parse_syntax(), to ensure that
an appropriate error message is printed and an error code returned to main().
(Causing a keywait before starting the pager.)

The data block is stored in the "buf[]" referenced by the descriptor, and
"buf_end" is set to the end of the data inside the buffer; "buf_ptr" is set
to the beginning of the buffer.

_uninit_load()_

uninit_load() is used to close the input stream, and free the memory used to
read the file.

If the input was a pipe created by popen() (RES_PIPE), it needs to be closed
with pclose() instead of fclose(). This function also passes the exit code from
wget, which is necessary to decide whether the load via wget was sucessfull.

If the input was a HTTP socket, close() is used instead of fclose().

After closing the stream, the input buffer and the "res" struct are freed.

_url.c_

This file contains a couple of functions for handling of URLs, which are used
chiefly by the file loading module.

_split_url()_

To allow operating on the URLs and loading the addressed files, the URL string
given by the user or a link needs to be split up into components. split_url()
parses the URL string, and returns the components by a pointer to a newly
allocated "struct Url".

The Url struct is used for all following processing steps. It contains the
following data:
- The protocol (resource type) specification string in "proto.str"
- The protocol type as an "enum Protocol" in "proto.type"
- The host name of a HTTP or FTP URL "host"
- The port number (HTTP/FTP) "port"
- The complete directory name "dir" (path without file name)
- The file name in "name"
- The parameters for CGI scripts in "params"
- The fragment identifier (anchor) in "frag"
- "full_url", which stores a complete URL string (but for the fragment
  identifier)
- "path", which stores a part of the complete URL, consisting of directory, file
  name, and CGI parameters
- for merged URLs, the "absolute" flag tells whether the URL was generated from
  an absolute URL, or was relative to a previous page (necessary to decide
  the document was from the same site when moving through page history)
- for merged URLs, the "local" flag tells whether the URL points to a local
  anchor (in the same document)

The parser is very similar to the HTML parser in parse_syntax.c (see _Parsing_
in hacking-layout.*): The URL string is processed char by char in a loop. In
every iteration, one char is examined, and action is taken (in a switch
statement) depending on what character it is, and in what mode the parser
currently is.

There is a parsing mode for every URL component. (There are also some additional
ones for constructs like the "://" after the protocol specification.) Every
time we encounter some special character seperating different components,
the mode is switched to the one fitting the next component, and its beginning
position (normaly the char after the one causing the mode change) is stored in
"word_start". Everything between the previous "word_start" and the current
char is stored to the respective split URL field of the component parsed up
to now, using store_component().

url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
             ^     ^url_char
             word_start
parse_mode: PM_HOST

url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
           url_char^^
                    word_start
parse_mode: PM_PORT
components->host: "domain"

If the separating char is not the one that would introduce the component
normally following now, that means that component is missing, and we immediately
have to proceed with the next one.

url: "http://domain/dir1/dir2/name.ext#fragment"
                              url_char^^
                                       word_start
parse_mode: PM_PARAMS

This is done by setting "recycle", meaning that the current character is to be
parsed again. In this new iteration the parser will see the separating char
again, thus introducing a second mode change. store_component() will store
a NULL string for that component, and the parser will go on with parsing the
next one.

url: "http://domain/dir1/dir2/name.ext#fragment"
                              url_char^^
                                       word_start
parse_mode: PM_FRAG
components->params: NULL

The first mode change is a bit more tricky: At the beginning of the URL, we do
not know if it is a full qualified one (starting with a protocol specification),
or a relative URL without a protocol. At first we assume that it starts with
the protocol. If a ":" follows the first word, our guess was right, and we
proceed normally with the host.

url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
  url_char^^
           word_start
parse_mode: PM_PROTO_END1
components->proto.str: "http"

If any other separating char occurs instead, we have to skip protocol, host,
and port, and switch immediately to path parsing. We also do a "recycle"
then, as the current char needs to be parsed in "PM_PATH" mode.

url: "dir1/dir2/name.ext?params#fragment"
          ^url_char
      ^word_start
parse_mode: PM_PATH

Path parsing is also a bit more complicated, as the directory name and the
file name are stored separately. For that purpose, while parsing the path
we keep track where the last '/' was (in "name_start"); everything before it
(inclusive) belongs to the directory, and what follows it is the file name.

url: "http://domain:80/dir1/dir2/name.ext?params#fragment"
                      ^          ^       ^url_char
             word_start          name_start
                      |<- dir ->|
                                 |<name>|
parse_mode: PM_PARA
components->dir: "/dir1/dir2/"
components->name: "name.ext"

"full_url" and "path" aren't filled in split_url(), as this URL won't be used
directly; they are only necessary for the final URL created in _merge_urls()_
. The "absolute" and "local" flags are also set only in merge_urls().

If an error occurs during URL parsing (either the protocol specification
contains an unknown protocol type, or an unexpected character is encountered),
split_url() sets "proto.type" to "PT_INTERNAL" and immediately returns. As
"PT_INTERNAL" normally can't ever be generated in split_url(), the caller
knows it was an error by this. Misusing "PT_INTERNAL" for this is surely
a bit confusing; it has a big advantage, though: The created (empty) pages
are correctly handled as temporary by all the page history functions without
needing any exception handling.

_merge_urls()_

When loading a relative URL using the ":e" command, or when following a link,
the absolute URL of the target needs to be determined by combining the URL
of the current page with the given relative URL; and for both relative and
absolute URLs, fields not given in the URL(s) need to be set to default values.
All that is done by merge_urls().

This function takes a base URL supplied as a "Url" struct (already split
up), and a main url given as string and split up from within merge_urls();
it returns a newly allocated "Url" struct pointer. Base being NULL means to
treat the main URL as an absolute one.

merge_urls() also takes an optional "form" parameter. This string contains
the (URL encoded) data of a form to be submitted to the HTTP server as part
of the URL. ("GET" method.) If present, it is stored in the resulting URL in
place of any other CGI parameters.

But for a few exeptions, merging is done component by component. If a
component is present in the main URL, it is taken from there; if it's
not, either it is taken from "base_url", or a default value is used if no
"base_url" is given. After the first component was taken from "main_url",
"base_url" is no longer used; all following components have to be specified,
or default values are taken. This is achieved by setting "base_url" to NULL.

The "port" component has no test on its own -- it is always taken from where
the "host" is taken.

The handling of "dir" is a bit more complicated: If "main_url" contains a
relative "dir" (not starting with '/'), and a "base_url" is given, the new
"dir" has to be created by joining both. (Concatenating the one in "main_url"
to the one in "base_url".) There is also an exeption about the default value
if nothing was supplied: For local (or unknown) URLs there is no default dir
(the current directory is used), while HTTP and other use the root dir ('/')
as default.

Handling of CGI parameters works as the other components, except that if
"form" is present, this is stored as "url->params", and both the params in
the main and base URLs are ignored. The form data string is stored just as
it were part of the main URL -- the string is copied and "base_url" is set
to NULL. This ensures that form submits are always handled as new documents
("local" flag not set), even if the submit URL is otherwise identical to the
current URL, and that any fragment identifier from the old URL is discarded.

After merging the URLs, the "full_url" and "path" components have to be set.
The "full_url" is created by concatenating all components (and separators),
except for the fragment identifier. (Concatenating is done by the str_append()
function created for that. Maybe we should try to use asprintf() or something
instead?) "path" is simply a pointer to the starting position of the "dir"
(and/or following) component(s) inside the "full_url" string. Thus it contains
the directory, file name, and CGI paramters.

The "absolute" flag is determined after merging the protocol specification. If
"base_url" is NULL at this point, we know we have an absolute URL: Either it was
NULL from the beginning (meaning absolute in any case), or it was reset when
merging the protocol, because "main_url" contains a protocol specification --
also meaning an absolute URL.

Setting the "local" flag is very similar: If, after all components except
the fragment identifier have been merged, "base_url" is still not NULL,
we know that "main_url" didn't contain any components up to now; it *can*
only consist of a fragment identifier, meaning it references a local anchor.

If an error occured during URL splitting, merge_urls() only prints an additional
error message, and proceeds normally. The result is that the "PT_INTERNAL"
indicating the error is stored in the merged URL, and can be handled by the
caller, just as for split_url() itself.

_free_url()_

This function destroys a split URL structure by freeing the memory used by
all the component strings, and afterwards the struct itself.

_http.c_

The functions for handling HTTP resources are a bit more complicated, and
reside in a source file on their own.

http_init_load() is called from init_load(). It prepares the handle, and opens
the HTTP connection. This includes looking up the IP address, opening a socket,
connecting to the desired server, creating and sending a HTTP request for
the desired page, and reading/parsing the HTTP header of the file returned
by the server.

Most of the work is done in get_http_socket(), which is called from
http_init_load() after creating the HTTP handle.

First, this function checks for a possible proxy to send the request to.

Having this, it creates the connect structure "sap", and then looks up the
IP address of the connect server -- which is the proxy if one is present,
or the target host otherwise. This, among other data, is stored in "sap".

The next step is opening a socket using socket(), and then establishing a
TCP connection to the desired server using the "sap" structure prepared before.

Finally, the HTTP request is constructed in get_http_cmd(), and submitted
over the socket.

get_http_cmd() normally just puts together a very simple HTTP request, which
consists only of the request line with the path (or, for proxies, the full
URL), and a "Host:" header containing the target host. (This is required in
HTTP/1.1, to allow multiple hosts on one IP.)

Things get slightly more interesting when some form data is to be POSTed. (GET
needn't be handled here, as the form data is already encoded into the URL before
calling http_init_load() in this case.) Besides of adding the "Content-Type:"
and "Content-Encoding:" header fields, the form data has to be submitted
inside the body of the request. For that purpose, the form data is encoded
using _mime_encode()_ or _url_encode()_ (see hacking-links.*) -- dependending
on the desired encoding stored in the "method" field of the form item -- and
stored in "form_data", which is a string submitted at the end of the request.

During the whole exection time of get_http_socket(), user breaks are enabled,
so that all the slow functions (DNS lookup, connecting to server, submitting
request), which are often even waiting for a timeout on failure, can be
interrupted.

After the connection has been established with get_http_socket(), parse_header()
is used to read and parse the HTTP headers.

Errors in HTTP loading are handled by setting "RES_FAIL" and "PT_INTERNAL",
just like file loading errors are handled in init_load().

_parse_header()_

The parser is very similar to the one in _parse_syntax()_ (see _Parsing_ in
hacking-layout.*): The input is processed character-wise by an FSM parser,
where each char is read from a buffer which is refilled (more or less)
transparently each time all chars have been processed.

The parser first skips the status line (we do not care about the return code,
for now...), and afterwards parses all header lines, extracting the name
and value of each, and storing them inside the "headers" structure of the
HTTP handle so they can be used later. (Presently, the only one used is the
"Location:" header, which is cheked in _init_load()_ to handle redirections.)

The parser tries to be as tolerant as possible about broken (or unknown...)
syntax. For that matter, '\r' characters are completely ignored, so both the
DOS-like '\r\n' linefeeds and unix-like '\n' do work. Illegal characters in
the header name are just ignored, as well as a missing space after the ':'
separating header name and value, and spaces at the beginning of a line which
can't be a folding.

There are a couple of both correct and broken example files in the test/
directory. Use them by piping the contents to tcplisten and pointing netrik
to the listening port. (You can also concatenate some HTML file to avoid
"No data" errors.)

The data (from the TCP socket) ending before the whole header was parsed is
treated as an HTTP loading error, except when a user break was performed,
in which case parse_header() simply returns without taking any more action.

_User Breaks_

The loading of a file or HTTP page can be interrupted by sending SIGINT.
interrupt.c contains a couple of functions to faciliate the SIGINT handling.
Those are called from various places in load.c and http.c; main() calls
init_int() at startup, which sets some constants used in the other functions.

At the beginning of init_load(), hold_int() is called. This function uses
sigprocmask() to block SIGINT. The signal is put on hold, i.e. it doesn't
have it's usual effect of aborting the program anymore, but it's also not
discarded; instead, it's stored, and awaits its release.

The signal is released using enable_load(), which is called in two
places: during read() or fread() in load(), and during get_http_socket()
in http_init_load(). (These are the functions which may take fairly long,
and need a way to be interrupted directly.)

enable_load() sets int_handler() as the handler for SIGINT (using sigaction()),
and then unblocks the signal with procsigmask(). However, not only arriving
new signals will invoke int_handler() now; if some signal was sent between
the hold_int() and enable_int(), it will be deliverd after enable_int(),
too. This way, every SIGINT during the whole loading process will cause a
break, not only if it was sent during the periods where it can be handled.

Before calling enable_load(), a return point is set using setjmp(), and stored
in "label_int". int_handler() does nothing else but immediately jumping
to that return point with longjmp(). That means, when an interrupt occurs
before or during the read()/fread() in load(), the signal handler jumps to
another position in load(). Here, "res->user_break" is set to indicate the
interrupt, and load() returns, instead of continuing the read(). Likewise
in http_init_load().

At the end of load() and http_init_load(), hold_int() is called, so SIGINT
will be put on hold again till the next call of load().

uninit_load() calls disable_int(). This function used sigaction() to set the
SIGINT handler to SIG_IGN, and then unblocks the signal. Thus, any SIGINTs
arriving after file loading has finished are discarded. They are only enabled
again in init_load(), when the next file load beginns.

