  netrik hacker's manual
>========================<

The manual contains detailed documetation on the netrik code. It should be
of interest to you if you want to hack on netrik, but also if you are only
curious how it works.

The manual includes:
- hacking.* (this file): general notes, module overview
- hacking-load.*: loading HTML files from local disk or HTTP connections
- hacking-page.*: page (history) handling
- hacking-layout.*: the layouting/rendering module
- hacking-pager.*: the interactive viewing module
- hacking-links.*: hyperlink handling
("*" means txt or html -- choose the one you prefer...)

_0. General notes_

The program (except for the layout engine) presently doesn't obey any specific
structure. It can be roughly divided into a copule of big modules: layout
engine, file loading, and pager; the page handling, which is another part of
the loading mechanism, may also be considered a module of its own. However,
these modules aren't seperated clearly; they intersect at many places. The
loading is tightly coupled to the layouting -- the page handling part invokes
certain layouting passes, while the file loader offers it's service to feed
the data into the layout engine. However, these two parts of the loading
mechanism need to exchange much information (URLs, error conditions etc.),
which can only be passed through the layout engine. The pager naturally has
to make use of the layout engine to get the screen content; but it also uses
some of it's data structures and helper functions for link handling. The
main programm basically makes calls to both the pager and the page handling
(which in turn invokes the layouting), but also has to use data structures and
functions directly; actually, it contains part of the page handlig. Finally,
the loading module affects and depends on some data used by the pager.

A problem arises form the fact that all control passing is done by direct
function calls. This was perfectly OK at the time when netrik wasn't much more
than a layouting engine; with all the link and anchor handling, various loading
mechanisms (including error handling, user breaks etc.), form handling, and
other functionality, this has gotten quite chaotic -- it's very hard to retrace
the various intermingled calls between the modules or the data dependencies.

This probably could be helped somewhat by using a kind of object-oriented
approach, i.e. trying to put all operations on a given data structure in one
place. (Encapsulation.) However, I don't think it would be possible to make
great progress without much hassle and considerable efficiency loss.

I've devised a much nicer, clearer solution instead (well, at least it looks
nice in my imagination ;-) ), which involves a fundamental change in program
structure. Basically it's about passing flow control to a main dispatcher,
using state handles and an event queue.

We will probably implement that concept in one of the next major releases. It
is inevitable for multitasking (displaying of partially loaded pages,
multiwindowing) anyways, as we want to stick to explicit multitasking
(real multithreading/multiprocessing is much more complicated, and terribly
inefficient), and only this concept allows resuming and a fine grained
scheduling. But again, it should also simplify and clearify the overall
program structure, so we will probably implement it even before it's absolutely
necessary.

As mentioned before, the layout engine, contrary to the other modules, has
a fairly clear concept from the beginning. The main idea was to split HTML
processing into a couple of independant passes. As the individual passes are
quite simple, this makes understanding and altering the code quite easy.

On the other hand, this isn't terribly efficient the way it is done now. We
think however that in the early stage netrik is now, it's more important to
keep the code as simple as possible to faciliate fast developement.

Also, splitting up the processing is necessary to allow rendering of not
completely loaded pages. As this is to be one of the major features of netrik
(it's not implemented yet), we pay much attention to that.

However, the splitting into passes isn't presently optimal by any means. It's
more or less the first which came to mind. In the new parser planned for the
next major release the segmentation will be somewhat different.

Most of the processing steps of the layouting are working recursively due
to the nature of SGML/XML. They arn't implemented recursively, though. This
makes understanding a tick harder, but there is a couple of reasons for that.
Efficiency is one of them, but not that important presently. The main reason is
that a real recursive implementation wouldn't work with the planned multitasking
system (s.a.) as it doesn't allow resuming.

_1. Module Overview_

Here is an overview how the different modules work together. Detailed
descripitons of the modules are located in the various hacking-... files.

_main.c_

The main program is responsible for two things: Program initialization,
and the main (pager) loop.

Initialization presently mostly comprises reading command line and config
file options.

This is somewhat tricky, but not too complicated when you get the idea: The
config file (~/.netrikrc) just contains options identical to the command line
options, one per line. They are simply prepended to the command line options,
and processed with getopt_long() together.

For that, netrik first reads the file. This is done by determining the file
name (using the HOME environment variable), then using stat() to get the file
size; this also tells us whether the file exists at all. Then it is opened
and read into a memory buffer, all at once. The next step is the intersting
one: The file contents is scanned, to extract the single options. Every
encounterd newline is replaced by a '\0' (string end), and a pointer to the
beginning of that line (option) is stored in the "argv_all" array . "argv_all"
and "argc_all" are identical to "argv" and "argc", only we also store the
config file options here. argv_all[0] is not filled, as argv[0] is always
the command that started the programm; the real arguments start with 1. After
extracting all config file options, the real command line options from "argv"
are appended to "argv_all". This way, defaults can be set in the config file,
but still overridden by command line options.

Having this, the arguments are scanned with getopt_long() in config_cmdln(),
where options are extracted and stored in the global "cfg" struct, and
non-option arguments singled out. (They are moved to the last argv-entries
by getopt_long().) The last argument is used as the resource name for the
document to load on startup; the other ones are ignored. (This allow setting
a "home page" in the config file, which will be overwritten if some URL is
specified on the command line.)

After scannig the arguments, the first document is loaded and, unless netrik
was invoked with the "--dump" option, the pager loop is entered.

At the beginning of each iteration, display() is called to start the pager. This
one performs all pure viewing operations, including scrolling and link
selection; it returns only when some page handling and/or leaving fullscreen
mode is required. main() then performs the desired action (indicated by the
return value of display()), before restarting the pager in the next iteration.

Most of those actions involve loading some new page -- actually, the main
loop can be considered part of the _Page Loading_ mechanism.

The new page (as well es the startup page, before entering the loop) is
always loaded by calling load_page() in one mode or the other. The mode varies
depending on the exact command: Following a link (or a form submit button),
opening a new URL on the command prompt (s.b.), history commands. The precise
actions are described in detail in hacking-load.*.

There are also situations that do not require loading a new page; in this case,
the viewer is simply restarted with the same page in the next iteration.

For one, this includes handling of non-submit form items. This is described
under _Manipulating_ in hacking-links.*. Note that all form elements,
including submit buttons, and also normal links are handled by one pager
return value; they are distinguished by the type of the active link item,
which is handled in another switch by main().

The second group of commands that do not involve loading a new page are the
URL displaying commands ('u', 'U', 'c'), which just print some information
on the screen before returning to the pager. Besides of that, they are also
related to the page handling, and thus described in hacking-page.* .

The command prompt is somewhat special also, as it doesn't cause loading some
new page immediately. readline() is used to get a command line from the user
instead, and the action taken depends on that command. However, presently
only the ":e" and ":E" commands are implemented, which also load an new page;
they are described in hacking-page.* also.

A somewhat unobvious aspect is switching between curses fullscreen mode and
normal scrolling mode. The viewer always operates in fullscreen mode, while
everything in main() is done in scrolling mode.

Curses fullscreen mode is initialized once, just before entering the main
loop.  The program doesn't switch to fullscreen mode immediately, however;
this is done only on the first refresh, which is induced by the getch()
inside display() -- after rendering the screen content.

Fullscreen mode is turned off using endwin() before display() returns, and
will be reactivated only when getch() is called again from display() in the
next iteration.

Some commands need to display some information to the user before returning
to the pager. Particularily the URL show commands, but also loading if some
error occurs or if in --debug mode. This is accomplised by the "pager_wait"
flag, which is set by any command that needs to display something. If the flag
was set during an iteration, main() waits for a keypress before proceeding
with the next one -- which will reactivate the pager.

Quitting netrik is also handled in the main loop: When the user typed 'q'
in the pager, display() returns a special value, and the loop terminates. A
few cleanup actions follow, whereafter main() returns.

_File Loading_

The loading module consists of two parts: One is responsible for actually
loading an HTML document from a local file or from a HTTP server. It consists
of the files load.[ch], http.[ch], http-parse-header.[ch], the url handling
functions in url.[ch], and interrupt.[ch] for user break handling. Some
functions from forms.c are necessary also, when submitting HTML forms. This part
is invoked from layout(), as part of the <a href="#layout">_Layouting_
process.

The other part of the loading mechanism is the _Page Loading_ system.

The main file of the file loader is load.c, which contains the init_load()
and load() functions.

The loading is intialized by a call to init_load() with the desired URL as
argument. A base URL is also passed, which is merged with the target URL
to create the effective URL, if the target URL is a relative one. (Following
links etc.) init_load() then decides whether it is a local file or an HTTP URL,
and initializes the loading. (Opens file or establishes HTTP connection.)

The loading itself is done by calling load(). This function fills a buffer,
which then can be processed. After ther buffer is processed, load() has to
be called again, loading the next chunk.

The loading module is described in detail in hacking-load.*.

_Page Loading_

The second part of the loading mechanism is the page handling system. This one
is responsible for getting a new document upon request, and preparing it so
it can be displayed in the _Viewer_ . It's also responsible for history
handling, and everything else connected with handling the _Page List_
(see hacking-page.*).

The page loading mechanism only has the page.c and page.h files for itself;
however, part of it is also located in _main()_ .

The main function is load_page(), with the basic functionality of loading
a new document and preparing it so it can (and will) be displayed in the
_Viewer_ , as soon as _display()_ (see hacking-pager.*) is called from
_main()_ . For this, a new page list entry is created, and _layout()_
(see hacking-layout.*) is used to load the page and prepare it for rendering.

load_page() also has other modes, depending on what kind of page is to be
displayed: If the new page uses the same HTML document, and the page only has
to jump to an anchor, the document isn't reloaded; a new page descriptor is
created, but the layout date is taken from the old page.

If a page from the page history is to be reloaded, no new page descriptor is
created; the old one is simply reactivated. The document is reloaded using
layout(), if necessary.

load_page() is called from _main()_ to load the start page given on the
command line, and then in various modes from the main loop, when the user
requests various functions in the _Viewer_ (loading new URL, following link,
going back/forward in page history).

The page loading mechanism is described in detail in hacking-page.*.

_Layouting_

The layout engine consists of the files parse-syntax.c, syntax.h, facilities.c,
dump-tree.c parse_elements.c, sgml.c, parse-struct.c, items.h, pre-render.c,
render.c, render.h, layout.c, and layout.h.

As mentioned before, layouting is done in several passes.

The first passes are parsing syntax (parse_syntax()), looking up the element and
attribute names (parse_elements()), (optionally) fixing a broken tree created by
SGML documents (sgml_rework(), interpreting the elements (parse_struct()), and
assigning positions and sizes to all items of the output page (pre_render()).

make_link_list() and make_anchor_list() are also called after parse_struct(),
creating the link list and anchor list data structures necessary for link
handling. (See hacking-links.*.)

All these passes are applied from layout(), immediately after the file is
opened; load() is called from within parse_syntax().

After these processing steps, the page is ready for rendering. The actual
rendering is done just in time by render(), which is called from the viewer,
each time some new region of the output page needs to be displayed.

The layout engine is described in detail in hacking-layout.*.

_Viewer_

The viewer module consists of the files pager.c and pager.h.

The pager uses curses to display the layouted page in an interactive manner.
(Presently, only scrolling, simple link selection, and various page history
commands are implemented.)

Every time the visible output page region changes, render() is called to
display the new region.

The viewer module is described in detail in hacking-pager.*.

_Link Handling_

Hyperlink (and anchor) handling is not a module in a classical sense; there
is no central place, no source files specific for that. There are the links.c
and links.h files, but they contain only some helper functions; most of the
code necessary for handling links is distributed among almost all of the
other modules.

The layout engine needs to extract all links and anchors while parsing the
page, and assign coordinates to them while pre-rendering. The pager needs
to highlight the selected link; provide commands for selecting and following
links; and inform the main program about the link following. The main program
needs to initiate loading of the link. The file loader needs to construct
the target URL from the current page URL and the link URL.

All necessary steps are described in hacking-links.*, or else pointers to
the specific module documentation are given.

_Form Handling_

HTML forms are very similar to links, and mostly they are handled together. Some
special handling is required at certain places, though. These are mentioned
in hacking-links.*. Some additional functions necessary for form handling
reside in forms.c; these are also covered in hacking-links.*.

_screen.c_

render.c uses the curses library for output. screen.c contains a couple of
helper functions for handling curses, both in raw and in full screen mode.

init_curses() is responsible for initializing curses in raw mode, and getting
some control sequences for setting colors etc.

start_curses() initializes curses in full screen mode, and sets the color pairs.
It also initializes several curses settings.

All 63 modifiable color pairs are initialized (pair 0 is hard-wired in curses),
in such a manner that every combination of the eight foreground and eight
background colors is covered. (Brightness is controled by extra flags, and not
coded in the color pairs.) The upper three bits of the six-bit pair number
specify the background, the lower 3 the foreground. While the background
colors are mapped directly, some magic is necessary for the foreground,
to ensure that color pair 0 (which is hard-wired to white on black) will
actually correspond to the right color in the color sheme. This is achieved
by rotating the foreground color space by one, so color 0 gets white, color 1
gets black etc., instead of color 0 being black and color 7 being white. (This
also solves some problems with the "bright black" (i.e. dark grey) background
color in a black xterm, but don't ask me why...)

For color pairs having black background or white foreground, the respective
parts aren't set; the terminal default is used instead. (Unless --force-colors
is used.) This is to produce the expected results in most xterms (or other
terminals with bright background and black text).

set_color() sets the text attributes in full screen mode. The foreground
color is set, and the "bold" attribute if it is a light color.

If the background color is bright, the "standout" attribute is set -- this seems
to be the only way to get a bright background in xterm. The drawback is that
the foreground is always bold in this mode in an xterm. As standout implies
reverse video, foreground and background colors have to be swapped in this mode.

set_color_raw() does the same in raw mode. As raw mode has no color pairs to be
set during initialization, handling of default colors for white foreground/black
background has to be done here. The respective parts simply aren't set at all,
thus keeping the terminal default colors.

reset_colors_raw() resets all text attributes to their default values.

