This file describes the command line options of PVMPOV, as well as the general
theory behind how it operates.

Theory of Operation
-------------------
Using the PVM model, there is one master and many slave tasks. The master has 
the responsibility of issuing the work sections to the slaves, and receiving 
portions of the rendered sections back.  The master does not render anything
by itself.  The PVM component is only active if the user gives the "+N" option
to POV.  Otherwise, PVMPOV behaves the same as regular POV-Ray and runs a
single task only on the local machine.

Valid POV-Ray command-line options relating to PVM are:

    pvm=on
    +N      This is the default for starting PVMPOV.  One slave will be
	    started on each available host, regardless of architecture, and
	    the blocks will be 32x32 pixels in size.  The slaves will be
	    started with a nice value of 5, which means they will run at a
	    lower priority than other user jobs. 
    pvm=off
    -N      Turns of PVM support. PVMPOV now runs exactly as normal POV-Ray. 
            You should do this when there are still errors in your pov-file. 
            PVMPOV will not display error message otherwise.

    pvm_tasks=xx
    +NTxx   Start xx tasks on the available PVM hosts.  This is usually only
            useful if debugging on a single machine, or for starting more
	    than 1 task on multi-processor hosts.  If, for example, you have
	    10 machines with 4 CPUs each, you could specify +nt40 to start 4
	    processes on each host (and the OS will hopefully run 1 on each
	    CPU).  

            Note that PVM is stupid in the way it starts tasks, so if,
	    in the previous example, one of the hosts has only one CPU, it will
	    still have 4 slaves started on it. You can use the "pvm_hosts"
            option (see below) to control on which machines the tasks are 
            started.

            Starting multiple tasks on a single processor will always be less 
            efficient than a single task because of context switching and extra 
            message passing.

    pvm_arch=arch
    +NAarch Start the tasks only on the PVM architecture "arch". If +NT is not
	    given, one task will be started on each of the hosts of the given
	    architecture.

    pvm_nice=xx
    +NNxx   Run the slaves at a niceness factor of xx.  The default niceness
	    is 5.  In general, changing the niceness value will not affect
	    performance very much, but may get others upset with you.  The
	    nicest setting for pvmpov is 20, while the least nice setting is
	    0.  Note that these values are always used even on systems that
	    use nice values from 20-40.  See the installation document and
	    the man page for nice for more information.

    pvm_width=xx
    +NWxx   Change the width of the blocks to xx pixels.  The default width
	    is 32 pixels.

    pvm_height=xx
    +NHxx   Change the height of the blocks to xx pixels.  The default height
	    is 32 pixels.

            It is important to note that by varying the size of the grid
	    sections, you can affect the performance of the rendering.  If you
	    have particular renderings that are very complex in a small portion
	    of the display, then a finer grain may help.  In this way, more of
	    the tasks are able to migrate towards the grid sections that are
	    more complex.  Conversely, if you have a shorter render or a 
	    slower network, it may be advantageous to have larger blocks to
	    reduce network overhead, as well as ensure the slaves are not idle
	    waiting for blocks to render.

	    You must also consider overhead if using anti-aliasing.  Anti-
	    aliasing requires the line segment above and below the grid section
	    to be traced so that super-sampling may occur.  If the height of the
	    grid is reduced in size, and anti-aliasing is turned on, your
	    percentage of overhead goes up.  For example, setting a height of
	    four (-NH4) using anti-aliasing would have more than 25% overhead.
	    If the image size is not an integer multiple of the grid size, the
	    edge blocks are smaller (ie extra pixels aren't rendered), so it
	    is not necessary to evenly divide the image into blocks.

     pvm_slave=slave
     +NSslave  Uses slave as filename for the slave tasks. If you do not 
            specify this option PVMPOV will use the current executable name
            also for the slaves. 

            Using this option you can for example run the X11 version
            (x-pvmpov) as master (to display the results) and a version
            without any display support (pvmpov) as slaves.

     pvm_wd=dir
     +NDdir Set the working directory for the slaves. By default PVMPOV
            tries to run the slaves in the same directory, as the master.
            But sometimes 'getcwd' gives misleading output (when automounting
            is used), or you simply want to run the slaves in some other
            directory, then you can use this option.

     pvm_hosts=name1,name2,...
            Set the names of the hosts to use for slaves. Note, that there are
            not spaces allowed between the names!
         
            By default PVM distributes the processes in some order to the
            available machines. Sometimes the choice of PVM is not the best,
            so you can specify explicitly which machines to use. 
            
            You can use more tasks, then you specify here. You can use the
            options in this way: pvm_hosts=darkstar,darkstar,baby +nt6
            This will start 6 tasks (+nt6 must come after the pvm_hosts),
            4 on darkstar and 2 on baby.
   

For example, if I wanted to distribute an 1024x768 rendering and use eight PVM 
tasks to do the work, I would use the +NT8 option, and let the PVM master 
define 768 32x32 sections.  Eight slave tasks would be launched, and when ready,
they would request work to do from the master.  The master then issues a column
and row range back to the requesting slave.

% pvmpov +nt8 +w1024 +h768 +iskyvase.pov +oskyvase.tga

As each quarter of a grid is completed, the slave sends it back to the
master, which buffers the incoming blocks until an entire line recieved, at
which time it writes the line to disk with the specified file format.  If
all slaves did perfectly equal work in this example, they would be issued 96
sections of work, and would have returned 384 messages of 32x8 pixels each.

Note that the work issues are done essentially one at a time.  As each slave
starts up it requests a single block, and as it recieves a work request, it 
requests the next block.  In this way, effective load balancing can occur.
Tasks rendering the "easy" sections will eventually all migrate towards the
harder sections.  Seriously imbalanced distributions will be seen by widely
varying percentages shown at the end of the rendering, assuming all of the
hosts are identical.  Since each slave always has a block to be rendered, it
will be kept busy without waiting for the master to recieve its work request
and reply with a new block assignment.

If for some reason any of the slaves are very slow, or they are interrupted
during their operation, their blocks are reassigned to other slaves after
all of the blocks have been assigned a first time.  This means that the render
can be completed quickly even if some of the slaves have problems, or if some
blocks have been assigned to slower slaves while the faster slaves are idle.
This will show up as some percentage of late blocks for that machine in the
statistics at the end.  Note that blocks will NOT be reassigned to slaves that
are significantly slower than average, in preference to reassigning it to a
faster slave when it becomes available.

Since only complete lines are written out to disk, interrupted traces will
be restarted at the last complete line that was written to disk, regardless
of how many blocks were buffered by the master at the time.  However, it is
possible to re-start an interrupted trace in the same method as POV-Ray.

Known bugs (and other limitations)
-------------------------------------
All of the slaves must to share the same file system as the master process,
and the source file must be mounted in the same location on each one.  This
may limit the number of hosts that can be used in some circumstances.

The command line arguments for specifying hosts is not cumulative, so

% pvmpov +naSUNMP +nt24 +n +iskyvase.pov +oskyvase.tga

will not start 24 slaves on the multi-processor SUN system(s) (which may have
4 CPUs each), and then start 1 slave on each of the remaining systems.

The statistics for the image being rendered may not be accurate if any of the
slaves is too slow, or stops for some reason.  As well, since some sections
may be rendered twice if the block is reassigned, the statistics will not be
exactly the same as if it were rendered on a single host.  The master only
waits a few seconds for statistics to arrive before printing them.

The master does not parse the input file. Because of that it displays
no error messages concerning the input file. As long as your input file
still contains error you should disable PVM support (-N or pvm=off)

Execution example (from version 2.9):
-------------------------------------
You first must have a PVM daemon launched on each machine that will be 
participating in the rendering.  Refer to the PVM 3.3 documentation.  The 
following is an example is from Jason Hough, and was generated on a group
of six Solaris based 4-processor SPARCstation 20s.  His home directory is
auto-mounted to all of these hosts.

He keeps the pvm daemon installed in a directory called "bin", given by
"dx=./bin/pvmd3" relative to his home directory, and pvmpov is in various
subdirectories under "bin" (ie bin/SUN4, bin/SUNMP, bin/LINUX, etc.), given
by the executable path "ex=./bin", so his pvm.hosts file looks like:

% cat pvm.hosts
glee    dx=./bin/pvmd3 ep=./bin
elation dx=./bin/pvmd3 ep=./bin
ecstasy dx=./bin/pvmd3 ep=./bin
bliss   dx=./bin/pvmd3 ep=./bin
delight dx=./bin/pvmd3 ep=./bin
rapture dx=./bin/pvmd3 ep=./bin

Note that, by default, PVM looks for executables in $HOME/pvm3/bin/$PVM_ARCH,
and pvmd should be in $PVM_ROOT/lib, so if this is the case on your systems,
you don't need to have anything in your pvm.hosts file except the hostnames,
one per line.  An exception is RS6K, which needs the dx= no matter what.  See
the PVM documentation and/or the pvmd3 man page for more information on what
all of this means if you don't understand it.

The following command launches the PVM daemons.

% pvm pvm.hosts
3.3.7
t40001
pvm> conf
6 hosts, 1 data format
		    HOST     DTID     ARCH   SPEED
		    glee    40000    SUNMP    1000
		 elation    80000    SUNMP    1000
		 ecstasy    c0000    SUNMP    1000
		   bliss   100000    SUNMP    1000
		 delight   140000    SUNMP    1000
		 rapture   180000    SUNMP    1000
pvm> quit

pvmd still running.

Now that the PVM daemons are up and waiting for work to do, we can render.
Note that for these MP machines he forces pvmpov to start more tasks than
the default 1 per host, and uses a 64x64 block size:

% pvmpov -Iskyvase.pov -Oskyvase.tga +nt24 +nw64 +nh64 +v

POV-Ray Options in effect: +v1 +ft +mb25 +nt24 +nn5 +nw64 +nh64 +a0.300
+j1.000 +b999 +r3 -q9 -w1024 -h768 -s1 -e768
-k0.000 -mv2.0 -iskyvase.pov -oskyvase.tga 
   ...at least 13 tasks successfully spawned in time.
   ...Don't worry, more are on the way, I'm just not waiting
PVM Task Distribution: Tasks-24  Grid width-64  Grid height-64  Sections-192

Waiting for slave stats.

PVM Task Distribution Statistics:
           host name  [ done ] [ late ]           host name  [ done ] [ late ]
                glee  [ 4.17%] [ 0.00%]                glee  [ 4.17%] [ 0.00%]
                glee  [ 4.17%] [ 0.00%]                glee  [ 4.17%] [ 0.00%]
             elation  [ 4.69%] [ 0.00%]             elation  [ 4.17%] [ 0.00%]
             elation  [ 4.17%] [ 0.00%]             elation  [ 4.17%] [ 0.00%]
             ecstasy  [ 3.65%] [ 0.00%]             ecstasy  [ 4.69%] [ 0.00%]
             ecstasy  [ 4.69%] [ 0.00%]             ecstasy  [ 4.17%] [ 0.00%]
               bliss  [ 3.65%] [ 0.00%]               bliss  [ 4.17%] [ 0.00%]
               bliss  [ 4.69%] [ 0.00%]               bliss  [ 3.65%] [ 0.00%]
             delight  [ 3.65%] [ 0.00%]             delight  [ 4.17%] [ 0.00%]
             delight  [ 4.17%] [ 0.00%]             delight  [ 4.17%] [ 0.00%]
             rapture  [ 4.69%] [ 0.00%]             rapture  [ 4.17%] [ 0.00%]
             rapture  [ 4.17%] [ 0.00%]             rapture  [ 3.65%] [ 0.00%]


skyvase.pov statistics
--------------------------------------
Resolution 1024 x 768
# Rays:     3773743    # Pixels:      798720  # Pixels supersampled:      17381
  Ray->Shape Intersection Tests:
   Type             Tests    Succeeded   Percentage
  -----------------------------------------------------------
  Sphere          6304452     1170727       18.57
  Plane          63822062    35385552       55.44
  Quadric         6304452     2770858       43.95
  Cone            5918163     4839298       81.77
  Bounds          5918163     3152226       53.26
  Calls to Noise:      4327871
  Calls to DNoise:     5141872
  Shadow Ray Tests:   10498615     Blocking Objects Found:      254807
  Reflected Rays:      2818594
  Time For Trace:    0 hours  0 minutes 47.00 seconds

------
NB: Note that for comparison purposes with other skyvase benchmarks that this
    is rendered at 1024x768 instead of the usual 640x480!

Enjoy!

Andreas Dilger
adilger@enel.ucalgary.ca

Harald Deischinger
k3096e5@c210.edvz.uni-linz.ac.at
