emirror(1)

NAME
emirror - a utility for automatic mirroring of ftp trees

USAGE
emirror [options] URL

DESCRIPTION
EMirror is a tool designed for retrieving files from different sites using the FTP (File Transfer Protocol) and saving them to disk. EMirror is non-interactive, which means it can work in the background, while the user is not logged in.

While other programs already have some good functionality, EMirror has some things which differentiates it from similar products:
- Nice HTMLized logfiles using template files,
- Parallel download of n sites,
- Disk space saving "only-latest" mode,
- Special options for deleting & keeping files,
- Notification of users about updates.

AVAILABILITY
The latest version of EMirror can always be found at <http://eclipt.uni-klu.ac.at> and <ftp://ftp.uni-klu.ac.at/pub/projects/emirror>

OPTIONS

The list of available options is pretty long, so they are split into several categories: Startup options, Logging and input file options, Download and FTP options and Error handling options. There are two ways to specify all options. The first one is via commandline and the second one is to use configuration files (which is common). Configuration files usually consist of one main section and one only-latest section (if you want to use this feature). Sections start with [sectionname] and then of a list of name: value pairs. The name is one of the options described now. For example configuration files have a look at the examples included within the distribution. Some settings require boolean arguments, which can be in the form True/False, Yes/No or 1/0.

  Startup options:

These options are general options concerning the startup and general behaviour of EMirror

-V --version
    Display the version and version history of EMirror

-h --help
    Display a list of available commands and a short description.

-v --verbose
    Verbose output. The default output doesn't contain any messages except error messages, so usually using this option is a good idea.

  Logging and input file options:

These options change the look and behaviour of log- and configuration files.

-o --output-file
    This names the name of the logfile (e.g. kernel.html).

-O --output-directory
    Directory on your local harddrive, where emirror will save logfiles. This should be in the tree of your WebServer, if you want to access the log-files via the web (e.g. /home/httpd/htdocs/mirrors).

-f --output-format
    Format of Logfiles. Possible values are Plain for plain text, HTML for standard HTML tags or PHP for PHP.

-c --config-file
    The name of the general configuration file. This configuration file should contain options, which are common to all mirrors (e.g. output-directory, output-format, username,...).

-d --config-directory
    The direcotry, which contains all configuration files for the specific mirrors. For mirroring one single package you can specify a single file.

-n --index-directory
    Build the index page for logfiles. You have to specify the directory, where EMirror can find the log-files (which is usually the same as specified with --output-directory).

--category
    Set the category of the mirror (e.g. Utilities).

--description
    Set some short textual description of the mirror

--title
    Set the title of the mirror (e.g. Linux Kernels).

--email
    Set the Admin-EMail address which will appear in the log-file (e.g. adm-ftp@ftp.uni-klu.ac.at).

--log-header-file
    Include a header file in the log-file, which contains user-defined HTML-code.

--log-footer-file
    Include a footer file in the log-file, which contains user-defined HTML-code.

--index-header-file
    Include a header file in the index-file, which contains user-defined HTML-code.

--mail-user-cmd
    Specify a command for sending mail to a list of users, when there have been any updates (e.g. mail -s "EMirror EMail-Information")

--mail-users
    Specifies a file, which contains a list of valid email-addresses of users, who want to get information about updates (e.g. /usr/local/etc/email-addresses)

--info-mailcmd
    Command to send mail to users when one spcific mirror has been updated (e.g. mail -s "EMirror - Package updated: %s" user1@host1 user2@host2)

  Download and FTP options:

These options change the behavior of EMirror concerning lokal download paths and FTP options:

-P --parent
    This specifies the full path, where files will be saved locally (e.g. /home/ftp/pub/mirrors/kernel)

-s --parent-format
    Set the format for creating directory trees. Possible values are:
    FULL (parent)/hostname/path
    PATH (parent)/path
    NONE (parent)

-D --download-url
    This sets the URL for downloading files locally (usually your hostname plus the last part of --parent, e.g. ftp://ftp.uni-klu.ac.at/pub/mirrors/kernel)

-t --tries
    Set the number of login-tries (0 unlimits).

-w --wait
    Set the number of seconds to wait between retries.

-r --recursive
    Download files recursively.

-F --follow-symlinks
    Follow symbolic links.

-e --exclude
    Regular expression of filenames to ignore when downloading files (see below).

-i --include
    Regular expressions of filename to include when downloading files. This is useful, if you just want to mirror a few files or if you want to use the --only-latest option (see below).

--ignore
    Regular expressions containing filenames to ignore completely. These files, won`t be deleted if they exist locally

-l --only-latest
    This option is used to just include the latest version of files. You can use this option multiple times and each value has to be a regular expression. When using configuration files you need to make a section [only-latest] and include name: value pairs for each regular expression.

-u --username
    This specifies the username used for logging into the remote server (e.g. anonymous).

-p --password
    The password used for logging into the remote server (e.g. adm-ftp@).

-m --max-parallel
    Sets the number of mirrors to execute parallel (e.g. 5).

-R --real-hostname
    Override the hostname in the URL specified with a given hostname (this is useful for hiding the name of mirror-only ftp-sites).

-M --max-days
    Do NOT download files which are older than XXX days.

-A --max-age
    Delete local files, which are older than XXX days.

-E --max-delete
    Do not delete more than XXX KB, MB or percent of local files.

-U --umask
    Specifies a umask for creating files and directories.

--ignore-mdtm
    Do not use the file modification time for deciding, wether files shall be downloaded or not.

--owner
    Set the owner and group of downloaded files to UID:GID. UID and GID have to be the numeric values.

--delete-local
    Always delete local files, which do not exist on the remote side. Normally only files, which match --include will be deleted.

--passive
    Set passive mode

--proxy
    Use a proxy server. Format for this option is hostname:port ! Note: Only real proxy servers are valid (e.g. delegate). Squid is not a valid FTP proxy (unfortunately).

--continue
    Continues (resumes) aborted downloads by using REST & RETR.

--test
    Run in test mode (do NOT download or delete anything).

  Error handling options

These options change the look and behaviour of log- and configuration files.

--error-mailcmd
    Specify a mail command, which will be executed, when errors occur (usually this should be mail -s "EMirror Error Report" preisl@stuco.uni-klu.ac.at

  Debug options

--debug
    Debug output. This is basically for debugging purpose. It contains lots of information which is normally nothing interesting for users.

--debug-no-exceptions
    Do not use built in exceptions, print real errors instead. This spits out python errors which helps the author to track down errors.

  Advanced options

-k --no-kill
    Do not kill old mirror-processes. Per default, ECLiPt Mirror kills old (probably hanging) processes, to help to keep the machine clean.

--pid-file
    Specify a different pid-file than the default file (default: $HOME/.emirror.pid)

  Auto-Installer

There is a special section labelled [auto-installer] in configuration files. With those options you can automatically install updated files after they've been downloaded. Available options are:

BuildDir
    This is the directory, where everything is built. Default value is "/var/tmp", another good choice is "/usr/src/redhat/BUILD".

RPMOptions
    RPMs are handeled by the redhat package manager. RPM-Options are commandline parameter for RPM for installing files. Default is "-Uvh".

RPMBuildOptions
    Those options control rebuilding of Source-RPMS. Default value is "--rebuild"

LogFile
    Output of rebuilding and installing files is being written into this specified file.

RPMs
    This specifies a list of regular expressions of RPMs to be installed.

SRPMs
    This specifies a list of regular expressions of Soure-RPMS to be rebuild and installed.

Other[n]
    This specifies a list of other files to be installed. Usually this is a regular expression of one file plus in [] a list of commands, which shall be executed. An example would be: Other1 = yafc.*tar.gz [tar xzf %f;cd %d;configure --prefix=/home/preisl/source/yafc;make;make install] %f expands to the filename matching to the regular expression, %d expands to the directory which gets created .e.g. after extracting the archive.

TEMPLATE FILES
There are two template files necessary, if you want to use logfiles. One template file is for the logfile itself and one is for the index file.

  Logfile template
The template file for logfiles is a plain text or html document which contains special keywords which will be replace by EMirror. These keywords are optional, but the template file has to include those lines:
<!-- TITLE: $title$ -->
<!-- CATEGORY: $category$ -->
<!-- STATUS: $mdtm$ -->
<!-- TIMESTAMP: $timestamp$ -->
<!-- STATISTICS: $statistics$ -->
<!-- DESCRIPTION: $description$ -->
Those tags are necessary to properly built the index-file. There are two more tags:
<!-- ERRORCOLOR: #CC0033 -->
<!-- WARNINGCOLOR: #660099 -->
which set the color of error and warning messages. EMirror understands the following keywords:

$title$
    This is the title specified with --title.

$url$
    The URL you mirror.

$category$
    The category specified with --category.

$description$
    The description specified with --description.

$localurl$
    This is the local downloadurl (the URL on your site) specified with -D,--download-url.

$email$
    This is the EMail-address of the responsible persone set by --email.

$hostname$
    The hostname of the remote site (part of the URL)

$port$
    The port of the remote site (part of the URL)

$username$
    The username used for logging into the remote site (just visible if it's 'ftp' or 'anonymous') specified with -u/--username.

$password$
    The password used for logging into the remote site (just visible if the username is 'ftp' or 'anonymous) specified with -p/--password.

$remotedirectory$
    The directory on the remote site (part of the URL).

$localdirectory$
    The full local directory specified with -P/--parent.

$tries$
    The number of tries specified with -t/--tries.

$delay$
    The delay between retries specified with -w/--wait.

$downloadrate$
    The average download rate in Byte/sec or KByte/sec.

$starttime$
    The start time of the mirror.

$endtime$
    The end time of the mirror.

$duration$
    The duration of the mirror.

$downloadstats$
    The download statistics (number of files/directories downloaded).

$deletestats$
    The delete statistics (number of files/directories deleted).

$mdtm$
    The latest execution time of the mirror (this one is usually only used in the <!-- STATUS: $mdtm$ --> tag !

$logstart$ and $logend$
    These two commands specify the part of the template file, where the log entries itself are filled in.

$type$
    This is the type of entry (e.g. delete or retrieve).

$message$
    This contains the log message itself (warnings, links to files downloaded).

  Indexfile template
This template file contains information (order by category) for each mirror. There are two more special tags for setting the color of error and warning messages:
<!-- ERRORCOLOR: #CC0033 -->
<!-- WARNINGCOLOR: #660099 -->
The template file is a plain text or HTML file which expands the following keywords:

$categorylist$
    This expands to a "|" seperated list of categories.

$lastupdate$
    This expands to the current time.

$downloadrate$
    The average download rate of all mirrors in Byte/sec or KByte/sec.

$starttime$
    The start time of the first mirror.

$endtime$
    The end time of the last mirror.

$duration$
    The duration of all mirrors together.

$downloadstats$
    The download statistics (number of files/directories downloaded) of all mirrors.

$deletestats$
    The delete statistics (number of files/directories deleted) of all mirrors.

$categorystart$ and $categoryend$
    These two keywords mark the area of code, which is dedicated to each category.

$category$
    This expands to the current category.

$mirrorstart$ and $mirrorend$
    These two keywords mark the area of code, which is dedicated to each mirror.

$status$
    This contains the status of the mirror (e.g. unchanged)

$title$
    This contains the title of the mirror.

$update$
    This is the date of the last update of the mirror (which is a link to a file containing information about the latest update).

$proxy$
    This the proxyhost and the port (format host:port).

REGULAR EXPRESSIONS

The regular expressions used for the --include, --exclude and --only-latest command are perl-style regular expressions. A complate description can be found at <http://www.python.org/doc/lib/module-re.html> It is possible to use filenames including the full path within regular expressions or just the filename itself.

EXAMPLES

Here are some examples how to use EMirror
Get URL ftp://ftp.uni-klu.ac.at/pub/projects/emirror
emirror ftp://ftp.uni-klu.ac.at/pub/projects/emirror
Get URL ftp://ftp.uni-klu.ac.at/pub/projects/emirror and be verbose
emirror -v ftp://ftp.uni-klu.ac.at/pub/projects/emirror
Get a list of mirrors with a general configuration file and a directory
which contains mirror-specific configuration files (verbose mode)
emirror -v -c general.cfg -d active
Get a list of mirrors with a general configuration file and a directory
which contains mirror-specific configuration files plus create
a index page (verbose mode)
emirror -v -c general.cfg -d active -n /home/httpd/htdocs/mirrors

ENVIRONMENT
EMIRRORCFGPATH

FILES
/usr/local/lib/emirror/*, /usr/local/etc/general.cfg, /usr/local/etc/email-addresses, /usr/local/etc/active/*, /usr/local/etc/unused/*

UNRESTRICTIONS

EMirror is distributed under the General Public License Version 2. A copy of this license (COPYING) accompanies each copy of EMirror

SEE ALSO
lynx(1), ftp(1), wget(1)

AUTHOR

The author of EMirror is Martin Preishuber <Martin.Preishuber@stuco.uni-klu.ac.at>. Thanks to all people who sent me comments, bug-reports and suggestions !