AFIO(1)

NAME
afio - manipulate archives and files

SYNOPSIS
... | afio -o [ options ] archive : write archive
afio -i [ options ] archive : install archive
afio -t [ options ] archive : list table-of-contents of archive
afio -r [ options ] archive : verify archive against filesystem
afio -p [ options ] directory [ ... ] : copy files

DESCRIPTION
Afio manipulates groups of files, copying them within the (collective) filesystem or between the filesystem and an afio archive. Note that afio archives are portable, as they contain only ASCII-formatted header information. They are also compatible with ASCII cpio(1) archives (ala cpio -c, for GNU cpio(1) also cpio -H odc). However, archives made with using -4 option are not portable.

With -o, reads pathnames from the standard input and writes an archive.

With -t, reads an archive and writes a table-of-contents to the standard output.

With -i, installs the contents of an archive relative to the working directory.

With -p, reads pathnames from the standard input and copies the files to each directory. Cannot be combined with the -Z option.

With -r, reads archive and verifies it against the filesystem. This is useful for verifying tape archives.

Creates missing directories as necessary, with permissions to match their parents.

Removes leading slashes from pathnames when reading, writing, and cataloging an archive, unless instructed not to.

Supports multi-volume archives during interactive operation (i.e., when /dev/tty is accessible and SIGINT is not being ignored).

OPTIONS

-@ address
    Send email to address when a volume change (tape change, floppy change) is needed, and also when the entire operation is complete. Uses sendmail(1) to send the mail.

-a  Preserve the last access times (atimes) of the files read when making or verifying an archive. Warning: if this option is used, afio will change the last inode changed times (ctimes) of these files. Thus, this option cannot be used together with an incremental backup scheme that relies on the ctimes being preserved.

-b size
    Read or write size-character archive blocks. Suffices of b, k, m and g denote multiples of 512, kilobytes, megabytes and gigabytes, respectively. Defaults to 5120 for compatibility with cpio(1). In some cases, notably when using ftape with some tape drives, -b 10k is needed for compatibility. Note that -b 10k is the default block size used by tar(1), so it is usually a good choice if the tape setup is known to work with tar(1).

-c count
    Buffer count archive blocks between I/O operations. A large count is recommended for efficient use with streaming magnetic tape drives, in order to reduce the number of tape stops and restarts.

-d  Don't create missing directories.

-e bound
    Pad the archive to a multiple of bound characters. Recognizes the same suffices as -s. Defaults to 1x (the -b block size) for compatibility with cpio(1).

-f  Spawn a child process to actually write to the archive; provides a clumsy form of double-buffering. Requires -s for multi-volume archive support.

-g  Change to input file directories. Avoids quadratic filesystem behavior with long similar pathnames. Requires all absolute pathnames, including those for the -o archive and the -p directories.

-h  Follow symbolic links, treating them as ordinary files and directories.

-j  Don't generate sparse filesystem blocks on restoring files. By default, afio creates sparse filesystem blocks (with lseek(2)) when possible when restoring files from an archive, but not if these files were stored in a compressed form. Unless stored in a compressed form, sparse files are not archived efficiently: they will take space equal to the full file length. (The sparse file handling in afio does not make much sense except in a historical way.)

-k  Skip corrupt data at the beginning of an archive (rather than complaining about unrecognizable input).

-l  With -o, write file contents with each hard link.

    With -t, report hard links.

    With -p, attempt to link files rather than copying them.

-m  Mark output files with a common current timestamp (rather than with input file modification times).

-n  Protect newer existing files (comparing file modification times).

-s size
    Restrict each portion of a multi-volume archive to size characters. This option recognizes the same size suffices as -b. Also, the suffix x denotes a multiple of the -b block size (and must follow any -b specification). size can be a single size or a comma-seperated list of sizes, for example '2m,5m,8m', to specify different sizes for the subsequent volumes. If there are more volumes than sizes, the last specified size is used for all remaining volumes. This option is useful with finite-length devices which do not return short counts at end of media (sigh); output to magnetic tape typically falls into this category. When an archive is being read or written, using -s causes afio to prompt for the next volume if the specified volume length is reached. The -s option will also cause afio to prompt if there is a premature EOF while reading the input. The special case -s 0 will activate this prompting for the next volume on premature EOF without setting a volume length. When writing an archive, afio will prompt for the next volume on end-of-media, even without -s 0 being supplied, if the device is capable of reporting end-of-media. If the volume size specified is not a multiple of the block size set with the -b option, then afio(1) will silently round down the volume size to the nearest multiple of the block size. This rounding down can be suppressed using the -9 option: if -9 is used, afio(1) will write a small block of data, smaller than the -b size, at the end of the volume to completely fill it to the specified size. Some devices are not able to handle such small block writes.

-u  Report files with unseen links.

-v  Verbose. Report pathnames as they are processed. With -t, gives an ls -l style report (including link information).

-w filename
    Treats each line in filename as an -y pattern, see -y.

-x  Retain file ownership and setuid/setgid permissions. This is the default for the super-user; he may use -X to override it.

-y pattern
    Restrict processing of files to names matching shell wildcard pattern pattern. Use this flag once for each pattern to be recognized. With the possible exception of the presence of a leading slash, the complete file name as appearing in the archive table-of-contents must match the pattern, for example the file name 'etc/passwd' is matched by the pattern '*passwd' but NOT by the pattern 'passwd'. See `man 7 glob' for more information on shell wildcard pattern matching. The only difference with shell wildcard pattern matching is that in afio the wildcards will also match '/' characters in file names. For example the pattern '/usr/src/*' will match the file name '/usr/src/linux/Makefile', and any other file name starting with '/usr/src'. Unless the -S option is given, any leading slash in the pattern or the filename is ignored when matching, e.g. /etc/passwd will match etc/passwd. Use -Y to supply patterns which are not to be processed. -Y overrides -y if a filename matches both. See also -w and -W. Note: if afio was compiled without using the GNU fnmatch library, then the full shell wildcard pattern syntax cannot be used, and matching support is limited to patterns which are a full literal file name and patterns which end in '*'.

-z  Print execution statistics. This is meant for human consumption; use by other programs is officially discouraged.

-A  Do not turn absolute paths into relative paths. That is don't remove the leading slash.

-B  If the -v option is used, prints the byte offset of the start of each file in the archive. If your tape drive can start reading at any position in an archive, the output of -B can be useful for doing quick selective restores.

-D controlscript
    Set the control script name to controlscript, see the section on control files below.

-E filename
    Read file extensions, separated by whitespace, from filename. Files with these extensions are not to be compressed when using the -Z option. filename may contain comments preceded by a #. If no -E is given, files with the extensions .Z .z .gz .bz2 .tgz .arc .zip .rar .lzh .lha .uc2 .tpz .taz .tgz .rpm .zoo .deb .gif .jpeg .jpg .tif .tiff and .png will not be compressed.

-F  This is a floppy disk, -s is required. Causes floppy writing in O_SYNC mode under Linux. With kernel version 1.1.54 and above, this allows afio to detect some floppy errors while writing. Uses shared memory if compiled in otherwise mallocs as needed (a 3b1 will not be able to malloc the needed memory w/o shared memory), afio assumes either way you can malloc/shmalloc a chunck of memory the size of one disk. Examples: 795k: 3.5" (720k drive), 316k (360k drive)
    At the end of each disk this message occurs:
     Ready for disk [#] on [output]
     (remove the disk when the light goes out)
     Type "go" (or "GO") when ready to proceed
     (or "quit" to abort):

-G factor
    Specifies the gzip(1) compression speed factor, used when compressing files with the -Z option. Factor 1 is the fastest with least compression, 9 is slowest with best compression. The default value is 6. See also the gzip(1) manual page. If you have a slow machine or a fast backup medium, you may want to specify a low value for factor to speed up the backup. On large (>200k) files, -G 1 typically zips twice as fast as -G 6, while still achieving a better result than compress(1). The zip speed for small files is mainly determined by the invocation time of gzip (1), see the -T option.

-H promptscript
    Specify a script to run, in stead of using the normal prompt, before advancing to the next achive volume. The script will be run with the volume number, archive specification, and the reason for changing to the next volume as arguments. The script should exit with 0 for OK and 1 for abort, other exit codes will be treated as fatal errors.

-J  Try to continue after a media write error when doing a backup (normal behavior is to abort with a fatal error).

-K  Verify the output against what is in the memory copy of the disk (-F required). If the writing or verifying fails the following menu pops up
        [Writing/Verify] of disk [disk #] has FAILED!
    	Enter 1 to RETRY this disk
    	Enter 2 to REFORMAT this disk before a RETRY
    	Enter quit to ABORT this backup
    Currently, afio will not process the answers 1 and 2 in the right way. The menu above is only useful in that it signifies that something is wrong.

-L Log_file_path
    Specify the name of the file to log errors and the final totals to.

-M size
    Specifies the maximum amount of memory to use for the temporary storage of compression results when using the -Z option. The default is -M 2m (2 megabytes). If the compressed version of a file is larger than this (or if afio runs out of virtual memory), gzip(1) is run twice of the file, the first time to determine the length of the result, the second time to get the compressed data itself.

-P progname
    Use the program progname instead of the standard gzip for compression and decompression with the -Z option. See also the -Q, -U and -3 options.

-Q opt
    Pass the option opt to the compression or decompression program used with the -Z option. For passing multiple options, use -Q multiple times. If no -Q flag is present, the standard options are passed. The standard options are -c -6 when the program is called for compression and -c -d when the program is called for decompression. Use the special case -Q "" if no options at all are to be passed to the program.

-R Disk format command string
    This is the command that is run when you enter 2 to reformat the disk after a failed verify. The default (fdformat /dev/fd0H1440) can be changed to a given system's default by editing the Makefile. You are also prompted for formatting whenever a disk change is requested.

-S  Do not ignore a leading slash in the pattern or the file name when matching -y and -Y patterns. See also -A.

-T threshold
    Only compress a file when using the -Z option if its length is at least threshold. The default is -T 0k. This is useful if you have a slow machine or a fast backup medium. Specifying -T 3k typically halves the number of invocations of gzip(1), saving some 30% computation time, while creating an archive that is only 5% longer. The combination -T 8k -G 1 typically saves 70% computation time and gives a 20% size increase. The latter combination may be a good alternative to not using -Z at all. These figures of course depend heavily on the kind of files in the archive and the processor - i/o speed ratio on your machine.

-U  If used with the -Z option, forces compressed versions to be stored of all files, even if the compressed versions are bigger than the original versions, and disregarding any (default) values of the -T and -2 options. This is useful when the -P and -Q options are used to replace the compression program gzip with an encryption program in order to make an archive with encrypted files. Due to internal limitations of afio, use of this flag forces the writing of file content with each hard linked file, rather than only once for every set of hard linked files.

-W filename
    Treats each line in filename as an -Y pattern, see -Y.

-Y pattern
    Do not process files whose names match shell wildcard pattern pattern. See also -y and -W.

-Z  Gzip the files on the way out, in, and passing without links (valid w/ or w/o -F or -K), requires gzip(1) to be in your path. See also the -G, -P, -Q, -T, -2, and -3 options.

-0  Assume input filenames to be terminated with a '\0' instead of a '\n'. When used with find ... -print0, can be used to ensure that any filename can be handled, even if it contains a newline.

-1 warnings-to-ignore-on-exit
    Control if afio(1) should exit with a nonzero code after printing warning messages. This option is sometimes useful when calling afio(1) from inside a backup script or program. afio(1) will exit with a nonzero code on encountering various 'hard' errors, and also (by default) when it has printed certain warning messages during execution. warnings-to-ignore-on-exit is a list of letters which label the warning messages that should not lead to afio(1) exiting with a nonzero code. Defined letters are a for ignoring all possible warnings on exit, and m for ignoring the warning about missing files, which will occur when, on creating an archive, a file whose name was read from the standard input is not found. The default is -1 m. For afio versions 2.4.3 and earlier, the default was -1 a. For afio versions 2.4.4 and 2.4.5, the default was -1 ''.

-2 maximum-file-size-to-compress
    Do not compress any files which are larger than this size when making a compressed archive with the -Z option. The default value is -2 200m (200 Megabytes). This maximum size cutoff lowers the risk that a major portion of a large file will be irrecoverable due to small media errors. If a media error occurs while reading a file that afio has stored in a compressed form, then afio and gzip will not be able to restore the entire remainder of that file. This is usually an acceptable risk for small files. However for very large files the risk of loosing a large amount of data because of this effect will usually be too big. The special case -2 0 eliminates any maximum size cutoff.

-3 filedescriptor-nr
    Rewind the filedescriptor before invoking the (un)compression program if using the -Z option. This is useful when the -P and -Q options are used to replace the compression program gzip with some types of encryption programs in order to make or read an archive with encrypted files. The rewinding is needed to interface correctly with some encryption programs that read their key from an open filedescriptor. If the -P program name matches 'pgp' or 'gpg', then the -3 option must be used to avoid afio(1) reporting an error. Use the special case -3 0 to supress the error message without rewinding any file descriptor. The -3 0 option may also be needed to sucessfully read back encrypted archives made with afio version 2.4.5 and older.

-4  Write archive in the `extended ASCII' format which uses 4-byte inode numbers. Archives using the extended ASCII format are not compatible with any other archiver. This option should not be used unless the set of files to be archived contains over 60 thousand hard links and all set-internal hard links need to be preserved in the archive. A complete news spool could be an example of such a set of files. For such sets, the standard archive format would not necessarily perserve all internal hard links (see the BUGS section).

-9  Do not round down any -s volume sizes to the nearest -b block size. See the -s option.

NOTES
Special-case archive names:

    o   Specify - to read or write the standard input or output, respectively. This disables multi-volume archive handling.

    o   Prefix a command string to be executed with an exclamation mark (!). The command is executed once for each archive volume, with its standard input or output piped to afio. It is expected to produce a zero exit code when all is well.

    o   Use system:file to access an archive in file on system. This is really just a special case of pipelining. It requires a 4.2BSD-style remote shell (rsh(1C)) and a remote copy of afio.

    o   A more elaborate case of the above is [user@]host[%rsh][=afio]:file where the optional user@ component specifies the user name on the remote host, the optional %rsh specifies the (local) name of the remote shell command to use, and the optional =afio specifies the name of the remote copy of the afio command.

    o   Anything else specifies a local file or device. An output file will be created if it does not already exist.

Recognizes obsolete binary cpio(1) archives (including those from machines with reversed byte order), but cannot write them.

Recovers from archive corruption by searching for a valid magic number. This is rather simplistic, but, much like a disassembler, almost always works.

Optimizes pathnames with respect to the current and parent directories. For example, ./src/sh/../misc/afio.c becomes src/misc/afio.c.

CONTROL FILES
Afio archives can contain so-called control files. Unlike normal archive entries, a control file in not unpacked to the filesystem. A control file has a label and some data. When afio encounters a control file in the archive it is reading, it will feed the label and data to a so-called control script. The control script is supplied by the user. It can perform special actions based on the label and data it receives from afio.

Control file labels. The control file mechanism can be used for many things. Examples are putting archive descriptions at the beginning of the archive and embedding lists of files to move before unpacking the rest or the archive.

To distinguish between different uses, the label of a control file should indicate the program that made the contol file and the purpose of the control file data. It should have the form

   programname.kindofdata

where programname is the name of the backup program that generated the control file, and kindofdata is the meaning of the control file data. Some examples are

   tbackup.movelist  tbackup.updatescript
   blebberfiler.archivecontents
   backup_script_of_Joe_User.archivedescription

The user-supplied control script should look at the label to decide what to do with the control data. This way, control files with unknown labels can be ignored, and afio archives maintain some degree of portability between different programs that restore or index them.

Control file labels that are intended to be portable between different backup programs could be defined in the future.

Making control files. When making an archive, afio reads a stream containing the names of the files (directories, ...) to put in the archive. This stream may also contain `control file generators', which are lines with the following format:

    //--sourcename label

Here, the //-- sequence signals that a control file is to be made, sourcename is the path to a file containing the control file data, and label is the control file label. The sourcename must be a regular file or a symlink to a regular file.

A control file will show up as

   //--CONTROL_FILE/label

in an archive listing, where label is the control file label.

Control scripts. A control script is supplied to afio with the

-D controlscript

command line option. The controlscript must be an executable program. The script is run whenever afio encounters a control file while doing a -i -t or -r operation. Afio will supply the control file label as an argument to the script. The script should read the control file data from its standard input. If the script exits with a non-zero exit status, afio will issue a warning message.

If a contol file is encountered and no -D option is given, afio will issue a warning message. To suppress the warning message and ignore all control scripts, -D "" can be used.

An example of a control script is

  #!/bin/sh
  if [ $1 = "afio_example.headertext" ]; then
    #the headertext control file is supposed to be packed as the first
    #entry of the archive
    echo Archive header:
    cat -
    echo Unpack this archive? y/n
    #stdout is still connected to the tty, read the reply from stdout
    read yn <&1
    if [ "$yn" = n ]; then
      #abort
      kill $PPID
    fi
  else
    echo Ignoring unknown control file.
    cat - >/dev/null
  fi

Afio never compresses the control file data when storing it in an archive, even when the -Z option is used. When a control file is encountered by cpio(1) or an afio with a version number below 2.4.1, the data will be unpacked to the filesystem, and named CONTROL_FILE/label where label is the control file label.

BUGS
There are too many options.

Restricts pathnames to 1023 characters, and 255 meaningful elements (where each element is a pathname component separated by a /).

Cannot archive of files larger than 2 GB, even if compiled with large filesystem support. (Pre-2.4.7 versions of afio did not deal with this problem gracefully, see HISTORY file for details.)

Does not use the same default block size as tar(1). tar(1) uses 10 KB, afio uses 5 KB by default. Some tape drives only work with a 10 KB block size, in that case the afio option -b 10k is needed to make the tape work.

There is no sequence information within multi-volume archives. Input sequence errors generally masquerade as data corruption. A solution would probably be mutually exclusive with cpio(1) compatibility.

Degenerate uses of symbolic links are mangled by pathname optimization. For example, assuming that "usr.src" is a symbolic link to "/usr/src", the pathname "usr.src/../bin/cu" is mis-optimized into "bin/cu" (rather than "/usr/bin/cu").

The afio code for handling floppies (-F and -f and -K options) has buggy error handling. afio does not allow one to retry a failed floppy write on a different floppy, and it cannot recover from a verify error. If the floppy handling code is used and write or verify errors do occur, it is best to restart afio completely. Making backups to floppies should really be done with a more specialised backup program that wraps afio.

The Linux floppy drivers below kernel version 1.1.54 do not allow afio to find out about floppy write errors while writing. If you are running a kernel below 1.1.54, afio will happily fail to write to (say) a write protected disk and not report anything wrong! The only way to find out about write errors in this case is by watching the kernel messages, or by switching on the verify (-K) option.

The remote archive facilites (host:/file archive names) have not been exhaustively tested. These facilities have seen a lot of real-life use though. However, there may be bugs in the code for error handling and error reporting with remote archives.

An archive created with a command like 'find /usr/src/linux -print | afio -o ...' will not contain the ownership and permissions of the /usr and /usr/src directories. If these directories are missing when restoring the archive, afio will recreate them with some default ownership and permissions.

Afio will not restore time stamps and owner/group information on symlinks. Afio will often change the time stamp on a directory after having restored it.

A restore using decompression will fail if the gzip binary used by afio is overwritten, by afio or by another program, during the restore. The restore will also fail if any shared libraries needed to start gzip are overwritten during the restore. afio should not normally be used to overwrite the system files on a running system. If it is used in this way, a flag like -Y /bin/gzip can often be added to prevent failure.

The -r option verifies the file contents of the files in the archive against the files on the filesystem, but does not cross-check details like permission bits on files, nor does it cross-check that archived directories or other non-file entities still exist on the filesystem.

There are several problems with archiving hard links. 1) Due to internal limitations, files with hard links cannot be stored in compressed form, unless the -l or -U options are used which force each hard linked file to be stored separately. 2) By default, unless the -t option is used when writing an archive, afio will store only one copy of each file with hard links in the archive, and re-create the hard links on unpacking the archive. However, the capacity for storing hard links is limited to 64K files which have hard links. After processing 64K files with hardlinks (either pointing inside or outside the set of files to be archived), each instance of a new hard linked file will be stored separately and separate files will be created when unpacking. The limitation to 64K files with hard links is not present when the -4 option is used. 3) Archives which contain hard links and which were made with older (pre-2.4.4) versions of afio or with cpio can not always be correctly unpacked. This is really a problem in the archives and not in the current version of afio. The risk of incorrect unpacking will be greater if the number of files or hard links in the archives is larger. Unlike pre-2.4.4 versions of afio and cpio, the current version contains heuristics which greatly reduce the risk of incorrect unpacking. Use of the current version of afio for unpacking older archives with hard links is strongly encouraged. 4) In a selective restore, if the selection predicates do not select the first copy of a file with archive-internal hard links, then all subsequent copies, if selected, will not be correctly restored. 4) Unless the -4 option is used, the inode number fields in the archive headers for files with hard links of the archive will sometimes not contain the actual (least significant 16 bits of) the inode number of the original file.

Some Linux kernels no not allow one to create a hard link to a symbolic link. afio will try to re-create such hard links when unpacking an archive, but might fail due to kernel restrictions.

Due to internal limitations of afio, the use of the -U option forces the writing of file content with each hard linked file, rather than only once for every set of hard linked files.

When it is run without super-user priviliges, afio is not able to unpack a file into a directory for which it has no write permissions, even if it just created that directory itself. This can be a problem when trying to restore directory structures created by some source code control tools like RCS.

EXAMPLES
Create an archive with compressed files:
find .... | afio -o -v -Z /dev/fd0H1440

Install (unpack) an archive with compressed files:
afio -i -v -Z achive

Install (unpack) an archive with compressed files, protecting newer existing files:
afio -i -v -Z -n achive

Create an archive with compressed files on floppy disks:
find .... | afio -o -v -s 1440k -F -Z /dev/fd0H1440

Create an archive with all file contents encrypted by pgp:
export PGPPASSFD=3
find .... | afio -ovz -Z -U -P pgp -Q -fc -Q +verbose=0 -3 3 archive 3<passphrasefile

Create an archive on recordable CDs using the crdrecord utility to write each CD:
find .... | afio -o -b 2048 -s325000x -v '!cdrecord .... -'

Extract a single named file from an archive on /dev/tape:
afio -i -v -Z -y /home/me/thedir/thefile /dev/tape
(If these do not exist yet, afio will also create the enclosing directories home/me/myfiledir under current working directory.)

Extract files matching a pattern from an archive on /dev/tape:
afio -i -v -Z -y '/home/me/*' /dev/tape
(If these do not exist yet, afio will also create the enclosing directories home/me under current working directory.)

SEE ALSO
cpio(1), find(1), tar(1), compress(1), gzip(1).

AUTHORS
Mark Brukhartz ..!ihnp4!laidbak!mdb
Jeff Buhrt uunet!sawmill!prslnk!buhrt
Dave Gymer dgymer@gdcarc.co.uk
Andrew Stevens as@prg.oxford.ac.uk
Koen Holtman (current maintainer) koen@hep.caltech.edu
Anders Baekgaard ab@osiris.cpk.auc.dk