nget(1)

NAME
nget - retrieve files from NNTP hosts

SYNOPSIS
nget [...]

DESCRIPTION
nget retrieves messages matching a regular expression, and decodes any files contained within. Multipart messages are automatically pieced together. Parts from multiple servers will be combined if needed.

OPTIONS

-q/--quiet
    When specified once, will disable printing of auto-updating text to allow the output to be redirected/logged without garbage in it. When specified twice, will disable printing of merely informative messages. Errors will still be printed.

-h/--host host
    Force only the given host to be used for subsequent commands. (Must be configured in .ngetrc.) Can reset to standard auto-choosing method with -h ""

-g/--group group
    Newsgroup to use for subsequent commands. If a host has been specified before with -h, it will retrieve headers only from that host. Otherwise it will retrieve headers for all hosts above _glevel (see configuration section for more info on priorities.)

-G/--quickgroup group
    Like --group, but does not retrieve new headers.

-F/--flushserver host
    Flush all headers for server from current group.

-r/--retrieve regex
    Matches regex against subjects of previously selected group, and retrieves ones that match.

-R/--expretrieve expression
    Like -r, but matches expression instead of merely a regexp. Expression is a postfix expression that can contain the keywords subject, author, lines, bytes, have, req, date, messageid(or mid). Note that the --limit argument does not affect the option, if you want to limit based on number of lines, add it as part of the expression.

-@/--list LISTFILE
    Specify a file to load a list of command line args from. Looks in ~/.nget4/lists/ dir by default.

-p/--path DIRECTORY
    Path to store subsequent retrieves. Also sets -P. Relative to path which nget was started in. (Except in the case of inside a -@, which will be relative to the cwd at the time of the -@.)

-m/--makedirs no,yes,ask
    Make dirs specified by -p and -P. Default is no. If yes, will make dirs automatically. If ask, nget will prompt the user when trying to change to a dir that does not exist. Currently this option only works for single level of non-existance, like mkdir. A future version of nget will support multi-level creating, like mkdir -p.

-T/--testmode
    Causes --retrieve to merely print out all matching files.

--test-multiserver OPT
    Causes testmode to display which servers have parts of each file. OPT may be no to disable(default), long for a verbose output, and short for a more condensed form. (In short mode, the shortname of each server is printed with no seperating space, and it is upper-cased if that server does not have all the parts. If the server has no shortname specified, it defaults to the first char of the server alias.)

-M/--mark
    Mark matched files as retrieved.

-U/--unmark
    Unmark matched files as retrieved. (Automatically sets -dI)

-t/--tries int
    Set maximum number of retries. -1 will retry indefinatly (probably not a good idea).

-l/--limit int
    Set the minimum number of lines a message (or total number of lines for a multi-part message) must have to be considered for retrieval.

-L/--maxlines int
    Set the maximum number of lines a message must have to be considered for retrieval. (-1 for unlimited)

-s/--delay int
    Set the number of seconds to wait between retry attempts.

-i/--incomplete
    Retrieve files with missing parts.

-I/--complete
    Retrieve only files with all parts.

-k/--keep
    Keep temp files.

-K/--keep2
    Keep temp files, and don't try to decode them.

-c/--case
    Match case sensitively.

-C/--nocase
    Match case insensitively.

-d/--dupecheck FLAGS
    Check to make sure you don't already have files. This is done in two ways. The first ("f") is by compiling a list of all files in the current directory, then checking against all messages to be retrieved to see if one of the filenames shows up in the subject. This works reasonably well, though sometimes the filename isn't in the subject. It can also cause problems if you happen to have files in the directory named silly things like "a", in which case all messages with the word "a" in them will be skipped. However, it is still smart enough not to skip messages that merely have a word containing "a".
    The second method ("i") is by setting a flag in the header cache that will prevent it from being retrieved again. You can use combos such as -dfi to check both, -dFi to only check the flag, -dfI to only check files, etc.
    The third ("m") will cause files that are found by the dupe file check ("f") to be marked as retrieved in the cache. (Useful for handling crossposted binaries and/or binaries saved with another newsreader.)

-D/--nodupecheck
    Don't check either of the --dupecheck methods, retrieve any messages that match.

-P/--temppath DIRECTORY
    Store temporary files in path instead of the current dir.

-N/--noconnect
    Do not connect to any server. Useful for trying to decode as much as you have. (if you got stuff with -K or ngetlite.)

-w/--writelite LITEFILE
    Write a list of parts to retrieve with ngetlite.

-?/--help
    Show help.

CONFIGURATION
Upon startup, nget will read ~/.nget4/.ngetrc for default configuration values and host/group aliases. An example .ngetrc should have been included with nget.

nget will also check ~/_nget4/ and _ngetrc if needed, to handle OS and filesystems that can't (or won't) handle files starting with a period.

Options are specified one per line in the form:
key=value
Subsections are specified in the form
{section_name
data
}
where data is any number of options.

Global options that can be specified are:
String: cachedir (Specifies a different location to store cache files. Could be used to share a single cache dir between a trusted group of users, to reduce HD/bandwidth usage, while still allowing each user to have their own config/midinfo files.)
Integral: limit, tries, debug, delay, usegz(-1=default, 0=uncompressed, 1-9=gzip compression level), timeout, maxstreaming, idletimeout, maxconnections, penaltystrikes(number of consecutive connect errors before penalizing a server, -1 to disable penalization), initialpenalty(number of seconds to ignore a penalized server for)
Float: curservmult, penaltymultiplier(multiplier for penalty time for each time the penalty time runs out and the server continues to be down)
Boolean(0/1): case, complete, dupeidcheck, dupefilecheck, quiet, tempshortnames(1=Use 8.3 tempfile names (for old dos partitions, etc), 0=Use 17.3 tempfile names), unequal_line_error, fullxover
yes/no/ask: makedirs
no/short/long: test_multiserver

Host configuration is done in the halias section, with a subsection for each host containing its address, unique id, and optionally shortname, user, pass, fullxover, maxstreaming, idletimeout, and linelenience items. The "id" field must be unique, and should not be changed after you have used it (since it could leave leftovers in your cache files, unless you delete them first). The "id" must be greater than 0 and less than ULONG_MAX. (usually 4294967295). In other words, don't use id 0.
The maxstreaming option sets how many xover commands will be sent at once, when using fullxover. maxstreaming=0 will disable streaming. Note that setting maxstreaming too high can cause your connection to deadlock if the write buffer is filled up and the write command blocks, but the server will never read more commands since it is waiting for us to read what it has already sent us. The default value is maxstreaming=64.
The linelenience option may be specified as either a single int, or two ints seperated by a comma. If only a single int, X is specified, then it will be interpeted as shorthand for "-X,+X". These values specify the ammount that the real (recieved) number of lines (inclusive) for an article may deviate from the values returned by the server in the header listings. For example, "-1,2" means that the real number of lines may be one less than, equal to, one greater than, or two greater than the expected amount. The default linelenience is 0.
For example, the following host section defines a single host "host1", with nntp authentication for user "bob", password "something", and the fullxover option enabled. (fullxover retrieves all headers from the server, including gaps in the header list, while normally it only retrieves headers more recent than the latest.)
{halias
{host1
addr=news.host1.com
id=3838
user=bob
pass=something
fullxover=1
linelenience=-1,2
}
}

Multiserver priorities are defined in the hpriority section. Multiple priority groups can be made, and different newsgroups can be configured to use their own priority grouping, or they will default to the "default" group. The hpriority section contains a subsection for each priority group, with data items of server=prio-multiplier, and the special items _level=float and _glevel=float. _level sets the priority level assigned to any host not listed in the group, and _glevel sets the required priority needed for -g to automatically use that host. Both _level and _glevel default to 1.0 if not specified. The priority group "trustsizes" also has special meaning, and is used to choose which servers reporting of article line/byte counts to trust when reporting to the user. When retrieving articles, nget will multiply the current servers priority by the global setting "curservmult" (default 2.0), this can be used to avoid excesive server switching. (set it to 1.0 if you want to disable it).
For example, the following section defines the default priority group and the trustsizes priority group. If all hosts have a certain article, goodhost will be most likely to be chosen, and badhost least likely. It also sets the default proirity level to 1.01, meaning any hosts not listed in this group will have a priority of 1.01. When using -g without first specifying a host, only those with prios 1.2 or above will be selected.
{hpriority
{default
_level=1.01
_glevel=1.2
host1=1.9
goodhost=2.0
badhost=0.9
}
{trustsizes
goodhost=5.0
badhost=0.1
}
}

Newsgroup aliases are defined in the galias section. An alias can be a simple alias=fullname data item, or a subsection containing group=, prio=, and usegz= items. The per-group usegz setting will override the global setting.
For example, the following galias section defines an alias of "abpl" for the group "alt.binaries.pictures.linux", and "chocobo" for the group "alt.chocobo". In addition, the chocobo group is assigned to use the chocoprios priority grouping when deciding what server to retrieve from.
{galias
abpl=alt.binaries.pictures.linux
{chocobo
group=alt.chocobo
prio=chocoprios
}
}

EXIT STATUS
On exit, nget will display a summary of the run. The summary is split into three parts:

OK  Lists successful operations.

    total
        Total number of "logical messages" retrieved (after joining parts).

    uu  Number of uuencoded files.

    base64
        Number of Base64 (Mime) files.

    XX  Number of xxencoded files.

    binhex
        Number of Binhex encoded files.

    plaintext
        Number of plaintext files saved.

    qp  Number of Quoted-Printable encoded files.

    dupe
        Number of decoded files that were exact dupes of existing files, and thus deleted.

    skipped
        Number of files that were queued to download but turned out to be dupes after decoding earlier parts and comparing their filenames to the subject line. (Same method thats used for the dupe file check when queueing them up, just that the filename(s) of any decoded files cannot be known until they are downloaded, so some of the checking must occur during the run rather than at queue time.)

    group
        Number of groups successfully updated.

WARNINGS

    group
        Updating group info failed for some (but not all) attempted servers.

    retrieve
        Article retrieval failed for some (but not all) attempted servers.

    undecoded
        Articles were not decoded (usually because -K was used).

    unequal_line_count
        Some articles retrieved had different line counts than the server said they should. (And unequal_line_error is set to 0).

    dupe
        Number of decoded files that had the same name as existing files, but different content.

ERRORS
    Lists errors that occured. In addition, the exit status will be set to a bitwise OR of the codes of all errors that occured.

    decode (exit code 1)
        Number of file decoding errors.

    path (exit code 2)
        Errors changing to paths specified with -p or -P.

    user (exit code 4)
        User errors, such as trying -r without specifying a group first.

    retrieve (exit code 8)
        Number of times article retrieval failed for all attempted servers.

    group (exit code 16)
        Number of times header retrieval failed for all attempted servers.

    fatal (exit code 128)
        Error preventing further operation, such as "No space left on device".

    other (exit code 64)
        Any other kind of error.

EXAMPLES
get listing of all files matching penguin.*png from alt.binaries.pictures.linux (note this is a regex, equivilant to standard shell glob of penguin*png.. see the regex(7) or grep manpage for more info on regular expressions.)
nget -g alt.binaries.pictures.linux -DTr "penguin.*png"

retrieve all the ones that have more than 50 lines:
nget -g alt.binaries.pictures.linux -l 50 -r "penguin.*png"

equivilant to above, using -R:
nget -g alt.binaries.pictures.linux -R "lines 50 > subject penguin.*png == &&"
(basically (lines > 50) && (subject == penguin.*png))

flush all headers from host goodhost in group alt.binaries.pictures.linux:
nget -Galt.binaries.pictures.linux -Fgoodhost

NOTES
Running multiple copies of nget at once should be safe. It uses file locking, so there should be no way for the files to actually get corrupted. However if you have two ngets doing a -g on the same group at the same time, it would duplicate the download for both processes. If you are using -G there is no problem at all. (Theoretically you might be able to cause some sort of problems by downloading the same files from the same group in the same directory at the same time..)

ENVIRONMENT

HOME
    Where to put .nget4 directory. (put nget files $HOME/.nget4/)

NGETHOME
    Override HOME var (put nget files in $NGETHOME)

NGETCACHE
    Override HOME/NGETHOME vars and .ngetrc cachedir option (put nget cache files in $NGETCACHE)

FILES

~/.nget4/
    All configuration and cache files are stored here. Changed to .nget4/ because cache format changed in nget 0.16. (The 4 in the directory name is for file format version 4, not nget version 4.) To upgrade a .nget3 directory to .nget4, simply run "mv ~/.nget3 ~/.nget4 ; rm ~/.nget4/*,cache*"

~/.nget4/.ngetrc
    Configuration file. If you store authentication information here, be sure to set it readable only by owner.

~/_nget4/_ngetrc
    Alternate location, use this if you can't create a dir/file starting with a period.

~/.nget4/lists/
    Default directory for listfiles.

BUGS
The config examples would be easier to read if they were indented, but I'm no man page guru. If someone wants to clean up this manpage and fix the little things like that, feel free.

In multi part messages, part 0/x is usually a description.. this will be saved. But sometimes the description will be in 1/x, in which case it is not saved. Well, I found the UUInfo stuff, so it'll now print out this information.. however a lot of the time it is useless stuff, so I'm not too sure about saving it to a file all the time.

AUTHOR
Matthew Mueller <donut@azstarnet.com>

The latest version, and other programs I have written, are available from:
http://www.azstarnet.com/~donut/programs/

ACKNOWLEDGEMENTS
Frank Pilhofer, who write uulib, which nget depends upon for uudecoding the files once they are downloaded. http://www.fpx.de/fp/Software/UUDeview/

The Unix-socket-faq, which my url for has gone bad, but is supposedly posted monthly on comp.unix.programmer.

Beej's Guide to Network Programming at http://www.ecst.csuchico.edu/~beej/guide/net/

Jean-loup Gailly and Mark Adler, for the zlib library.

SEE ALSO
ngetlite(1), regex(7), grep(1)