NNTPBTR(1)

NAME
nntpbtr - NNTP bulk transfer - retrieve netnews from a remote NNTP server

SYNOPSIS
nntpbtr [ -g newsgroups ] [ -d distribution ] [ -t timeout ] [ -s stacksize ] [ -m mindiskspace ] [ -D ] [ -r ] [ -k ] [ -i ] [ -f [ connecttimeout ] ] [ -I maxinfiles ] [ -l lockfile ] ... hostname

DESCRIPTION

Nntpbtr is an advanced passive NNTP client for bulk transfer of NEWS. It will connect to a remote NNTP server and retrieve articles in Usenet newsgroups that have arrived after a particular time (typically the last time it was invoked) for processing by your local news system.

OPTIONS

-g newsgroups
    If specified then overrides default "any" newsgroups for hostname. You have to use '*' for not-exact matching, see RFC977. Different names have to be separated by COMMAS. For example: -g 'relcom,relcom.*,comp.*' . (Don't forget about shell escape with '*'). Maximum length of this list have to be less than 500 bytes.

-d distribution
    If specified then overrides default "any" distribution. Using distributions is not recommended - they're only really included for completeness. Under current NNTP implementations, setting distributions requires the server to open each article, search through for the distributions line and check it against the supplied list. This will not only increase the load on the server substantially, but increase the amount of time for the connection.

-t timeout
    Maximum physical time to run. Nntpbtr will end run with save an unretrieved articles list after this time expire.

-s stacksize
    If specified then overrides default 25 articles in stack. The stack is used to speedup article transmission, nntpbtr attempt to push stacksize ARTICLE commands to server before first read from it. Also the size of TCP read buffer depends from stacksize .

-m mindiskspace
    If specified then check mindiskspace free blocks space on disk to continuation. Nntpbtr checks 30000 blocks by default.

-I maxinfiles
    In case of CNEWS, nntpbtr don't create number of files in in.coming directory greater than maxinfiles (no check by default and if it is 0).

-D  Enable debugging. This diverts reporting to stderr and turns on extra debugging output. Additional flags produce more output.

-r  Don't use "MODE READER" command to NNTP server. It is the only for future NNTP servers.

-k  Don't increase receive TCP buffer, use the system default.

-i  Use "incore" version of DBZ library. This option very accelerate the check of article existance in local history data base, but it essential increase the used memory, see dbz.

-f  [ connecttimeout ] If connect fails more than connecttimeout or 900 seconds than it is hard error.

-l lockfile
    If specified then nntpbtr check the locking of lockfile in addition to hostname lock. It is used to prevent from working two or more process on the same low throughput physical link.

hostname
    The hostname of the remote NNTP server to connect to. This must be specified.

CONFIGURATION FILES

There are one configuration file used by nntpbtr.

The file nntpbtr-<hostname> contains the time when nntpbtr last connected to the NNTP server at <hostname>. nntpbtr can then use this time to pick up all the articles that have arrived at the server since the last session. It may be followed on subsequent lines by a list of message IDs of articles that are to be retrieved from the server in the next session.

The time is in the standart NNTP presentation "YYMMDD HHMMSS". It is the first line of nntpbtr-<hostname> file.

RESETTING TIME OF LAST CONNECT

If you'll start nntpbtr first, it should ask ALL articles from remote server (most of them would not be requested due to _history_ data base). If you want to get only _NEW_ articles, call first 'nntpbtr' with illegal group name (for example, nntpbtr -g BAD hostname).

OPERATION

When run, nntpbtr will first lock the addition lock files, nntpbtr-<hostname> file and retrieve the start time for the specified server from this file. If the locking is not possible then run stop. If file doesn't exist or bad then 1-Jan-1970 will be used.

The current time will be obtained from UDP/TCP time service of NNTP server to use as the start time for the next session.

nntpbtr will now connect to the NNTP server at the remote host. if the -r option is specified, then a 'MODE READER' command will not be sent. It can be used if you don't shure about the correctness of NNTP server behaviour.

A NEWNEWS request will now be issued, asking for all the articles that have arrived since the specified start time. The server will respond with a list of message IDs. If there is already unprocessed articles list in nntpbtr-<hostname> file then nntpbtr will run through this list at first, before issue a NEWNEWS.

After it nntpbtr moves into an article retrieval stage. It will go through the list of message IDs, check it for non-existence in local history data-base and request them in turn from the server, adding each article to the batch of articles being either stored in the incoming news directory or piped to rnews. When a batch is found to be larger than the maximum size, it will be submitted to the news system.

During retrieval nntpbtr use stacking to speedup the articles read. It send some more commands (default 25) to NNTP server before it do the real read operation. It smooth the data flow from server and support the full usage of high throughput channels with delay (satellite link). For this purpose nntpbtr also increase receive TCP buffer size and it produce large effect with NNTP server on like Ultrix 4.X machines: it has 16 Kbytes default TCP buffers instead of 4Kb.

Once all the articles have been retrieved, the final batch of articles will be submitted and "newsrun" process started. The nntpbtr repeat its runs up to no new news exist in NNTP server. And finaly previously obtained time to use for the next NEWNEWS will be written to nntpbtr-<hostname>. If hard error has occurred, then the message IDs of any unretrieved articles are also written to this file, for retrieval in the next session.

But nntpbtr doesn't suppose the TCP channel breaks as hard error if it is not specified by -f option. It correctly reconnects to NNTP server and continue operation. The invalid answer of NNTP server during handshaking also produce the same. It can lead to hard loop if your NNTP server reject you. But it is implemented due to occasional errors during NNTP server start. The -f option restrict loop up to 900 seconds or connecttimeout time.

In process, nntpbtr monitor the remained disk space and the number of unprocessed CNEWS batch files. It don't overflow disk and attempt to do the wait of the relaynews process work or exit (it is controled by the configuration of nntpbtr during compiling).

Some critical information is logged to syslog and also is copied to stderr.

RETURN CODES

Nntpbtr returns a series of return codes which may be useful to controlling programs:-
    0 - Successful completion.
    1 - nntpbtr-<hostname> is busy by another nntpbtr proccess.
    2 - No disk free space or inodes for news.
    3 - Hard error occurred during nntpbtr work.

FILES
/usr/spool/news/nntpbtr-<hostname>

AUTHOR
Leonid Yegoshin <egoshin@ihep.su>, LY22

SEE ALSO
rnews(8), dbz(3), relaynews(8),
RFC977 - Network News Transfer Protocol (NNTP),
RFC1036 - Usenet Article Format standard.