ubh, The Usenet Binary Harvester - Documentation

Index
Home Past News Screenshots Download Documentation Mailing Lists Links
| # ubh - The Usenet Binary Harvester NAME SYNOPSIS DESCRIPTION OPTIONS INSTALLATION Win32 / ActivePerl Unix Mac OS X MacPerl SETUP USAGE EXAMPLES CREDITS WEBPAGE COPYRIGHT and LICENSE # NAME `ubh` - The Usenet Binary Harvester # SYNOPSIS `ubh -c file (-S|-M) -A -C -d -D -g [(-a|-s)]|[(-f|-l) num] -i [(-I|-X) pattern] -L -n -O -r -u -w -y -Z -z` # DESCRIPTION `ubh` is the Usenet Binary Harvester, a Perl program which automatically discovers, downloads, and decodes single and multi-part binary Usenet postings. `ubh` decodes single and multi-part uuencoded binaries. `ubh` also decodes single part MIME base64-encoded image, audio, and video attachments, and `application/octet-stream` attachments. It also combines and decodes multi-part `message/partial` binaries. `ubh` uses a standard `.newsrc` file to control which groups to process, and uses the `.newsrc` to keep track of articles already processed. You can specify search filters to select articles to download via Perl regular expression syntax. By default `ubh` will only consider articles with a well-formed `Subject` header. A well-formed `Subject` header is one that contains a file name with an extension which matches the extension filter. Multi-part `Subject` headers will be recognized if they contain a part/total designator of the form `[`m`/`n`]` or `(`m`/`n`)`, where m is the part number and n is the total number of parts. Part numbers begin with `1` and may or may not contain leading zeroes. A part number of `0` is ignored. `ubh` provides an interactive article preselection option to allow you to preview the `Subject:` headers for multi-part binaries, and specify which binaries you wish to download. `ubh` runs equally well under Unix-based Perl, Active Perl on Win32 platforms, and Mac OS X. `ubh` requires `Net:NNTP` and `News:Newsrc` (which itself requires `Set::IntSpan`). See the section INSTALLATION for details. # OPTIONS These options apply to single part article processing:
Option | Description
--- | ---
-S | Process only single part articles.
-g | This option ("g" for "greedy") will download and process each unread article even if the subject does not contain a filename which matches the single part extension filter.
These options apply to multi-part article processing:
Option | Description
--- | ---
-M | Process only multi-part articles.
-i | Interactive preselection of multi-part articles.
These options apply to both single part and multi-part article processing:
Option | Description
--- | ---
-A | Enables disk-based article assembly. This will download the articles to disk (instead of RAM) prior to decoding.
-C | Changes all non-alphanumeric characters in a filename to '_'. This will eliminate spaces in filnames, as well as all other undesireable (and possibly illegal) characters.
-d | Diagnostic mode. Downloads and writes all unread articles in raw form. This occurs prior to single and multi-part filtering. It's very useful to look at the raw articles to see why they are failing to be selected for downloading. Helpful for reverse-engineering new or bizarre encoding formats. You can also use this to perform your own post-processing directly on the raw articles.
-D | Multi-part diagnostic mode. Downloads and writes all selected, complete multi-part articles. Very useful to look at the raw articles to see why they are failing to be unencoded. Helpful for reverse-engineering new or bizarre encoding formats. You can also use this to perform your own post-processing directly on the raw articles.
-c file | Use file as configuration file, instead of the default. On Win32 platforms, the default is `ubhrc`. On Unix platforms, the default is `.ubhrc`.
-a | Process all articles, but disregard the newsrc (ie, consider all articles even if they are marked as read in the newsrc, and do not catch up the group at the end of processing of the group).
-f num | Process the first num articles. Updates newsrc.
-l num | Process the last num articles. Updates newsrc.
-s | Log all subjects to `subjects.log`. Log multi-part subjects to `multiparts.log`. Doesn't download anything. Disregards newsrc.
-I regexp | Inclusion search filter (double quote on command line). regexp is any valid Perl regular expression.
-X regexp | Exclusion search filter (double quote on command line). regexp is any valid Perl regular expression.
-L | Long filenames - uses the article subject as the filename. This makes life easier because many folks encode their files with terribly vague filenames.
-n | Updates the `.newsrc` every time an article is processed, instead of waiting until the entire group has been processed.
-O | Forces `ubh` to overwrite duplicate filenames instead of creating a unique filename.
-r | Logs rejected subjects to `rejects.log` in the group directory. Logs rejected single part and multi-part articles. Excellent for quality control to see if ubh is rejecting any binaries, and essential for diagnosing why articles are being rejected. Normally the rejected articles will contain SPAM or discussion, but occasionally ubh will reject an arcane format or mal-formed MIME-formatted message. Rejected multi-part articles logged with this option in are in their assembled stated, prefaced by the headers from the first article.
-u | Prints out a brief usage summary.
-w | Prints out warranty information.
-y | `chmod 0666` on all output files.
-Z | Produces lots and lots of logs.
-z | Marks articles that don't pass inclusion/exclusion as read. This cleans up the `.newsrc` dramatically.
# INSTALLATION ubh requires the following Perl modules: `Net::NNTP`, `News::Newsrc`, and `Set::IntSpan`. These modules may already be installed on your system. To determine whether these modules have already installed, use the following commands:
    perl -e "use Net::NNTP;"
    perl -e "use News::Newsrc;"
    perl -e "use Set::IntSpan;"
`Net::NNTP` is part of the `libnet` distribution, which provides an entire family of networking modules. If you need to install one or more of these modules, follow the instructions below for your particular platform. ## Win32 / ActivePerl You will use the Perl Package Manager (PPM) to download and install the modules. At the PPM prompt, enter the following commands:
    install libnet
    install Set-IntSpan
    install News-Newsrc
## Unix You will download the modules from your nearest CPAN mirror.
    modules/by-module/Net/libnet-1.0607.tar.gz
    modules/by-module/Set/Set-IntSpan-1.07.tar.gz
    modules/by-module/News/News-Newsrc-1.07.tar.gz
Feel free to use more recent versions of these modules, if available. For each of those files, execute the following commands:
    gunzip < module.tar.gz | tar xvf -
    cd module
    perl Makefile.PL
    make
    make test
    make install
For more details, review the installation instructions provided with each module. ## Mac OS X Use the Unix `.tar.gz` distribution. You need to download (from Apple) "Unix tools for OS X", and then proceed with the module installation as given above. ## MacPerl Use the Unix `.tar.gz` distribution. Proceed with the module installation as given above. # SETUP You specify which groups to process with a standard newsrc file. The format of a newsrc file is one newsgroup name per line, followed by a colon. `ubh` will update this file and place the numbers of articles it has already processed for a given group on that group's line in the newsrc file after the colon. The colon character denotes that you are subscribed to the newsgroup. You can un-subscribe from a group by changing the colon to an exclamation point; this will cause `ubh` to skip over that newsgroup during processing. Here is an example newsrc file:
    alt.binaries.sounds.mp3.1990s: 15558-23146
    alt.binaries.sounds.mp3.1980s: 30139-35021
    alt.binaries.pictures.autos:
    alt.binaries.pictures.cartoons! 63671-64660
    alt.binaries.pictures.hockey: 1-1406
You specify program options with a ubhrc file. On Win32 systems, the default ubhrc file name is `ubhrc`. On Unix systems, the the default ubhrc file name is `.ubhrc`. Here are the available options. Specify one option per line, OPTION`=`value. Comments begin with `#`.
Keyword | Description
--- | ---
NNTPSERVER | Specifies the name of the news server to connect to. Default is `news`.
NNTPRETRIES | Number of times to retry NNTP commands.
NEWSRCNAME | The name of the newsrc file. On Win32 systems, the default newsrc file name is `newsrc`. On Unix systems, the default newsrc file name is `.newsrc`.
DATADIR | Directory to store downloaded group subdirectories and downloaded binaries. Default is `data`.
FORCEDIR | Forces output to a specific directory instead of the default `data/`newsgroup-name. Handy for multiple newsgroups which hold similar stuff.
TEMPDIR | Directory to store encoded articles prior to decoding. Default is DATADIR.
PERMISSION | Permission to apply to created subdirectories. Default is `0777`.
MULTI_EXT | Multi-part article file extensions. Any valid Perl regular expression. Default is `(?i)asf|avi|gif|jpg|mov|mpg|mpeg|rm|rar`.
SINGLE_EXT | Single part article file extensions. Any valid Perl regular expression. Default is `(?i)asf|avi|gif|jpg|mov|mpg|mpeg|rm|rar`.
EXTENSIONS | Sets both `MULTI_EXT` and `SINGLE_EXT`.
ACCOUNT and PASSWORD | News server account and password. You must define both of these if your server requires them. By default, `ubh` will access the server without authentication.
OPT_g OPT_i OPT_d OPT_D OPT_a OPT_f OPT_l OPT_r OPT_s OPT_S OPT_M OPT_I OPT_X OPT_C OPT_L OPT_z OPT_Z OPT_y | All of the command-line options (except `-u`, `-c`, and `-w`) can be set in the ubhrc file, using `OPT_`x where x is the corresponding command line option letter.
Here is an example ubhrc file. This can be used to harvest single and multi-part binaries in the pictures and multimedia newsgroups.
    NNTPSERVER = binaries.newsfeeds.com
    NEWSRCNAME = newsrc_nfdc
    DATADIR    = data
    MULTI_EXT  = (?i)asf|avi|gif|jpg|mov|mpg|rm
    SINGLE_EXT = (?i)asf|avi|gif|jpg|mov|mpg|rm
    ACCOUNT = bart
    PASSWORD = the+simpsons
    OPT_g = 1
Here is another example ubhrc file. This demonstrates how to search for MP3s by a particular set of artists:
    NNTPSERVER = mp3.newsfeeds.com
    NEWSRCNAME = newsrc_mp3
    DATADIR    = data
    MULTI_EXT  = (?i)mp3
    SINGLE_EXT = dontcare
    OPT_a = 1
    OPT_M = 1
    OPT_I = (?i)oasis|horton.*heat|smiths|morrissey|bjork|foo.*fighters|green.*day
# USAGE EXAMPLES Here are some command-line usage examples:
    ubh -i -M -I "(?i)rem|r.e.m.|u2|korn|hanson"
    ubh -S -l 1000
# CREDITS base64 decoding is based on MIME::Base64. uuencoded decoding based on the uudecode implementation in the Perl Power Tools project. # WEBPAGE http://ubh.sourceforge.net/ # COPYRIGHT and LICENSE Copyright © 2000 Gerard Lanois `gerard 'at' users.sourceforge.net` P.O. Box 507264 San Diego, CA 92150-7264 This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

`gerard 'at' users.sourceforge.net`