4.2.4 How deep should we go?
Keywords: Plucker_Doc

Next: 4.2.5 Plucker Home and Up: 4.2 Invoking the parser Previous: 4.2.3 Description filenames Contents Index

##
4.2.4 How deep should we go?

The default operation of the parser is to parse the description file and any links in it, but this can be changed using the -M or --maxdepth option. To make sure that we don't bite off more than we can chew, we can add the --stayonhost option to avoid following any external links,

`% Spider.py -H http://www.plkr.org/ -M3 --stayonhost -f PluckerDB`

This would download the index page for Plucker's web site, any links within this page and also any links within those pages, but it would disregard any links outside the plkr.org host.

These options are useful when you specify a specific host using -H. If you use a description file it is better to use the MAXDEPTH/STAYONHOST flags for each link instead.