GNU.WIKI: The GNU/Linux Knowledge Base

  [HOME] [PHP Manual] [HowTo] [ABS] [MAN1] [MAN2] [MAN3] [MAN4] [MAN5] [MAN6] [MAN7] [MAN8] [MAN9]

  [0-9] [Aa] [Bb] [Cc] [Dd] [Ee] [Ff] [Gg] [Hh] [Ii] [Jj] [Kk] [Ll] [Mm] [Nn] [Oo] [Pp] [Qq] [Rr] [Ss] [Tt] [Uu] [Vv] [Ww] [Xx] [Yy] [Zz]


NAME

       webcheck - website link checker

SYNOPSIS

       webcheck [OPTION]...  URL

DESCRIPTION

       webcheck  will  check  the  document  at the specified URL for links to
       other documents, follow these links recursively and  generate  an  HTML
       report.

       -i,  --internal=PATTERN
              Mark URLs matching the PATTERN (perl-type regular expression) as
              an internal link.  Can be used multiple times.   Note  that  the
              PATTERN  is  matched  against  the full URL.  URLs matching this
              PATTERN will be considered internal, even if they match  one  of
              the --external PATTERNs.

       -x,  --external=PATTERN
              Mark URLs matching the PATTERN (perl-type regular expression) as
              an external link.  Can be used multiple times.   Note  that  the
              PATTERN is matched against the full URL.

       -y, --yank=PATTERN
              Do  not  check  URLs  matching  the  PATTERN  (perl-type regular
              expression).  Like the -x flag, though this  option  will  cause
              webcheck  to not check the link matched by regex whereas -x will
              check the link but not  its  children.   Can  be  used  multiple
              times.  Note that the PATTERN is matched against the full URL.

       -b, --base-only
              Consider  any URL not starting with the base URL to be external.
              For example, if you run
                  webcheck -b http://www.example.com/foo
              then http://www.example.com/foo/bar will be considered  internal
              whereas http://www.example.com/ will be considered external.  By
              default all the pages on the site will be considered internal.

       -a, --avoid-external
              Avoid external links.  Normally if webcheck is examining an HTML
              page and it finds a link that points to an external document, it
              will check to see if that external document exists.   This  flag
              disables that action.

       --ignore-robots
              Do   not  retrieve  and  parse  robots.txt  files.   By  default
              robots.txt files are retrieved and honored.  If you are sure you
              want to ignore and override the webmaster's decision this option
              can be used.
              For more  information  on  robots.txt  handling  see  the  NOTES
              section below.

       -q, --quiet, --silent
              Do not print out progress as webcheck traverses a site.

       -d, --debug
              Print  debugging  information  while  crawling  the  site.  This
              option is mainly useful for developers.

       -o, --output=DIRECTORY
              Output directory. Use to specify the  directory  where  webcheck
              will  dump  its reports. The default is the current directory or
              as specified by config.py. If this directory does not  exist  it
              will be created for you (if possible).

       -c, --continue
              Try  to  continue  from  a  previous run. When using this option
              webcheck will look for a webcheck.dat in the  output  directory.
              This  file  is  read to restore the state from the previous run.
              This allows webcheck to continue a previously  interrupted  run.
              When  this option is used, the --internal, --external and --yank
              options will be ignored as  well  as  any  URL  arguments.   The
              --base-only  and  --avoid-external options should be the same as
              the previous run.
              Note that this option is experimental  and  it's  semantics  may
              change  with  coming  releases  (especially in relation to other
              options).  Also note that the stored files are not guaranteed to
              be compatible between releases.

       -f, --force
              Overwrite  files  without  asking.   This option is required for
              running webcheck non-interactively.

       -r, --redirects=N
              Redirect depth. the number of redirects webcheck  should  follow
              when following a link. 0 implies to follow all redirects.

       -u, --userpass=URL
              Specify  a URL with username and password information to use for
              basic authentication when visiting the site.
              e.g. http://test:secret@example.com/
              This option may be specified multiple times.

       -w, --wait=SECONDS
              Wait SECONDS between document retrievals. Usually webcheck  will
              process  a  url  and immediately move on to the next. However on
              some loaded systems it may be desirable to have  webcheck  pause
              between  requests.   This  option can be set to any non-negative
              number.

       -v, --version
              Show version of program.

       -h, --help
              Show short summary of options.

URL CLASSES

       URLs are divided into two classes:

       Internal URLs are retrieved and  the  retrieved  item  is  checked  for
       syntax.   Also, the retrieved item is searched for links to other items
       (of any class) and these links are followed.

       External URLs are only retrieved to test whether they are valid and  to
       gather  some  basic  information  from them (title, size, content-type,
       etc).  The retrieved items are not inspected for links to other items.

       Apart from  their  class,  URLs  can  also  be  considered  yanked  (as
       specified  with  the --yank or --avoid-external options).  The URLs can
       be either internal or external and will not be retrieved or checked  at
       all.  URLs of unsupported schemes are also considered yanked.

EXAMPLES

       Check  the  site www.example.com but consider any path with "/webcheck"
       in it to be external.
           webcheck http://www.example.com/ -x /webcheck

NOTES

       When checking  internal  URLs  webcheck  honors  the  robots.txt  file,
       identifying itself as user-agent webcheck. Disallowed links will not be
       checked at all as if the -y option was specified for that URL. To allow
       webcheck to crawl parts of a site that other robots are disallowed, use
       something like:
           User-agent: *
           Disallow: /foo

           User-agent: webcheck
           Allow: /foo

ENVIRONMENT

       <scheme>_proxy
              Proxy url for <scheme>.

REPORTING BUGS

       Bug  reports  shoult  be  sent   to   the   mailing   list   <webcheck-
       users@lists.arthurdejong.org>.   More information on reporting bugs can
       be found on the webcheck homepage:
       http://arthurdejong.org/webcheck/

COPYRIGHT

       Copyright © 1998, 1999 Albert Hopkins (marduk)
       Copyright © 2002 Mike W. Meyer
       Copyright © 2005, 2006, 2007, 2008, 2009, 2010 Arthur de Jong
       webcheck is free software;  see  the  source  for  copying  conditions.
       There  is  NO  warranty;  not even for MERCHANTABILITY or FITNESS FOR A
       PARTICULAR PURPOSE.
       The files produced as output from the  software  do  not  automatically
       fall  under  the  copyright  of  the software, unless explicitly stated
       otherwise.



  All copyrights belong to their respective owners. Other content (c) 2014-2018, GNU.WIKI. Please report site errors to webmaster@gnu.wiki.
Page load time: 0.089 seconds. Last modified: November 04 2018 12:49:43.