rawler: Crawl your website and find broken links with Ruby

Need a quick-and-dirty way to find broken links on your web site? Rawler from Oscar Del Ben is a Ruby gem that gives you a command line tool to crawl your site, looking for errors.

Install via Rubygems:

gem install rawler

For usage, just execute the command rawler –help:

~ » rawler --help                                                                ~ 255 ↵ 
Rawler is a command line utility for parsing links on a website

      rawler [options]

where [options] are:
  --username, -u <s>:   HTT Basic Username
  --password, -p <s>:   HTT Basic Password
       --version, -v:   Print version and exit
          --help, -h:   Show this message

Point Rawler to your URL and you’ll get a list of followed links and their HTTP status codes:

~ » rawler                                               ~ 130 ↵ 
301 -
200 -
200 -
200 -
301 -
200 -
200 -
200 -
200 -


The roadmap includes:

  • Follow redirects, but still inform about them
  • Respect robots.txt
  • Export to html

If you want to help out, fork the project and contribute.

[Source on GitHub]


Sign in or join to comment

0:00 / 0:00