[Revised 8/14/07: updated the LinkTiger Review and the section on Xenu's reports.]
There are a number of web sites, and at least one desktop utility, that offer link checking for your web site. They attempt to find all of your broken links. I tried out several of the free ones, and what I found was that, basically, you get what you pay for.
Here's what I was hoping to find:
I ran each utility against the site belonging to the Angeles Chapter of the Sierra Club. This is an excellent test case because it has hundreds of pages, implemented by dozens of mostly-amateur web developers, with both LAMP and M$ hosting.
This web-based service found 55 invalid links, reporting mostly 403's and 404's, and including a smattering of false alarms.
When I clicked on the 403's, I got a "Directory Listing Denied" error, as if my page was linked to a directory (folder) instead of a page. None of the other checkers had this problem, so I consider it to be a malfunction on the part of the checker. More importantly, as the report did not tell me what page contained the bad link, I could not look to see if any such links actually exist.
This is a "lite" version of a paid web service; the paid version might offer a better report.
This one is a subscription service. It's free only for small sites (1000 links) but I decided to give it a try. As my Sierra Club site was too large, I tried it out first on a much smaller personal site. (I'm planning to try it out on the large site, but by all rights, they should simply scold me for trying to cheat, and send me to bed without my dinner.)
After some difficulties with their enrollment mechanism, which their tech support team was quite aggressive in fixing, I got my first report. Actually, they offer an extensive array of reports. You could get a report on all links, on all external links, on broken internal links, or on broken external links. What I really wanted was a combined report on all broken links, internal or external, sorted by source page and listing both source and destination pages. They came close enough that I won't press that quibble too far.
Given the size limitations, I can't tell you how well they did regarding false alarms. At my small site, it found exactly what it should have found -- no more and no less. But that's a very poor test case. They offered to give me their unlimited service for a month so I could run my stress test, but I didn't want to cross the line into pay services. If I did, I'd really have to review all the other pay sites. I'm sure somebody like PC Magazine has already done a good job of that.
If your site has less than 1000 unique links, LinkTiger is certainly worth a try.
This web-based checker did an excellent job of checking my home page ONLY. Now, if I run it a few hundred more times, I'll be in great shape. (In fairness, Dr. Watson is not primarily a link checker.)
Xenu is a Windows utility that runs on your desktop. It reported over 1200 bad links, after scanning over 37,000. The report included a good number of false alarms. According to their FAQ, "Some websites are programmed only for Netscape and Internet Explorer, and refuse everything else. Some may even specifically refuse Xenu because of past misuse." As most of my site is hosted on one server, I don't believe this is explains more than a small fraction of my false alarms.
I've never quite gotten Xenu to do the right thing regarding external sites. It has a "check external links" option, which seems to mean Xenu will ignore the fact that it's left my site and continue checking until maximum depth is reached. I just set the traversal depth to 99, then curse and grumble a bit.
Xenu offers one basic mega-report, listing links by source and by destination, with separate sections for internal and external links. A nice feature of their reporting is the fact that you can include or exclude any of the above, simply by setting options before you create the report. It can also generate a site map, a list of orphan files, and it has a couple of other options I that I didn't explore. (Does anybody know what "Gopher" is, anymore?)
For a while, Xenu would hang for hours when I asked it to generate a report; then I saw a warning in the FAQ. If your site (including any external links you allow Xenu to scan) is anywhere near as big as my test case, don't ask for a site map in your report. Apparently mapping large sites is a major drain for Xenu. If you want a site map, run the scan without external links, setting your depth to something modest, run the scan, verify that you have less than a zillion links, and then print a separate site map report.
This is a very polite service. I say that because it "respects" robot exclusion rules. And that's why it didn't traverse a large number of my navigation links. (This from the folks who define all those web standards we know and love.)
Giving credit where it's due, it did find five bad links.
I've been using Xenu for a long time, and it is far from perfect. But it has always seemed like the best option for my needs, and nothing in this recent survey has seriously challenged this perception. On the other hand, if your site is small enough, you may find Linkiger to be a better fit.
(I am not affiliated with, and have no financial interest in, any of the products discussed herein.)
I only evaluated free services. If you are interested in checking out the pay ones, some do offer free trials. Check these out: