The Art and Science of Sitemapping
Sitemapping is both an art and a science (isn’t most of good website design?). It’s a bit quirky, because it’s designing not for a human, but for a piece of software to easily comprehend instead.
Google has an XML-based sitemap specification that has been picked up by most of the other search engines. If you generate the site map in this format, Google can more quickly and accurately traverse our site and return more relevant results. It also gives the googlebot an idea of how frequently to re-index pages (do specific pages change often or rarely?).
To encourage Google and other engines to fully index your website, the “sitemap.php” (or “sitemap.html”, etc.) file, and especially the “sitemap.xml” file need to be at the root level of the site, and a link to the human-readable one needs to be included near the top of every page (above the top 20% of the page in code view, not just in display view) to be sure the sitemap is easily located and indexed. All 404 error pages and other error documents should include a link to the sitemap to help both people and bots locate an alternate choice if they followed a stale link.
