Sitemaps

You are here: > Text Link Guide > Sitemaps

What are sitemaps?

Sitemaps are used by webmasters to inform search engines about pages on their sites that are available for crawling. Search engine spiders usually discover webpages from links within a site and from other sites, but they sometimes don't index all pages, especially if a site has a large number of pages. Sitemaps supplement this data.

What are HTML sitemaps?

A HTML sitemap is a collection of hyperlinks, sometimes with descriptions, on one page or spread across several pages. Sitemaps help visitors to find what they're looking quickly, while at the same time helping search engine spiders crawl and index all the pages within the site.

What are XML sitemaps?

In its simplest form, a XML sitemap (see Google Sitemap Protocol) is a XML file that lists URLs for a website. It may also include additional metadata about each URL (date of last update, how often the pages are updated, and how important it is relative to other URLs in the site) to better help search engines more intelligently crawl and index the website.

Here is some sample XML sitemap code:

What is Google Sitemaps?

Google Sitemaps is a suite of webmaster tools offered by Google to provide webmasters a free and easy way to make their site more Google-friendly. Google Sitemaps shows you how Google crawls and indexes your site, and specific problems it might be having accessing it. You can also discover which search queries drive traffic to your site, and which version of Google is driving that traffic.

Which sitemap formats does Google support?

Google supports XML, RSS (Real Simple Syndication) 2.0, Atom 0.3 feeds, OAI-PMH (Open Archives Initiative Protocol for Metadata Harvesting), and a text file containing a list of URLs.

What is a sitemap generator?

You could create a sitemap by hand by following the sitemap protocol, but it would be a really tedious task, and you will probably miss a few links. A better method is to use one of the many sitemap generators available. Google offer a free sitemap generator but it is probably too complicated for most people to install. The installation instructions certainly put me off giving it a try. Luckily there are plenty of downloadable and online sitemap generator tools offered by third-parties.

Can you recommend a sitemap generator?

I've only tried a few free, downloadable sitemap generators, but here are my opinions on those I've tested:

• G-Mapper is a Google and Yahoo sitemap generator for static website's of all sizes. The main problem with G-Mapper is that it doesn't give you the option of adding filters. You have to manually delete URLs after they've been crawled which is a real pain.
  
  
• Gsitemap is another sitemap generator tool. The main problem with Gsitemap is that it only generates Google sitemaps. It doesn't generate sitemaps for Yahoo!.
  
  
• GSiteCrawler is the sitemap generator I ended up using mainly because it offered all of the basic features, including Google sitemap and Yahoo! URL text file generator. But what sold me were the extra features that made it easy to use. For example, the "Import Robots.txt" feature meant I could quickly filter out all unwanted directories and pages with one click.
  
  The other nice feature is that you can edit the metadata of one or more (individual or group) URLs at the same time, which is something not available in some of the other sitemap generators I tested. GSiteCrawler is free, although it does display a nag screen asking for a donation every so often.

The above applications are all for Windows computers only. Mac users may like to try Map-IT SiteMapper ($19.95 and also available for Windows) and RAGE Google Sitemap Automator ($29.95).

I found the above sitemap generators in Google's Sitemaps Third Party Programs & Websites guide. For a full list of the third-party sitemaps programs & websites, visit the Google page.

How do I validate my Google sitemap?

You can validate your Google XML sitemap with the Google Sitemap Validator by Validome. I'm glad I did as I discovered that the links to the Sitemaps.org XML schema in the sitemaps generated by GSiteCrawler were returning 404 not found errors. As a result I switched to the XML headers in Google's Using the Sitemap Protocol guide.

How do I submit a sitemap to Google Sitemaps?

Log in, or sign up, to Google Webmaster Tools with your Google Account, and add your sitemap URLs.

What are text sitemaps?

A text sitemap is simply a text file with a list of URLs, one on each line, and is supported by Yahoo! Site Explorer. Unlike XML sitemaps, it does not allow for metadata about each URL.

Here is some sample text sitemap code:

How do I submit a sitemap to Yahoo?

Submit your sitemap to Yahoo! Site Explorer.

Which sitemap formats does Yahoo! support?

Yahoo! supports RSS 0.9, RSS 1.0, RSS 2.0, Atom 0.3, and a text file containing a list of URLs. The filename of the URL list file must be "urllist.txt". The filename for a compressed file must be "urllist.txt.gz". Site Explorer recognizes files with a .gz extension as compressed files and will decompress them before parsing.

How do I submit a sitemap to MSN?

Good question! No one seems to know. The best bet is try submitting your sitemap URL to the Windows Live Search URL Submission form.

Do sitemaps guarantee my webpages will be included in a search engine?

No. Sitemaps do not guarantee that webpages will be included in search engines, but they should help search engine spiders do a better job of finding and indexing every page within a site.

Do sitemaps improve search engine rankings?

No. Sitemaps will not influence the way your pages are ranked by a search engine. However, it does help to get them crawled and indexed, which gives your pages a better chance of being ranked.

Does the position of a URL in a Sitemap influence its use?

No.

Do I need to remove session IDs from URLs?

Yes. Session IDs in URLs may result in incomplete and redundant crawling of your site.

Should I include the frameset URLs or the URLs of the frame contents?

Include both URLs in your sitemap.

This document was written with the help of Sitemaps.org, the official sitemaps website sponsored by Google, Yahoo! and Microsoft. Visit the site for more technical information on sitemaps.

Site Map:

• Text Link Guide - homepage
• Text Links - search engines, link traffic
• Anchor Text - variations, misspellings
• Image Links - image maps, alt text
• PageRank - increase PageRank, attribute
• Link Popularity - authority pages, link quality, quantity, relevancy, reputation
• Sitemaps - XML, Google, generators, Yahoo!
• Linkbait
• Link Seeding - link spam
• Link Exchange - bad neighborhoods, link exchange software
• Sponsored Links - PageRank
• Affiliate Links
• Link Brokers - RON, ROS, Text Link Ads
• Natural Linking - artificial linking, link triangles, www or non-www
• Broken Links - link rot, broken link testing