AboutSitemaps and How the Rich Get Richer
Sitemap files are a competitive advantage for Google – an example of how the rich get richer and why power curves exist. This is true even though Sitemap files are a public standard open to every website and endorsed by Yahoo and Microsoft as well as Google. This is not a complaint about Google; in this case they've done everything right. It's just an example of why dominance is self-reinforcing – for a while.
Sitemap files (see previous post) are a way for webmasters to help web crawlers look at their websites more constructively so that search engines will be able to search the websites better. Sitemaps are essential for the many websites which have some kinds of dynamic rather than static content since the crawlers would never find and index the dynamic content with the sitemaps.
Webmasters can and do put their sitemaps in discoverable places on their sites. However, there also procedures for notifying search engines when your sitemap file changes so that your site will be reindexed and new content made findable through search as soon as possible. You can "submit" a sitemap file to a search engine or just tell the search engine that a previously registered sitemap has changed. Which search engine would you notify first when your sitemap changes? Google, of course. Yahoo, probably next. Microsoft if you get around to it. What about some new search engine somebody just invented? Of course not, you never heard of it. So Google ends up more current than Yahoo, indexes stuff that Microsoft may or may not see and that a startup search engine will have to discover on its own.
The leaders get a longer lead over the challengers. But, in this case, there's more.
Sometimes sitemps are submitted to some search engines without being made public. Social media sites like Facebook and Twitter want to be found by search engines. The advantage to them is that you get led to a page which encourages singing up if you're not already a member when you search for the name of a member of the social media site (Facebook allows you to opt out of being found in such searches). If you're a member of a social media site, try googling your name and the name of the site as in "Tom Evslin Facebook"; then click through the resulting link. But how did Google know where to find all these dynamic pages on Facebook members which don't appear in any sitemap file I can find? Yahoo knows where to find them, too; Microsoft Live Search knows about them but doesn't handle them well.
A developer of a new application searched in vain through Facebook's developer documentation for information on how to find and index these pages and finally wrote to developer support. It took a long time to get a reply. When it came, the essence were these sentences: "Unfortunately you will need to make a specific arrangement with Facebook in order to achieve this functionality. You may submit your proposal to [email protected] for review. Please let us know if you have any further questions." Facebook probably has a legitimate concern about how this information might be misused; but that didn't stop them from making the information available to the major search engines. Maybe they'll make it available to little guys, maybe not. Advantage incumbents.
Google gets a third advantage (totally legitimate) by innovating in the sitemap space. Incidentally, only a leader really can innovate in this space because you need all the SEO consultants to tell all the webmasters that it's important to implement the new stuff.
The standard for sitemaps assumes that, when a crawler reaches a page, it will be able to read the text on that page for indexing purposes. But what if the page is a video? What if it's a map? Does standard crawling work for code that someone wants to sell, share, or promote?
Not to worry, Google has announced five extensions to the Sitemap format :
These are useful just for submitting to Google as far as I know so , for now, leader Google can do a better job of searching for video, mobile content, news, code and geographic stuff than the followers.
Comments