By Lee Cummings, June 2001

The World Wide Web has evolved from a library of static web pages to an increasingly database-driven and dynamic place. Ecommerce sites are springing up everywhere, and even the most primitive of these is powered by some sort of database mechanism. Unfortunately, in the haste to make a working shopping system, the consideration of the search-engines is often set aside, and the Web is awash in millions of pages which are completely invisible to the search-engines.

One reason for this is that search-engines tend not to follow links with question-marks (?) in them, as that clearly designates the page to be a script, as opposed to a static HTML page. But why should the server be so worried about indexing a script? There are several good reasons, and one might conclude that the search engines are justified in their policies:

1. A script implies that the output is in a state of change; if it was always the same, why use a script? Since pages remain in the index for months if not years before being re-indexed, it may not make sense to index very timely and changing information at all.
2. Some scripts map a space which is infinite, or extremely large. For example, consider a web script which plays a game of chess. Efforts to map a space of this size would consume massive resources, and ultimately fail.
3. Disreputable practices that some web programmers use against the search engines, such as cloaking, are more easily facilitated when scripts are indexed.

The consequences of this policy are simple to understand: database driven pages are almost entirely absent in the search engines, and that includes the product pages of over 90% of all ecommerce sites. If you are involved in an ecommerce venture, this news may be particularly disheartening. Go to your site and observe the URLs of the product pages. Product pages are the most important pages on your site for useful search engine traffic. If your URLs contain a "?", or the phrase "/cgi-bin/," or simply the telltale ".asp" extension, you know that these pages are not fully accessable to the search-engines.

What is to be done? One popular remedy is to use cloaking, the disreputable technique mentioned above. In short, the idea is to create a "dummy site" that contains normal HTML pages that mirror the real site. When the search engines come, it sees only the static HTML pages, which it indexes perfectly. But when real visitors come, they get quickly whisked away to the corresponding page on the real server.

There are three problems with cloaking for ecommerce sites. First, it requires a large amount of maintenance, which would be unneccesary if the site was constructed in a search-engine friendly manner to begin with. Second, the search engines are strongly opposed to this practice, as it interferes with their methods of rating web sites. In extreme cases, this can trigger a retaliation, such as banning your site from their service. Thirdly, search engines like Google are becoming very sophisticated in their ability to rate a site based on incoming links (sites that link to you.) The "dummy site" that is required for cloaking will suffer when evaluated by such criteria; therefore, you may find that it's hard to drive significant traffic to the site using such methods.

The best method is to simply take search-engines into account from the very beginning. This relates largely to the subject of compatibility. Just as we consider WebTV users are real, legitimate users of our services, and make accommodations for them, we must view the search engines as a kind of user with a somewhat limited browser. The search engines really want to index our pages, it's the designers of ecommerce systems who have created the obstacles. This doesn't mean abandoning scripts and databases, just to make sure that the crucial product pages appear to the search-engine to be a normal HTML page in every respect.

One online company who got their technology correct from the beginning is Amazon. Their system is clearly database driven, and yet the pages appear to be normal HTML pages, and do not contain the telltale question-mark. In combination with their excellent affiliate system, affiliates were able to insert their Amazon pages into the search engines, providing the site with a constant stream of highly targeted traffic.

When 20/20 Technologies set out to design an ecommerce system, the search-engine compatibility issue was paramount. By studying Amazon and other successful ecommerce systems, we perfected a method of presenting normal HTML pages to the engines which are, behind the scenes, dynamic database driven pages. The final product, dubbed Cart Nouveau, implements the best ideas we learned from Amazon and others. Skeptical at first, our doubts vanished when we saw one of our clients secure the #1 search result in Google for two highly competitive terms.

If you are involved in an ecommerce venture, don't make the same mistakes that thousands have made before you. Recognize that the ecommerce platform and search-engine promotions are inseparable. If you are interested in Cart Nouveau, or if you have any questions, please contact us today for a complete evaluation of your company's online promotion needs.