B Y  W E R N E R    S C H W E I B E N Z

Searching the Web is a problem

There are several reasons for the problems in information retrieval on the Web. One reason is the rapidly growing size of the Web. The Princeton Report indicates that in late 1997 the best search engine covered only 34% of the Web (Lawrence/Giles 1998, 98) and the Digital Systems Research Center found that the best search engine covered 62% of the Web (Bharat/Broder 1998, 380). Another reason is the way search engines work. Web searching is keyword searching on some kind of index extracted from Web documents (Sherman 1999, 55). Therefore, in order to promote a Web page, it is important to place important information on a Web page in the places where search engines look for it. This article contains ten recommendations for proactive Web design that include several measures that you can take to increase the retrievability of your Web page either before or after publishing a page on the Web.

Ten recommendations for promoting your Web page:

1. Submit your Web page to several search engines.
2. Link your Web page with related Web pages.
3. Provide indexable information for search engines.
4. Use a flat structure for your Web site.
5. Use a meaningful Uniform Resource Locator.
6. Use an informative title for your Web page.
7. Repeat important title words, especially in headings and the first sections of the page.
8. Do not use tricks to try to achieve a higher ranking.
9. Use Metadata.
10. Test the retrievability of your Web page.

If you consider the size and the growth of the Web, it is obvious that you cannot wait for your Web page to be picked up by chance. It is important to let search engines know that your page is there. So the most important step is to register your page with all search engines you think important. Almost every search engine offers an option to submit a Web page. The submission is mostly done with submission forms that are linked to from the search engine’s home page. Moreover it is important to re-submit your Web page after each change because changes may influence the indexing and ranking by the search engines.

Search engines as well as human searchers traverse the Web by following hyperlinks. Therefore linking your Web page with those of other Web authors makes it more likely that your page is found both by search services and Web users. So ask the owners of related pages to link to your page and offer a link in return. In this way your page becomes part of a network of links which makes it more likely to be retrieved.

Apart from being found, being linked to other pages has another advantage. A link to another Web page is almost like a citation in a book. It shows that the Web author thinks that this page is so important that he links to it. Search engines plan to make use of this kind of peer review. New search engine technology tries to locate authoritative sources on the Web, measures their link importance and uses it in relevance ranking (Sherman 1999, 57).

Search engines are text-based search tools (Sherman 1999, 55) and therefore only textual information that exists in static, hard-coded HTML files can be indexed properly. Multimedia content, frames and dynamic pages cause trouble for search engines.

Although the Web allows the use of different media (that is, text, images, audio, and video), search engines can only index text, even though some search engine provider state that their engines can index multimedia content. The solution is to add textual information to multimedia content. For example, using the HTML tag for an image with additional textual information: <IMG SCR= "URL or file name of the image file" ALT="textual description of the image">.

Using frames in your web pages can be problematic as far as printing, bookmarking and search engines are concerned because no single source exists that describes the particular combination of frames the browser shows in a certain moment (Sonnenreich/Macinta 1998, 64). Therefore a lot of search engines refuse to index pages with frames. So if you use frames, make sure to provide information on your frame pages in metadata or in static pages that do not contain frame information.

Dynamic pages are virtual spaces that do not exist as HTML files, (that is, image maps, query-based forms as front-ends to databases and dynamic web pages generated "on the fly" by server-side applications). Dynamic pages are ignored by most search engines (Sullivan 1999a, 35) because no static, hard-coded information exists that they can index. Again the solution is to provide metadata and static pages that contain information about the content of those virtual spaces on your web pages.

Most search engines use a strategy that is called the breadth-first approach; they try to cover a wide range of links to a large number of different Web pages instead of going deeper into the structure of single Web site. The findings of Tunender and Ervin (1998, 177) indicate that it takes a considerable amount of time (at least more than 46 days) before most search engines crawl deeper into the hierarchical structure of a Web site. Therefore it is better to use a flat structure for a Web site instead of a deep structure because Web pages within a flat structure are found earlier and more easily than pages within a deep structure.

Often the address of a Web page, the Uniform Resource Locator (URL), is a complex string of characters and numbers. Although the URL is a machine-level addressing system, research (Maze, Moxley/Smith 1997, 24) indicates that search engines index the words in the URL and that the words in the URL have some influence on the ranking by search engines. Therefore it is important to use a meaningful URL for your Web page.

Words within the <TITLE> tag of an HTML document are not only displayed in the title of a Web page in the browser, but they also appear as the name of the page in electronic bookmarks. Many search engines display the title in the results pages they present to the user (Maze/Moxley/Smith 1997, 26). Apart from that, the title is important for retrieval by search engines (Tunender/Ervin 1998, 178). Therefore, provide an informative title for your Web page as well as a good description of your Web page’s content.

Writing for the Web is in some aspects different from writing for a publication in print. This is due to the electronic environment in which the Web page is presented. If you keep retrieval techniques in mind while writing for the Web, consider using the complete name of a noun more often than you would in normal writing, rather than using pronouns and other indirect ways of referring to subjects. This is necessary because for statistical analysis it is relevant how often a word occurs in the document. The statistical analysis is important because of the way search engines weigh terms in documents. Search engines do not only look for words but also for the context in which the words are used (Maze/Moxley/Smith 1997, 26). Therefore it is important to create a context for the weighing and ranking of terms because weighted word indexing seems to be more prevalent than thought (Tunender/Ervin 1998, 178). Therefore, repeat important words from the title in headings and the first sections of your Web page to increase retrievability.

It is not a good idea to use search engine persuasion to achieve a higher ranking in search engines. Search engine persuasion consists of various activities that are aimed at achieving high rankings in the search engine results pages. High rankings are desirable especially for companies who advertise products on the Web and want to achieve maximum exposure to potential clients. Most of these search engine persuasion activities are underhanded techniques; some are devious, if not illegal. They range from attempting to deceive the ranking algorithms of search engines by keyword spamming (one or more keywords are repeated over and over again in order to achieve a high ranking by search engines), to illegally using trademarks and well-known company names. These tricks do not work very well any more because most search engines use countermeasures and some even penalize pages that use keyword spamming or other tricks (Laursen 1998, 44). Moreover your Web page does not appear in a favorable light with users if you use tricks like search engine persuasion.

Metadata is data about data or objects. Catalog cards in a library catalog is an example of how metadata can be used. On the Web, metadata is used as special META tags in HTML where the web author can, for example, give a description of the page with the META tag description or name keywords for indexing by search engines with the META tag keywords. Many search engines use the META tag description to extract the summary of the Web page that is shown in the results list and use keywords from the META tag keywords for indexing (Laursen 1998, 44). Search Engine Watch's Search Engine Features for Webmasters (Sullivan 1999b) gives a survey of the search engines that harvest and index metadata (cf. section Crawling/Meta robots tag). Some search engines give a boost in ranking to Web pages that contain metadata (cf. section Ranking in Search Engine Watch's Search Engine Features for Webmasters). Others ignore metadata completely or partly. Overall, Metadata can enhance the accessibility of your Web page’s multimedia content as well as your web page’s dynamic, framed and static information.

Most search engines offer the possibility to check if your Web site is registered with them. Usually you enter your Web site’s Uniform Resource Locator (URL) in a form that checks it against the index of the database. As usual, all search engines offer this service in a different way (cf. Checking Your URL in Search Engine Watch). In addition to search engines, there are commercial services that offer to check search engine databases for URLs, keywords and titles. These URL Checking Services usually offer not only URL checking but also offer to submit your Web site to dozens or hundreds of search engines. One of these services is Rank This! (Internet, URL http://www.rankthis.com/). It allows you to search for your Web page for free by using keywords or the title of your Web page eight different search tools, such as AltaVista, Excite, HotBot, Infoseek and Yahoo. Moreover it shows how your page is ranked for the keywords or title in relation to other Web pages, for example, among the Top Ten or among the first two hundred hits. Another free service for checking metadata is NorthernWeb’s tool Meta Medic that checks only the META tags description and keywords of a Web page and evaluates the use of metadata (Internet, URL http://www.northernwebs.com/set/setsimjr.html). Free services like these allow you to adapt your proactive Web design measures if necessary.

These recommendations for proactive Web design provide ways to promote your Web pages by using general design issues. However, they cannot guarantee that your Web page will be placed among the Top Ten results of a certain search engine because there is too little known about the way individual search engines work. Nevertheless research shows that these recommendations will work reasonably well for the majority of search engines because they are based on the findings of various experiments.

What is important for you as a Web author or designer to do is to include these recommendations in an early stage of the design process and keep retrieval techniques in mind while designing your Web pages. Information design is more than what is seen in the Web browser.

Laursen, Jesper Vissing (1998): Somebody Wants To Get In Touch With You: Search Engine Persuasion. In: Database 21 (1) Feb/Mar 1998, 42-44, 46.

Lawrence, Steve/Giles, C. Lee (1998): Searching the World Wide Web. In: Science, April 3, 1998, Vol. 280, 98-100.

Maze, Susan/Moxley, David/Smith Donna J. (1997): Neal-Schuman Authoritative Guide to Web Search Engines. New York/London: Neal-Schuman Publishers.

Sherman, Chris (1999): The Future of Web Search. In: Online, May/June 1999. Special Issue Web Search Engines. 23 (3) 1999, 54-61.

Sonnenreich, Wes/Macinta, Tim (1998): Web Developer.com Guide to Search Engines. New York, NY: John Wiley & Sons.

Sullivan, Danny (1999a): Crawling Under The Hood. An Update on Search Engine Technology. In: Online, May/June 1999. Special Issue Web Search Engines. 23 (3) 1999, 30-38.

Sullivan, Danny (1999b): Search Engine Watch. Internet: URL http://searchenginewatch.internet.com/webmasters/features.html. Version: 01/25/99. Visited: 08/20/99.

Tunender, Heather/Ervin, Jane (1998): How to Succeed in Promoting Your Web Site: The Impact of Search Engine Registration on Retrieval of a World Wide Web Site. In: Information Technology and Libraries, 17 (3) September 1998, 173-179.

Werner Schweibenz is a member of the Department of Information Science, University of Saarland, Germany. You can reach him at w.schweibenz@rz.uni-sb.de. This article is a condensed version of a presentation given at the Second International Workshop: Exploring a Communication Model for Web Design, Seattle, WA, July 10-17, 1999. The full paper, "How To Use General Design Issues and Metadata In Order To Get Your Web Page Picked Up By Search Engines" is available at
http://www.phil.uni-sb.de/fr/infowiss/projekte/webdesign.html.

Fall 1999
Volume 2, # 4