What is a search engine?

One way to begin answering the question what is a search engine? is to find a way to categorise search engines.

The highest category used by this book is mega search engines, which is a catch all term for the best search engines out there. These can be divided into two types: purse search engines and meta search engines.

Most pure search engines can be placed into two categories depending on the kind of database they use to index the content over which searches are performed. These are:

• Spider Database. When you submit a web page to a spider database a reference to the page is placed in a queue. A spider visits the page at regular intervals and provides it with an index entry. Examples of spider databases include Google, Alta Vista, Hotbot, and Alltheweb. Recently, Yahoo have introduced a spider database which is now the main competition for Google.
• Directory. A search engine can also use a directory, which is a database of hand picked entries. Yahoo and Looksmart have been running two of  the most popular 'pay to enter' directories for some time. The Open Directory of the Open Directory Project is the most popular directory where entries are free.

Meta-search engines are usually categorised as search engines. But they are 'above' doing the searches themselves. Meta-search engines, otherwise known as portals, obtain results from several pure search engines in an attempt to provide a more comprehensive service.

If you are a website owner, be careful if someone says they can get you listed in thousands of search engines at a 'bargain price'. They actually mean that, at best, they will  get you listed in dozens of true data sources, which will then be accessed by thousands of portals. It would not take long to do this yourself.

Meta-search engine firms also provide data to other service providers. For instance, Metacrawler and Dogpile provide data to Wordtracker, which provides incredibly useful data on the frequency of keywords and key phrases that people are searching for.