|
The Major Search
Engines
[Home]
Why are the services below considered to be the major search engines?
They are all either well known or well used.
For webmasters, these services are the most important places to be
listed, because they can potentially generate so much traffic.
For searchers, these well-known, commercially-backed search engines
generally mean more dependable results. These search engines are more
likely to be well maintained and upgraded when necessary, to keep pace
with the growing web.
Not all of the services below are "true" search engines
that crawl the web. For instance, Yahoo and the Open Directory both are
"directories" that depend on humans to compile their listings.
In fact, most of the services below offer both search engine and
directory information, though they will predominately feature one type
of results over the other. See the How Search Engines Work page
to understand more about the difference between search engines and
directories. This information was gathered at www.SearchEngineWatch.com.
Visit this site for more information and lots more search engines,
metacrawlers, etc.
q Ask Jeeves
www.ask.com
Ask Jeeves is a
human-powered search service that aims to direct you to the exact
page that answers your question.
q Excite www.excite.com
Excite offers a medium-sized crawler-based web page index, as well
as access to human-powered directory results from LookSmart. Excite
was launched in late 1995. It grew quickly in prominence and
consumed two of its competitors, Magellan in July 1996, and
WebCrawler in November 1996. Magellan was discontinued in April
2001. WebCrawler continues to operate as a separate service, but it
provides the same results at the Excite.com site itself.
q FAST Search
www.alltheweb.com
FAST Search consistently has one of the
largest indexes of the web. FAST also offers large multimedia and
mobile/wireless web indexes, available from its site. The site, also
known as AllTheWeb.com, is a showcase for FAST's search
technologies. FAST's results are provided to numerous portals,
including those run by Terra Lycos. FAST Search launched in May
1999.
q Google www.google.com
Google is a top choice for web searchers. It offers the largest
collection of web pages of any crawler-based search engine. Google
makes heavy use of link analysis as a primary way to rank these
pages. This can be especially helpful in finding good sites in
response to general searches such as "cars" and
"travel," because users across the web have in essence
voted for good sites by linking to them. The system works so well
that Google has gained widespread praise for its high relevancy.
Google provides web page search results to a variety of partners,
including Yahoo and Netscape Search (see below). Google also
provides the ability to search for images, through Usenet
discussions and its own version of the Open Directory (see
below).
q HotBot www.hotbot.com
HotBot is a favorite among researchers due
to its many power-searching features. In most cases, HotBot's first
page of results comes from the Direct Hit service (see above), and
then secondary results come from the Inktomi search engine, which is
also used by other services. It gets its directory information from
the Open Directory project (see below).
q Lycos www.lycos.com
Lycos started out as a search engine,
depending on listings that came from spidering the web. In April
1999, it shifted to a directory model similar to Yahoo. Its main
listings come from the Open Directory project, and then secondary
results come from the FAST Search engine. Some Direct Hit results
are also used. In October 1998, Lycos acquired the competing HotBot
search service, which continues to be run separately.
q MSN www.msn.com
Microsoft's MSN Search service is a
LookSmart-powered directory of web sites, with secondary results
that come from Inktomi. RealNames and Direct Hit data is also made
available.
q Iwon
home.iwon.com Backed by US television
network CBS, iWon has a directory of web sites generated
automatically by Inktomi, which also provides its more traditional
crawler-based results. iWon gives away daily, weekly and monthly
prizes in a marketing model unique among the major services. It
launched in Fall 1999.
q Yahoo www.yahoo.com
Yahoo is the web's most popular search service and has a
well-deserved reputation for helping people find information easily.
The secret to Yahoo's success is human beings. It is the largest
human-compiled guide to the web, employing about 150 editors in an
effort to categorize the web. Yahoo has well over 1 million sites
listed. Yahoo also supplements its results with those from Google
(beginning in July 2000, when Google takes over from Inktomi). If a
search fails to find a match within Yahoo's own listings, then
matches from Google are displayed. Google matches also appear after
all Yahoo matches have first been shown. Yahoo is the oldest major
web site directory.
q Alta Vista
www.altavista.com
AltaVista is one of
the oldest crawler-based search engines on the web. It has a large
index of web pages and a wide range of power searching commands. It
also offers news search, shopping search and multimedia search.
q LookSmart
www.looksmart.com LookSmart
is a human-compiled directory of web sites. In addition to being a
stand-alone service, LookSmart provides directory results to MSN
Search, Excite and many other partners. Inktomi provides LookSmart
with search results when a search fails to find a match from among
LookSmart's reviews.
q Netscape Search
search.netscape.com
Netscape Search's results come primarily
from the Open Directory and Netscape's own "Smart
Browsing" database, which does an excellent job of listing
"official" web sites. Secondary results come from Google.
At the Netscape Netcenter portal site, other search engines are also
featured.
q Open Directory www.dmoz.org
The Open Directory uses volunteer editors to catalog
the web. Formerly known as NewHoo, it was launched in June 1998. It
was acquired by Netscape in November 1998, and the company pledged
that anyone would be able to use information from the directory
through an open license arrangement. Netscape itself was the first
licensee. Netscape-owner AOL also uses Open Directory information,
as does Google and Lycos
q About.com www.about.com
About.com, formerly the Mining Company, features
hundreds of "guides" offering original content in various
areas. While About.com isn't really a search service, the guides do
have extensive links to other sites -- not to mention top-notch
content of their own.
q Britannica.com www.britannica.com
Links to top web sites and content from the Encyclopedia Britannica,
in one place.
q WebCrawler www.webcrawler.com
WebCrawler is operated by Excite@Home and offers the
same results as can be found at Excite@Home's flagship site,
Excite.com. WebCrawler was originally a completely independent
service, opened to the public on April 20, 1994. It was started as a
research project at the University of Washington.
Metacrawlers
Unlike search engines, metacrawlers don't crawl the web themselves to
build listings. Instead, they allow searches to be sent to several
search engines all at once. The results are then blended together onto
one page. Below are some of the major metacrawlers.
q Ixquick
www.ixquick.com
Meta search engine that ranks results based on
the number of "top 10" rankings a site receives from the
various search engines.
q Infonetware
RealTerm Search www.infonetware.com
This site is primarily designed to demonstrate
classification technology from Infogistics. It's a meta search
engine, and it does topical classification of results, like Vivisimo.
However, it is unique in that you can select several different
topics, then "drill down" to see results from all of them,
rather than being restricted to the results from only one
topic.
q Vivisimo
www.vivisimo.com Enter a search term,
and Vivismo will not only pull back matching responses from major
search engines but also automatically organize the pages into
categories. Slick and easy to use.
q QbSearch
www.qbsearch.com
Want to get multiple
pages of results from a search engine combined into one single page?
QB-Search will quickly join up to 200 pages of listings from major
search engines. It's a simple, helpful tool that you'll no doubt
need at some point. Another powerful offering is QB-Masterpage,
which will combine URLs that you select into a single page for quick
browsing. Both are available from QuickBrowse.
q SurfWax
www.surfwax.com
Searches against major engines or provides
those who open free accounts the ability to chose from a list of
hundreds. Using the "SiteSnaps" feature, you can preview
any page in the results and see where your terms appear in the
document. Allows results or documents to be saved for future use.
q ProFusion
www.profusion.com
Brings back listings from several major search
engines as well as "Invisible Web" resources. Formerly
based at the University of Kansas, the site was purchased by search
company Intelliseek in
April 2000.
q Query
Server www.queryserver.com
Search against major web-wide search engines,
as well as major news, health, money and government search services.
q Kartoo
www.kartoo.com
If you like the idea of seeing your web
results visually, this meta search site shows the results with sites
being interconnected by keywords.
Popular Choices
Many of the meta search engines below are popular
with users, but they aren't listed as top choices by Search Engine
Watch. This is because they typically mix paid listings within
editorial results, without any type of disclaimers. The article
below covers this issue more. It also explains how you can control
the ads you see and turn the metacrawlers listed into better
resources, by making use of customization options.
Meta Search Or Meta Ads?
The Search Engine Report, June 4, 2001
A review of meta search services by Search Engine Watch shows
that some are providing results where more than half of their
listings are paid links. A guide to what's paid, what's not and how
to get the most from your meta search service. q Mamma
www.mamma.com Sends search requests to
major search services.
q Dogpile
www.dogpile.com Popular metasearch site
that sends a search to a customizable list of search engines,
directories and specialty search sites, then displays results from
each search engine individually.
q MetaCrawler www.metacrawler.com
One of the oldest meta search services,
MetaCrawler began in July 1995 at the University of Washington.
MetaCrawler was purchased by Go2Net, an online content provider, in
Feb. 97. The commercial backing has helped improve the
responsiveness of the service. MetaCrawler now also powers searches
at the Go2Net portal site.
q Search.com
www.search.com Search.com is a meta
search engine operated by Cnet. It offers both web-wide search and a
wide variety of specialty search options. Search.com uses technology
from SavvySearch, which was acquired by Cnet in October 1999. The
SavvySearch site itself no longer operates.
How Search Engines Work
By Danny Sullivan, Editor, SearchEngineWatch.com
Updated: June 26, 2001
The term "search engine" is often used generically to
describe both crawler-based search engines and human-powered
directories. These two types of search engines gather their listings in
radically different ways.
Crawler-Based Search Engines
Crawler-based search engines, such as HotBot, create their listings
automatically. They "crawl" or "spider" the web,
then people search through what they have found.
If you change your web pages, crawler-based search engines eventually
find these changes, and that can affect how you are listed. Page titles,
body copy and other elements all play a role.
Human-Powered Directories
A human-powered directory, such as Yahoo, depends on humans for its
listings. You submit a short description to the directory for your
entire site, or editors write one for sites they review. A search looks
for matches only in the descriptions submitted.
Changing your web pages has no effect on your listing. Things that
are useful for improving a listing with a search engine have nothing to
do with improving a listing in a directory. The only exception is that a
good site, with good content, might be more likely to get reviewed for
free than a poor site.
"Hybrid Search Engines" Or Mixed Results
In the web's early days, it used to be that a search engine either
presented crawler-based results or human-powered listings. Today, it
extremely common for both types of results to be presented. Usually, a
hybrid search engine will favor one type of listings over another. For
example, Yahoo is more likely to present human-powered listings.
However, it does also present crawler-based results (as provided by
Google), especially for more obscure queries.
The Parts Of A Crawler-Based Search Engine
Crawler-based search engines have three major elements. First is the
spider, also called the crawler. The spider visits a web page, reads it,
and then follows links to other pages within the site. This is what it
means when someone refers to a site being "spidered" or
"crawled." The spider returns to the site on a regular basis,
such as every month or two, to look for changes.
Everything the spider finds goes into the second part of the search
engine, the index. The index, sometimes called the catalog, is like a
giant book containing a copy of every web page that the spider finds. If
a web page changes, then this book is updated new information.
Sometimes it can take a while for new pages or changes that the
spider finds to be added to the index. Thus, a web page may have been
"spidered" but not yet "indexed." Until it is
indexed -- added to the index -- it is not available to those searching
with the search engine.
Search engine software is the third part of a search engine. This is
the program that sifts through the millions of pages recorded in the
index to find matches to a search and rank them in order of what it
believes is most relevant
Major Search Engines: The Same, But Different
All crawler-based search engines have the basic parts described
above, but there are differences in how these parts are tuned. That is
why the same search on different search engines often produces different
results
[Back To Top] |
