Internet Research

Search Engines & Subject Directories

Search engines

Search engines are the means by which most people search the Web.

Common examples are Google, Altavista, Direct Hit.

Yet they don’t search the Internet

Yet a search engine does not actually search the Web during your search.

A search engine searches itself.

It’s a three-step process.

1) Bots index words

Search engines continually send out hundreds of “robots” or “bots” (or “spiders” or “crawlers” )

Bots visit web sites, read word by word, and then index those words.

2) A database is created

A huge database of Web sites thus is gathered and indexed by word.

These databases can be huge, with millions of links.

3) The Interface gives you access

Using the keywords you give it, a search engine then searches its own current index.

Interfaces are based on rankings

Search engines return results based on a ranking system.

Ranking is the order that files are listed when they are retrieved.

The ranking system is secret

These systems are proprietary and often “secret.” In general:

Altavista ranks web pages higher if your search terms are found in the first few words of the page

Google ranks by document “popularity” with other similar searches

Direct Hit ranks by the length of time other users spent at the site

Not even half the Web

With all of this software and sophistication, even the best search engines cover only 40-50% of the Web.

And they miss much else on the Internet.

Bots hit and miss

Bots miss:

XML pages, pdf files

Dynamically created HTML pages

Frames-based pages

New pages or recent updated text

Some say the Invisible Web is 500 times larger than Web

Subject Directories

A subject directory is also a database of web sites and references.

But a subject directory is organized not by keywords but by category or subject.

Yahoo!

Yahoo! Is the most popular subject directory.

www. about.com takes the idea a step further with subject guides for selected topics.

Subjects are organized by people.

Information is selected, organized and cataloged by a person, not software.

You can usually be more assured that the search results will make sense.

You get an index of sites.

Subject directories will not often provide you with ranked web sites.

Instead, you will get a broad index related to your topic, divided further by subheadings.

Use for early searching.

Use a subject directory early in your search process to learn about your subject.

You will get fewer links of higher quality.

When you get more specific questions, you should use a search engine.


	Search engines are the means by which most people search the Web.
	Common examples are Google, Altavista, Direct Hit.


	Yet a search engine does not actually search the Web during your search.
	A search engine searches itself.
	It’s a three-step process.


	Search engines continually send out hundreds of “robots” or “bots” (or “spiders” or “crawlers” )
	Bots visit web sites, read word by word, and then index those words.


	A huge database of Web sites thus is gathered and indexed by word.
	These databases can be huge, with millions of links.


	Using the keywords you give it, a search engine then searches its own current index.


	Search engines return results based on a ranking system.
	Ranking is the order that files are listed when they are retrieved.


	These systems are proprietary and often “secret.” In general:
	Altavista ranks web pages higher if your search terms are found in the first few words of the page
	Google ranks by document “popularity” with other similar searches
	Direct Hit ranks by the length of time other users spent at the site


	With all of this software and sophistication, even the best search engines cover only 40-50% of the Web.
	And they miss much else on the Internet.


	Bots miss:
	XML pages, pdf files
	Dynamically created HTML pages
	Frames-based pages
	New pages or recent updated text
	Some say the Invisible Web is 500 times larger than Web


	A subject directory is also a database of web sites and references.
	But a subject directory is organized not by keywords but by category or subject.


	Yahoo! Is the most popular subject directory.
	www. about.com takes the idea a step further with subject guides for selected topics.


	Information is selected, organized and cataloged by a person, not software.
	You can usually be more assured that the search results will make sense.


	Subject directories will not often provide you with ranked web sites.
	Instead, you will get a broad index related to your topic, divided further by subheadings.


	Use a subject directory early in your search process to learn about your subject.
	You will get fewer links of higher quality.
	When you get more specific questions, you should use a search engine.