Mining the Deep Web: Search strategies that work

Arguably the most valuable Deep Web resources are searchable databases. There are thousands of high-quality, authoritative online specialty databases. These resources are extremely useful for a focused search.

Many Web sites act as front ends to searchable databases. Complete Planet, IncyWincy Spider and The Librarians' Internet Index provide quick links for quality Web database searching. This technique is called split-level searching. Enter the key phrase "searchable database" into the above for more.

You can find other subject searchable databases by entering the keyword phrase

"subject_name database" into your favorite search engine (e.g., "jazz database," "virus database").

A naive searcher typically enters a keyword into a general-purpose search engine, gets too many hits and then expends time and energy sorting through relevant and irrelevant results. Alternatively, they get no hits and wonder why. It is difficult to get all relevant hits and no irrelevant hits. (Information scientists call this the Law of Recall and Precision.)

Almost by definition, authoritative searchable specialty databases contain relevant information and minimal irrelevant information.

Don't forget to bookmark a variety of special topic searchable databases into a Deep Web folder for ready reference.

Deep Web Search Strategies

-- Be aware that the Deep Web exists.

-- Use a general search engine for broad topic searching.

-- Use a searchable database for focused searches.

-- Register on special sites and use their archives.

-- Call the reference desk at a local college if you need a proprietary Web site. Many college libraries subscribe to these services and provide free on-site searching (and a friendly trained librarian to help you).

-- Check the Web site of your local public library. Many libraries offer free remote online access to commercial and research databases for anyone with a library card.

Summary

The Deep Web contains valuable resources not easily accessible by automated search engines but readily available to enlightened searchers.

Make the online search process more efficient and productive with resources missed in the Shallow Web. The truth is out there.

Lee Ratzan is a system analyst at a health care agency in New Jersey and teaches library technology at Rutgers University. Contact him at lratzan@scils.rutgers.edu.