Webspin Graphics and Web Design - Newsletter Sept 03, What is a search Engine

September 1, 2003
Volume 1, Issue 9

Welcome to the Webspin Newsletter for September 1, 2003

What is a Search Engine? - Rick Sweeney

Search engines help people find relevant information on the Internet. Major search engines maintain huge databases of web sites that users can search by typing in some text.

To compile their databases, search engines rely on computer programs called "robots" or, more specifically, "spiders." These programs "crawl" across the web by following links from site to site and indexing each site they visit. Each search engine uses its own set of criteria to decide what to include in its database. For example, some search engines index each page in a web site, while others index only the main page.

On Saturday Afternoon.

You have this overwhelming craving for an award winning texas hot chili. You need the recipe fast. Only a few short years ago, the quest for such an obscure piece of information would probably entail a visit to the library. It certainly wasn't a project you could do in your underwear on Saturday afternoon. Today, a savvy searcher can access information on a gazillion possible topics through two tiny copper wires (or a cable) coming into her home. The Internet has made unprecedented volumes of information freely available to the connected masses, and that's great. OK, then... so how do you find a good recipe for an award winning texas hot chili, given that it could be in any of about a billion documents housed on one of hundreds of thousands of computers all over the world? For that, you need a search engine.

Spiders and People

Search engines send automated information-gathering computer programs called "spiders" all across the web to seek out available content. Each engine maintains an index of all its information, allowing you to type in search queries that hopefully will point you to appropriate web sites. Directories such as Yahoo and ODP use human editors to compile their indexes, which are cleaner but more limited.

"Hey, Do You Google"?

Not long ago, sophisticated computer searching was mostly the domain of librarians, researchers and those who could afford expensive proprietary services. Today, go to any cocktail party and you're likely to hear Missy chirping about her search engine fave.

It has also become fashionable for seasoned Net users and newbies alike to complain about the difficulties of searching the web. This is understandable. Any one search engine provides a limited slice of the whole web pie. Anyone can publish information on the Web, so what's out there may be false or biased. Too often, the top results for a given search query are more commercial than informative.

So....a search engine is a database of resources extracted from the Internet through an automated "crawling" process. This database is searchable through user queries using a search engine such as www.google.com, www.tahoo.com, www.altavista.com...ect

How does a search engine work?

Words or phrases you enter in the search box are matched to resources in the search engine's database that contain your terms. These are then automatically sorted by their probable relevance and presented with the most "relevant" sites appearing first.

How search results are organized

Once a search engine has used your search terms (keywords) to gather "hits" from its database, it lists or "ranks" the resulting sites in order of its own estimation of their relevance. The procedures and factors used to create this ranking are often company secrets, so understanding exactly why one hit is listed higher than another is difficult.

The following is a survey of some of the factors search engines use to automatically sort web sites for presentation to the user.

Relevance Prediction

Currently, search engines predict relevance based on two sets of factors: those based on a site's content and those external to the site.

Factors based on a web site's content

Keyword Word frequency (How often search terms occur in a page in relationship to other text)
Location of search terms in the document (Are they in the title? Are they near the top of the page?)
Relational clustering (How many pages in the site contain the search terms?)
The site's design (Does it use frames? How fast does it load?)

Factors external to the site

Link popularity -- Sites with more links pointing to them are prioritized
Click popularity -- Sites visited more often are prioritized
Sector" popularity -- Sites visited by certain demographic or social groups are prioritized (Note: This system requires user-provided information)
Business alliances among services -- Results from a partner search service are ranked higher
Pay-for-placement rankings --
Site owners pay for high rankings

Ways you can search

The Internet is the most rapidly growing collection of resources in human history. Fortunately, most search engines now offer ways to narrow your searches to produce fewer and more focused results. Learning how to use advanced search options effectively can save you hours of frustration. While many search engines offer some or all of the following enhancements, they are not all activated or expressed in the same way.

Searching for a phrase

Enclose any phrase of 2 words or more within quotation marks to limit the search to that exact sequence of words.

Examples:
"food and drug administration" "New York Times"

If the search engine you're using doesn't specify phrase searching, it will usually provide sites that contain your search terms anywhere and in any order.

Searching by word stem (truncation)

Some engines allow you to search for all variants of a word by entering the word stem followed by a truncation indicator.

Example:
"fem*" retrieves female, females, feminine, feminist, feminists, feminism

Using truncation may greatly increase the number of results to your searches. Only use it when it's particularly necessary.

Using Boolean expressions

Boolean expressions allow you to require, combine and exclude words or phrases in your results. Using them well can make your searches much more focused and successful.

To REQUIRE a term to be present in your results, put a plus (+) immediately before it or use the Boolean operator "and."

Examples:
+Scotland +golf
Scotland AND golf

To search for documents containing ANY of your search terms, you can either simply list them or put the Boolean operator "or" between each one.

Examples:
jazz swing
jazz OR swing

To EXCLUDE a term from your search results, put a minus immediately before it or use the Boolean operator "not."

Examples:
sharks -card
sharks NOT card

What is Search Engine Optimization?...Click Here for article two in this issue

Access your newsletter Archives...Click here
Use your first name and e-mail address to logon

contact...

Webspin offers a free initial consultation.

e-mail:
Webspin Graphics and Web Design

<% rsWebspinLogon.Close() Set rsWebspinLogon = Nothing %>