The Search Engine Stuff

What is a Search Engine?
A Search Engine is a tool for finding information on the Internet. A Search Engine typically uses a spider or "bot" (robot) that travels the internet collecting information about websites. The information is collected into a database, then indexed and sorted using algorithms(rules). Users search for information by typing a query (keywords)into a search box on a web interface. The rules determine the order of the website pages shown in answer to your query. Google is currently the most popular Search Engine.
How many Search Engines are there?
There are a few thousand if you include specialty search engines and directories and foregn language search engines. But, there are fewer than 20 in the USA that have enough market share to make a difference.
How do Search Engines find web pages?
Search Engine spiders or robots surf the Internet crawling from link to link adding new or changed information to the Search Engine database. The spiders surf very quickly, and the databases are very large, but they do not contain all or even half of the information on the Internet. It is still worth taking the time to submit or register websites directly to each Search Engine. It can take days or months to get indexed. Some Search Engines offer free registration, and others charge from $20-$300 a year to be in their database.
What are Search Engine Algorithms?
An algorithm is a mathematical formula that a computer uses to make a determination. Think of it as a rule. Search Engines use many rules (Google uses over 100) to determine the relevancy of a web page for each query that you make. note: Each Search Engine has a proprietary set of rules.
How do I improve my Search Engines Ranking?
Search Engines vary a bit in how they determine the most relevant websites, but there are some general guidelines.
**TIP** A few things that can improve your ranking: Properly submitting your pages, improving your page design, using clean coding, HTML meta tags, lots of other worthwhile related sites linking to your site.
What are Meta-Search Engines?
Meta-Search Engines transmit your search term to multiple search engines simultaneously. They can save time for specific types of searches. The results are only as good as the databases queried.
What are directories?
Humans collect information about websites and sort and classify the information and enter the information into a database that users can query.
What is the difference between a Search Engine and a Directory?
A Directory typically has fewer sites indexed but better relevancy in your searches due to the human intervention. A Search Engine (using robots) can collect information on many more sites, but has to rely on complex computer algorithms or rules to try and sort the data into order.

TOP SEARCH ENGINES
Ask Jeeves Search Engine Powered by Teoma. Query in plain English.
Google Search Engine Database of 8+ billion web pages, also searches images, newsgroups, excellent auto-generated news with 4,500 sources, Froogle collects information about products for sale, Catalogs to search and browse catalogs online, University Search, Language Tools, UncleSam for searching Government websites, Google Personalized , and so much more that I had to make a separate Google Stuff section.
Yahoo Search Engine (owns Inktomi)
MSN Search Engine (Powered by Yahoo/Inktomi)
MSN Search Engine (Beta MsnBOT Crawler)

SEARCH ENGINES
Alexa Web Search Powered by Google with additional information including traffic ranking. note: Traffic ranking is determined by users of the Alexa bar
AllTheWeb Powered by Fast and Yahoo Crawler, owned by Yahoo, large databse plus multimedia and news searches.
AltaVista owned by Yahoo Search Engine that searches websites and multimedia and news.
AOL Search Engine (enhanced by Google and DMOZ)
Excite
Gigablast Search Engine (up and coming)
Hotbot
LookSmart
Lycos Search Engine / Powered by Fast
Netscape Search
SearchEDU.com : Searches .edu, domain; also offers to search well-known dictionaries, encyclopedias, almanacs, etc.
SearchGOV : Searches .gov domain websites
SearchMIL : Searches .mil domain websites
WiseNut

DIRECTORIES
DMOZ (Open Directory Project add url
Gimpsy (Up and coming)add url
GoGuides (spinoff from Go Directory - up and coming)
Google Web (DMOZ)
Jayde
JoeAnt Directory (up and coming)
Links2Go (powered by GoGuides)
LookSmart
Lycos Directory
Open Directory Project (DMOZ)
SearchKing Directory Network
Skaffe Directory (New)
Websavvy Directory (New)
Xoron
Yahoo Directory
Zeal (owned by LookSmart)

HYBRID
YaGoohoogle : Yahoo and Google results side-by-side

OTHER
Inktomi revising

META-SEARCH ENGINES
note: databases searched change regularly
Chubba : Also provides weather forecasts, Usenet searching, dictionary and encyclopedia lookup.
Dogpile
WEB : Google, Yahoo, AltaVista, Ask Jeeves, About, LookSmart, Overture, FindWhat
Mamma
Metacrawler
WEB : Google, Yahoo, AltaVista, Ask Jeeves, About, LookSmart, Overture, FindWhat
Vivisimo customizable
WEB : Lycos, MSN Search (Yahoo! Search) Netscape (Google), Open Directory (DMOZ), Overture, WiseNut
NEWS : CNN, New York Times, USA Today, Washington Post, BBCNews, YahooNews
OTHER : BBC, Delphion, eBay, GigaBlast, LII, PubMed@NIH, YahooSportNews, YahooTechNews, YahooBizNews
2Trom
800Go
FindIt!
MultiMeta : Bi-lingual Meta Search Engine
Webcrawler : Searches Google, Yahoo, Ask Jeeves, About, LookSmart, Overture, Teoma, FindWhat

PROFESSIONAL SEARCH TOOLS
Copernic Agent : over 90 search engines
SurfWax : over 1850 resources

PPC (Pay Per Click) SEARCH ENGINES/DIRECTORIES
Business.com : Submit here
ePilot : Submit here
eSpotting : Submit here
findwhat : Submit here
Google AdWords™ : Its All About Results™ - CPC : Submit here
Kanoodle : Submit here
Mamma : Submit here
Overture : Submit here
Sprinks : Submit here (owned/powered by Google)
Yahoo! Site Match : Submit here (powered by Overture)

OTHER USEFUL DIRECTORIES
AnyWho : online directory of White Pages, Yellow Pages, reverse lookup and more.
Information Please Almanac : Almanac, encyclopedia and dictionary.
InfoUSA : provider of sales and marketing support for products for all types of businesses, compiles databases of 14 million U.S. businesses and 200 million U.S. consumers, and 1.2 million Canadian businesses and 12 million Canadian consumers.
iTools : Provides quick access to many internet tools.
LookUpUSA : includes over 14 million Yellow Page listings. You can find businesses and professionals by specific Yellow Page headings, and retrieve listings for any city.
RefDesk : Internet reference desk.
Reference.com : Language reference products and services on the Internet.
NIYP SmartPages : interactive Yellow Pages
SuperPages : Verizon Yellow Pages, People Pages, City Pages, Driving Directions, and more.
Switchboard : Provider of local online advertising solutions and Internet-based yellow pages, interconnecting consumers, merchants and national advertisers
Web World Internet Directory : Quality sites on the net.
Who Where : Lycos People Search
World Pages : Business, people, maps.
WWW Virtual Library : the oldest catalog of the web, started by Tim Berners-Lee, the creator of html and the web itself. Run by a loose confederation of volunteers, who compile pages of key links for particular areas in which they are expert; even though it isn't the biggest index of the web, the VL pages are widely recognised as being amongst the highest-quality guides to particular sections of the web.
Yellow : Yellow Pages made simple
YellowPages :


GLOSSARY
Anchor Text : The visible text for a hyperlink.
< href="http://www.thewebstuff.com/">This is the anchor text< /a >
Back Link : A link from another page that points to the page being discussed. Also referred to as an inbound link (IBL).
Bot : Abbreviation for robot (also called a spider). It refers to software programs that scan the web. Bots vary in purpose from indexing web pages for search engines to harvesting e-mail addresses for spammers.
Cloaking : Cloaking describes the technique of serving a different page to a search engine spider than what a human visitor sees. This technique is abused by spammers for keyword stuffing. Cloaking is a violation of the Terms Of Service of most search engines and could be grounds for banning.
Conversion : Conversion refers to site traffic that follows through on the goal of the site (such as buying a product on-line, filling out a contact form, registering for a newsletter, etc.). Webmasters measure conversion to judge the effectiveness (and ROI) of PPC and other advertising campaigns. Effective conversion tracking requires the use of some scripting/cookies to track visitors actions within a website. Log file analysis is not sufficient for this purpose.
CPC : Cost Per Click. It is the base unit of cost for a PPC campaign.
CTA : Content Targeted Ad(vertising). It refers to the placement of relevant PPC ads on content pages for non-search engine websites.
CTR : Click Through Rate. It is a ratio of clicks per impressions in a PPC campaign.
Doorway Page : Also called a gateway page. A doorway page exists solely for the purpose of driving traffic to another page. They are usually designed and optimized to target one specific keyphrase. Doorway pages rarely are written for human visitors. They are written for search engines to achieve high rankings and hopefully drive traffic to the main site. Using doorway pages is a violation of the Terms Of Service of most search engines and could be grounds for banning.
FFA : Abbreviation for Free For All. FFA sites post large lists of unrelated links to anyone and everyone. FFA sites and the links they provide are basically useless. Humans do not use them and search engines minimize their importance in ranking formulas.
Gateway Page : Also called a doorway page. A gateway page exists solely for the purpose of driving traffic to another page. They are usually designed and optimized to target one specific keyphrase. Gateway pages rarely are written for human visitors. They are written for search engines to achieve high rankings and hopefully drive traffic to the main site. Using gateway pages is a violation of the Terms Of Service of most search engines and could be grounds for banning.
Google Dance : Up to June, 2003, Google has updated the index for their search engine on a roughly monthly basis. While the update is in progress, search results for each of Google's nine datacenters are different. The positions of a site appears to "dance" as it fluctuates minute to minute. "Google dance" is an unofficial term coined to refer to the period when Google is performing the update to its index. Google may be changing their index calculation method to allow for a continuous update (which will effectively end the roughly monthly dances).
IBL : Abbreviation for In Bound Link. Any link on another page that points to the subject page. Also called a back link.
Keyword/Keyphrase : Keywords are words which are used in search engine queries. Keyphrases are multi-word phrases used in search engine queries. SEO is the process of optimizing web pages for keywords and keyphrases so that they rank highly in the results returned for search queries.
Keyword Stuffing : Keyword stuffing refers to the practice of adding superfluous keywords to a web page. The words are added for the 'benefit' of search engines and not human visitors. The words may or may not be visible to human visitors. While not necessarily a violation of search engine Terms of Service, at least when the words are visible to humans, it detracts from the impact of a page (it looks like spam). It is also possible that search engines may discount the importance of large blocks of text that do not conform to grammatical structures (ie. lists of disconnected keywords). There is no valid reason for engaging in this practice.
Link Farm : A link farm is a group of separate, highly interlinked websites for the purposes of inflating link popularity (or PR). Engaging in a link farm is a violation of the Terms Of Service of most search engines and could be grounds for banning.
Mirror : In SEO parlance, a mirror is a near identical duplicate website (or page). Mirrors are commonly used in an effort to target different keywords/keyphrases. Using mirrors is a violation of the Terms Of Service of most search engines and could be grounds for banning.
PFI : Abbreviation for Pay For Inclusion. Many search engines offer a PFI program to assure frequent spidering / indexing of a site (or page). PFI does not guarantee that a site will be ranked highly (or at all) for a given search term. It just offers webmasters the opportunity to quickly incorporate changes to a site into a search engine's index. This can be useful for experimenting with tweaking a site and judging the resultant effects on the rankings.
Portal : Designation for websites that are either authoritative hubs for a given subject or popular content driven sites (like Yahoo) that people use as their homepage. Most portals offer significant content and offer advertising opportunities for relevant sites.
PPC : Abbreviation for Pay Per Click. An advertising model where advertisers pay only for the traffic generated by their ads.
PR : Abbreviation for PageRank - Google's proprietary measure of link popularity for web pages. Google offers a PR viewer on their Toolbar.
Robots.txt : Robots.txt is a file which well behaved spiders read to determine which parts of a website they may visit.
Scumware : Scumware is a generic/catch-all label that applies to software that:
Installs itself secretly, dishonestly or without consent
Does not allow for easy uninstallation / removal
Monitors or tracks users actions without the users awareness or consent (aka spyware)
Alters the behavior/default options of other programs without the users consent or awareness (aka thiefware)
SEM : Search Engine Marketing. SEM encompasses SEO and search engine paid advertising options (banners, PPC, etc.)
SEO : Search Engine Optimizer
SERP : Search engine results page
Spam : In the SEO vernacular, this refers to manipulation techniques that violate search engines Terms of Service and are designed to achieve higher rankings for a web page. Obviously, spam could be grounds for banning. Alan Perkins has published an excellent white paper on Search Engine Spam that is highly recommended. Here are some definitions of spam from the search engines themselves
Spamdexing : Spamdexing was describes the efforts to spam a search engine's index. Spamdexing is a violation of the Terms Of Service of most search engines and could be grounds for banning.
Spider : Spider TrapAlso called a bot (or robot). Spiders are software programs that scan the web. They vary in purpose from indexing web pages for search engines to harvesting e-mail addresses for spammers.
Splash Page : Splash pages are introduction pages to a web site that are heavy on graphics (or flash video) with no textual content. They are designed to either impress a visitor or complement some corporate branding.
Stop Word : Stop words are words that are ignored by search engines when indexing web pages and processing search queries. Common words such as the.
www2/www3/www-xx : Google dance watchers use these terms as short-hand to refer to Google's different datacenters. You can add .google.com to the end of them to visit the data center that corresponds to the term.

SEARCH ENGINE OPTIMIZATION TOOLS
Digital Point Tools : Free tools, including Google AdSense Charts, Google AdSense Sandbox, Back Link Tracking Tool, Coop Ad Network, Keyword Ranking Monitor, and Yahoo! Web Rank Tool.
2kmediat.com Keyword Ranking Tool : Free professional keyword ranking and tracking tool.
Poodle Predictor : see what your site will look like in search-engine results
SEO Guy Tools : Keyword + PageRank Finder, Link Popularity Tool

SEARCH ENGINE OPTIMIZATION ARTICLES
SEO and Your Web Site by Alan K'necht
Screwed: Is This an Inevitability in the SEO World? : By Courtney Heard

SEARCH ENGINE STATISTICS
Nielsen NetRatings Search Engine Ratings : April 22, 2005, Article by Danny Sullivan
Vividence Research Report May 25, 2004

SEARCH ENGINE NEWS
CNet - The Net
Search Engine Watch
Wall Street Journal E-Commerce Section
WebProNews
Wired News Technology Section

REMOVE OLD PAGES FROM SEARCH ENGINE
Google
Yahoo

SEARCH ENGINE OPTIMIZATION FORUMS
Search Engine Optimization Forum
JimWorld Search Engine Marketing Forums
SitePoint Search Engine Optimization
Spider Food Forums
Brett Tabke's WebMaster World
SearchEngineWatch
SEO Chat
Cre8asite Forums
Fathoms emarketing forum
I Help You Services
V7n
DigitalPoint Forums
SEO Guy Forums
WebProWorld
SEO Town
LilEngine
Search Engine Forums

SEARCH ENGINE BLOGS
ABAKUS Blog
Battelle's Searchblog
Cre8pc Usability and SEO Blog
Traffick Blog
Jeremy Zawodny's Blog
O'Reilly Network Weblog
Peabody's Cre8tive Flow Blog
Search Engine Blog
Search Engine Lowdown
Search Engine News Blog
Search Visibility Report

ROBOTS.TXT RESOURCES
Robot Exclusion Standards
Robots.txt Editor : visual editor for Robot Exclusion Files and log analyzer software
SEORank's Guide to robots.txt

ISC SEARCH ENGINE LINKS
Copyscape Internet infringement protection - find copies of your content on the web.

Search Engine Spam Detector : This tool analyzes a web page, searching for characteristics that search engines could consider spam.