The Web Robots Pages
Martijn Koster's FAQs about the programmes that index the Web, how to direct visiting robots, a database of robots, an archived mailing list, background reading, and related links.
http://www.robotstxt.org/
Home / Internet & Online / Internet / Searching / Search Engines / Robots /
|
Search Engine Robots and Other User Agents
John A. Fotheringham presents data in tabular form on the robots sent by search engines and other sites to read and index Web pages: their origins, names and IP addresses.
http://www.jafsoft.com/searchengines/webbots.html
Home / Internet & Online / Internet / Searching / Search Engines / Robots /
|
List of Robot Agent Strings
A list from PGTS of Web robots with the identifying data they leave in Web site logs.
http://www.pgts.com.au/pgtsj/pgtsj0208d.html
Home / Internet & Online / Internet / Searching / Search Engines / Robots /
|
All About Search Indexing Robots and Spiders
Search Tools Consulting explains how the search engine programs called "robots" or "spiders" work, and reviews related sites.
http://www.searchtools.com/robots/
Home / Internet & Online / Internet / Searching / Search Engines / Robots /
|
Search Engine IP Addresses
Lists IP addresses of search engine spiders. Can be searched by IP address. Also links to resources on spiders.
http://www.iplists.com/
Home / Internet & Online / Internet / Searching / Search Engines / Robots /
|
Psychedelix List of User-Agents
Andreas Staeding's large list of search engine spiders, similar Web robots, and Web browsers: their web-log identification and links to their originators.
http://www.psychedelix.com/agents.html
Home / Internet & Online / Internet / Searching / Search Engines / Robots /
|
The User-Agent Database
Icehouse has gathered a list of spiders (robots or crawlers), browsers and harvesters, sorted into a searchable database.
http://www.icehousedesigns.com/useragents/
Home / Internet & Online / Internet / Searching / Search Engines / Robots /
|
HTTP User Agent Index
An alphabetical list of user agents and the deployer behind them, compiled by Christoph Rüegg.
http://www.siteware.ch/webresources/useragents/db.html
Home / Internet & Online / Internet / Searching / Search Engines / Robots /
|
Robots.txt Syntax Checker
Online tool for syntax verification to robots.txt files, provided by Simon Wilkinson.
http://www.sxw.org.uk/computing/robots/check.html
Home / Internet & Online / Internet / Searching / Search Engines / Robots /
|
