| A | - | B | - | C | - | D | - | E | - | F | - | G | - | H | - | I | - | J | - | K | - | L | - | M | - | N | - | O | - | P | - | Q | - | R | - | S | - | T | - | U | - | V | - | W | - | X | - | Y | - | Z |
| Crawler | A web crawler is one type of bot, or software agent and also known as a robot, or spiders. It starts with a list of URLs to visit and identifies all the hyperlinks in the page and adds them to the list of URLs to visit. It recursively browses the Web according to a set of policies. Web crawlers are used to 1.Create a copy of all the visited pages by the search engine for later processing and use. The downloaded pages are indexed to provide fast searches. 2.Gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam). 3.For automating maintenance tasks on a web site including checking links or validating HTML code. |
|