What is the invisible web?
Understanding how web search engines work

Despite the power of robots, it's not possible for search engines to explore all web pages. The "invisible" web, also sometimes called the "deep web", corresponds to pages that cannot be located or indexed by robots.

Several types of content can be included, including :

• pages that have been voluntarily excluded by the webmaster; any publisher can decide to prevent a robot from indexing his site – or part of his site –; all he has to do is specify this in the page meta tags;

• sites for which prior identification is required (whether access is charged for or not);

• pages that are not indexed by robots for technical reasons. This may be due to the format of the page, but more often than not, "invisible" pages are dynamic pages, generated "on the fly" in...