Comment gérer un site que les moteurs de recherche aiment explorer.
"To provide the best answers to search queries, search engines maintain an index, or catalog, of known URLs that can be retrieved by topic, search intent, keyword, etc.
To create, maintain and expand this index, search engines crawl links on known URLs to discover new URLs and revisit URLs that may have been updated. Each new URL is analyzed and catalogued as a potential answer to future search queries.
When creating a results page for a search query, search engines rank URLs in their index by various qualities in an attempt to list the URL that best matches the query first.
But in reality...
Crawling is a complex topic. Search engines invest enormously in discovering, analyzing, and cataloging URLs on the web. These investments cover not only the technology they use, but also the energy required to handle the sheer size of the internet."
