Skip to content
← SEO Dictionary

Crawling

Crawling is a bot fetching content by requesting a site's URLs.

From discovery to possible indexing

URL discovery → Possible crawl → Processing / rendering → Possible indexing
Simplified model, not a guaranteed sequence. A URL can be excluded at different stages; discovery and crawling do not guarantee indexing.
Text version
  • URL discovery
  • Possible crawl
  • Processing / rendering
  • Possible indexing

How it works

The bot discovers URLs through links and other signals and checks accessible responses; this is separate from indexing.

Example

Example: A new category has an internal link through which a crawler can discover it.

Common mistakes

A crawled URL is not necessarily indexed.

How can you check this in practice?

Inspect requested URLs, statuses and times in logs. Verify the crawler before calling requests Googlebot. A URL may be discovered through a link or sitemap without being fetched immediately. There is no guaranteed recrawl schedule.

Distinctions and limitations

Crawling requests content; indexing is a decision after processing it. HTTP 200 proves a successful response for a particular request, not inclusion in the index. Blocked resources can also constrain rendering.

Official sources

Editorial responsibility: Martin Zhelyazkov · Founder of Z-Consult

Updated: 05/11/2026

For a correction or question: office@z-consult.bg