How it works
It is supplied through robots metadata or an X-Robots-Tag; the crawler must be able to read the response.
Example
Example: A temporarily accessible test page may use noindex, but private data needs access control.
Common mistakes
Do not block crawling when you expect a crawler to see noindex.
How can you check this in practice?
Inspect the HTML head and X-Robots-Tag response header. The crawler needs access to read the directive. Removing noindex requires a subsequent crawl; the index does not update immediately.
Distinctions and limitations
Disallow restricts requests; noindex instructs exclusion from results. Combining them can prevent the crawler reading noindex. Neither protects personal data: private content requires access control.
Official sources
Editorial responsibility: Martin Zhelyazkov · Founder of Z-Consult
Updated: 05/11/2026
For a correction or question: office@z-consult.bg