Skip to content
← SEO Dictionary

robots.txt

robots.txt is a text file containing URL crawling rules for crawlers that support it. It is located at the root of the relevant website.

Example robots.txt

User-agent: *
Disallow: /filter/
Allow: /products/
Sitemap: https://example.com/sitemap.xml

Example for example.com, not Z-Consult’s actual rules. This does not protect private data.

Crawling rules

Compliant crawler → robots.txt · User-agent → Allow → permitted path → Disallow → prohibited path
These rules govern crawling, not privacy, and do not guarantee exclusion from the index.
Text version
  • Compliant crawler
  • robots.txt · User-agent
  • Allow → permitted path
  • Disallow → prohibited path

Why it matters

Its rules guide crawling toward the necessary parts of a site. They can reduce requests to low-value variants, but do not protect private information.

How it works

User-agent identifies the crawler to which rules apply. Allow and Disallow describe permitted and prohibited paths. Rules apply to the protocol, host and port where the file is published.

Example

Example: User-agent: * Disallow: /filter/ This requests that compliant crawlers avoid URLs under /filter/.

Common mistakes

A URL blocked from crawling may still appear in search. When a crawler cannot fetch a page, it may not see that page’s noindex instruction. Use actual access control for private data.

How can you check this in practice?

The file sits at the root of a specific protocol and host, for example https://example.com/robots.txt. Check the User-agent group and matching path. Google's rules are case-sensitive; with Allow/Disallow, the most specific matching rule matters.

Distinctions and limitations

robots.txt governs crawling, not indexing or confidentiality. Sitemap identifies a URL list but does not allow an otherwise blocked path. Google does not support noindex as a robots.txt directive; use accessible metadata or an HTTP directive instead.

Directive syntax

User-agent selects a rule group. Disallow specifies a blocked path; Allow an allowed path. An empty Disallow does not block every URL. Sitemap uses the absolute sitemap URL. Comments start with #; the example file is not Z-Consult's actual policy.

Resource access

Blocked images, CSS or JavaScript can prevent accurate page understanding. Check resources as well as the HTML URL. Rules apply to their host and protocol, not automatically to all subdomains.

Official sources

Editorial responsibility: Martin Zhelyazkov · Founder of Z-Consult

Updated: 05/11/2026

For a correction or question: office@z-consult.bg