What is robots.txt? Crawling rules, examples, and AI bots

Robots.txt is a plain text file located in a website’s root directory that tells search engine crawlers which addresses on the site they can access. The robots.txt file manages crawling, not indexing: to remove a page completely from Google results, the correct tool is noindex or password protection.

This guide bases its definition on Google Search Central documentation and then moves on to practical matters: the file’s syntax, how rules are matched, the difference between blocking and noindex, commonly used examples, editing in WordPress, testing with Search Console, and managing AI bots such as GPTBot, OAI-SearchBot, and Google-Extended, a topic frequently asked about recently.

What does the robots.txt file do?

This file is primarily used to manage crawler traffic and prevent crawling of unimportant or very similar sections of a website. Google’s introductory documentation on the subjectalso emphasizes that this file is not a mechanism for keeping web pages out of Google.

The answer to the question of what robots.txt is becomes complete by understanding what the file affects, and that effect varies by content type. For web pages, it is used to reduce crawling load or exclude sections that generate an infinite number of addresses, such as filters and search results pages. For images, videos, and audio files, it can prevent those files from appearing in Google search results. Resources such as stylesheets and scripts can also be blocked, but if proper page rendering depends on those resources, blocking them makes it harder for Google to understand the page.

The file is not a security tool either. The rules do not force a crawler to comply; Googlebot and other trustworthy crawlers follow them, while malicious bots may not. Furthermore, the file is public; writing the name of a folder that should remain private here means announcing it to everyone.

How do you write robots.txt? Basic syntax

The file consists of one or more rule groups; each group begins with a user-agent line and lists, line by line, the paths that crawler can or cannot access. The file must be plain text encoded in UTF-8, named robots.txt, and published in the root directory of the host to which it applies; for example, https://www.example.com/robots.txt. A copy placed in a subfolder is disregarded.

The fields supported by Google are as follows:

• user-agent: which crawler the rule applies to. The asterisk (*) matches all crawlers, except Google’s AdsBot crawlers, which must be specified separately by name.
• disallow: the path that should not be crawled; it begins with / and ends with / if it refers to a folder.
• allow: allows a specific subpath inside a blocked folder to be crawled.
• sitemap: the full address of the sitemap; it is optional and can be used more than once in the file.

Other fields, such as crawl-delay, are not supported by Google. Below is a simple file consisting of two groups: the first group blocks the administration panel folder for all crawlers, while the second line states the location of the sitemap.

User-agent: *
Disallow: /yonetim/

Sitemap: https://www.example.com/sitemap.xml

How are robots.txt rules matched?

A crawler selects the most specific user-agent group matching itself in the file and ignores the other groups; within the group, the most specific rule according to path length is applied. According to Google’s documentation explaining how it interprets the robots.txt specification , when equally specific allow and disallow rules conflict, the less restrictive allow rule is applied.

This rule has a practical consequence: when a separate group is written for Googlebot, Googlebot no longer reads the general group with the asterisk. If a restriction added to the general group should also cover Googlebot, the rule must also be written in the Googlebot group. Paths are case-sensitive; /Urun/ and /urun/ are different paths. In a path, the asterisk (*) represents any sequence of characters, and the dollar sign ($) represents the end of the address.

The limits relating to the file itself should also be understood. Google applies a size limit of 500 KiB to this file and ignores the portion exceeding that limit; it generally caches the file’s contents for up to 24 hours. If the file returns a 4xx error such as 404 (except 429), Google assumes there are no crawling restrictions; if it returns a server error (5xx), Google first stops crawling and then uses the last valid version for a while. In other words, a file returning a server error can cause more problems than a file that does not exist at all.