How to use the robots.txt generator
- Pick a preset: allow everything, block everything, block private folders, or block AI training crawlers.
- Edit each group's user-agents and the paths to allow or disallow, one per line.
- Add your sitemap URL(s) as absolute links.
- Review the checks, then copy or download robots.txt and upload it to the root of your domain.
Features
- Multiple user-agent groups with add and remove
- Allow and Disallow rules, with Allow exceptions inside blocked folders
- Crawl-delay with a note about which crawlers honour it
- Sitemap lines, validated as absolute URLs
- Lint warnings for blocking the whole site, relative paths, blocked CSS/JS and missing sitemaps
- Presets including one for GPTBot, ClaudeBot, CCBot and Google-Extended
How robots.txt is read
Crawlers fetch /robots.txt before anything else on a host. The file is a list of groups: one or more User-agent lines followed by Allow and Disallow rules. A crawler uses the most specific group that names it (Googlebot picks the Googlebot group over *). Within a group, the longest matching path wins, and Allow beats Disallow on a tie. Paths are case-sensitive prefixes, with * as a wildcard and $ to anchor the end.
robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it; use a noindex meta tag (on a crawlable page) to keep something out of the index. It is also public and advisory, so never rely on it to hide sensitive content.
Crawl-delay and AI crawlers
Crawl-delay asks a bot to wait a number of seconds between requests. Bing, Yandex and several others respect it; Googlebot ignores it entirely and adjusts its rate automatically, with a manual limit available in Search Console.
GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and Google-Extended (Google's AI training token) each declare themselves so site owners can opt out of training data collection. Blocking them is a legitimate choice, but note that Google-Extended does not affect Search ranking and that some AI-assisted search features may stop citing your pages.