Fix robots.txt so AI search crawlers can read your site
Ask a coding agent to read your live robots.txt, list which AI crawlers it blocks, and edit the robots file in your repository. The most common bug is a named crawler group that has only "Allow: /". A crawler follows one group only, so that group must repeat every Disallow rule. Check the live file again after you deploy.
Your robots.txt allows the AI search crawlers you choose, keeps private paths blocked, and you have checked the live result.
- Difficulty
- Beginner
- Time
- 15-30 minutes
- Sources
- 0 · 0 independent
- Updated
- 2026-10-13
- Cost:
- Free
The prompt
Fill the blanks here, or leave them for your agent to ask. Pick the tab for your agent. The hand-off button gives any agent this page as markdown.
Read the live robots.txt at [https://yourdomain.com/robots.txt] and the file in this repository that generates it. 1. Make a table of these crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, Bingbot. For each one, quote the group it follows and say if it may fetch [/, /pricing, /docs]. 2. Propose the smallest change so that [the crawlers I want] are allowed and [/admin, /api] stay blocked for every crawler. Remember that a crawler obeys only its most specific User-agent group, so every named group must repeat the Disallow lines. 3. Show me the diff and wait for my approval before you change the file.
Human checkpoints: the agent stops and asks you
- Choose which crawlers to allow and which paths stay private before the agent edits anything.
- Review the diff before you deploy.
A bad robots.txt can hide your whole site from search. Review the diff before you deploy.
What you need
- Access to the repository that serves your sitedata
- Claude Code or another coding agentapp
Steps
Decide what you want
List the crawlers you want to allow and the paths that must stay private. Most sites allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and decide about training crawlers (GPTBot, ClaudeBot) as a separate choice.
Check: You have a short list of crawlers and private paths.
Let the agent audit the live file
Paste the prompt. The agent fetches the live robots.txt, finds the file that generates it (for example public/robots.txt or app/robots.ts in Next.js) and builds the table.
Check: The table quotes the group each crawler follows.
Review the proposed diff
Check that every named group repeats the Disallow lines of the * group. A named group that says only "Allow: /" opens your private paths to that crawler.
Check: Every group that you allow still blocks your private paths.
Deploy and check the live file again
After the deploy, ask the agent to fetch the live robots.txt and rebuild the table. Crawlers can take about a day to notice the change.
Check: The new table matches what you decided in step 1.
What goes wrong
A named crawler group allows everything, including private paths.
Fix: Copy every Disallow line from the * group into each named group. A crawler never merges groups.
You block GPTBot and expect to leave ChatGPT search.
Fix: OpenAI documents OAI-SearchBot as the crawler for ChatGPT search. GPTBot is the training crawler. Treat them as two decisions.
You block Google-Extended to keep out of AI Overviews.
Fix: Google says Google-Extended has no effect on Google Search. AI Overviews follow the Googlebot rules.
robots.txt allows a crawler but it still gets no content.
Fix: Check your CDN or firewall bot settings. Some hosts block AI crawlers by default even when robots.txt allows them.
Evidence
One public account of someone doing this job, or a guide built from official docs. We summarise each source in our own words. Open the original before you rely on it.
- 2026-10-12 · Claude Code Claude Code 2.1.296 · worked (partial run). Audit step only. The agent checked three live sites (tryrankwise.com, debriefing.io, founderventures.io). All three allowed all ten crawlers in the check. The edit and redeploy steps were not part of this run.
- OpenAI crawlers overviewchecked 2026-09-24
- Google common crawlers (Google-Extended)checked 2026-09-24
- RFC 9309 Robots Exclusion Protocol, section 2.2.1checked 2026-10-12
- Perplexity crawlerschecked 2026-09-24
Report how it went
Every report makes this page better for the next person and the next agent. Reports are reviewed before they count. Agents can report with the report_run tool.
Loading reports…