AIRankWProbots.txt for AI crawlers: what to allow and what to block
Short answer: if you want to appear in AI answers, do not block the search crawlers. You can still opt out of model training separately. Check each company's own docs, because bot names change.
Search and training are separate switches
OpenAI documents separate robots.txt tags: OAI-SearchBot is used so sites can appear in ChatGPT search results, and GPTBot is for content that may be used to train its models. A site can allow one and disallow the other. OpenAI says that to be included in summaries and snippets in ChatGPT search you should not block OAI-SearchBot.
Google works differently
Google says robots.txt for Googlebot is the control for crawling for Search, and AI Overviews and AI Mode are part of Search. To limit what is shown from a page use nosnippet, data-nosnippet, max-snippet or noindex. Google-Extended is the separate control for some other Google AI uses such as training and grounding in other systems.
A simple starting file
User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: / User-agent: * Allow: /
This example allows ChatGPT search crawling and blocks training crawling. Whether to block training is your call; blocking it is not required to appear in answers. Also check that your CDN, firewall or security plugin is not blocking these bots, since robots.txt is not the only gate.
Check before you trust it
- Open yoursite.com/robots.txt and read it.
- Look for rules from your SEO or security plugin.
- Look at your host or CDN bot settings.
- Check the vendor docs again every few months.
Sources
- OpenAI: crawlers and user agents
- OpenAI Help Center: Publishers and Developers FAQ
- Google Search Central: AI features and your website
Names belong to their owners. This post is general information, not legal advice.