Which AI Crawlers Should You Allow? The Bot-by-Bot Chart

On October 30, 2026, an IETF draft defined a single robots.txt token, __usage_ai-training, that states a training preference for every company that honors it. It’s still a work-in-progress draft, not a finished standard, and support arrives on different schedules. It also isn’t a crawl block: bots keep fetching your pages, and the token only says how the content may be used. Participation is voluntary, so it speaks only to the providers that choose to honor it. Existing vendor-specific rules keep working. One wrinkle with Google: a Google-Extended rule is all or nothing, so Allow: / keeps Gemini grounding but also permits Gemini training.

Sources: Anthropic, Google, Microsoft.

A perfect robots.txt means nothing if your firewall, or your host’s, turns the bots away at the door. That line reads every matching log file, plain and compressed, so the date range is whatever your host keeps, and keeps bot, URL and status together, so a 200 on /robots.txt can’t hide a 429 on an article. It assumes the standard combined log format, so check yours first. If your host keeps a duplicate file like access.log.0, leave it out so a day isn’t counted twice.

On Shopify, Squarespace or Wix you won’t get raw server logs. If your site runs behind Cloudflare, its AI Crawl Control dashboard shows which AI crawlers got through and the status codes they received.

A 403, 429 or 503 deserves a look, but the status alone won’t tell you the cause or where it came from. The same firewall that throttled Perplexity on my site was also, correctly, turning away bots probing my login page. So check which bots got it, on which pages. User agents are easy to fake, so confirm the IP against the operator’s list: OpenAI has one per bot, Perplexity has two, and Anthropic and Apple publish theirs too.

llms.txt is a proposed Markdown index of a site’s key pages, written for AI tools (here’s the original proposal). It gets a lot of discussion in SEO circles. Google’s AI optimization guide says Google Search doesn’t use AI text files, and that keeping one will neither harm nor help your visibility. OpenAI, Anthropic and Perplexity publish llms.txt files for their own developer docs, but none of their crawler pages lists it as a factor in search inclusion or citations.

My own logs show the same. This site has served an llms.txt since September. Over eight days it got 7 requests: one identified itself as GPTBot, four came from SEO tools and two from browsers. None identified as the AI search crawlers I was tracking. That matches Ahrefs’ study of 137,210 domains using Ahrefs Web Analytics that got traffic in May 2026: of the roughly 38,000 that published an llms.txt, 97% got zero requests in May 2026, and AI search crawlers made just 1.1% of the requests that did happen. And a fetch doesn’t prove the file shaped a single answer.

Where it earns its keep is developer documentation. A coding agent can use it as a documentation index when pointed to it, which is why Anthropic publishes one for Claude Code’s docs and Cursor for its API docs. For a marketing site, it’s optional. I keep mine because my theme generates it automatically and it costs nothing. I wouldn’t spend money on one.

The same goes for publishing Markdown copies of your pages, the way Meta does for its developer docs. AI agents and coding tools do request them (one site owner logged about 1,400 Markdown requests in 44 days), but nobody has shown it earns citations, and Google’s guide says you don’t need Markdown files to appear in Search. If your site already runs on Cloudflare’s Pro plan, its Markdown for Agents toggle does it for free. Otherwise, I wouldn’t build it by hand.