Should I Allow GPTBot and Other AI Crawlers?

AI crawler

An automated fetcher operated by an AI company, either to build a search index, to collect model training data, or to retrieve a page a user just asked about.

Also known as: AI bot, LLM crawler, AI user agent

The three purposes matter more than the brand names, because they have different consequences. A search crawler decides whether you can be cited. A training crawler decides whether the model knows your category vocabulary at all. A user-directed fetcher retrieves a page because a person asked for it, and blocking that one refuses a request a human made deliberately.

GPTBot is a training crawler. The case for allowing it is that a model with no exposure to your product will still answer questions about your category, using whatever it did see. For a publisher whose revenue comes from archive access, that trade looks different. For a business that wants to be a known option in its market, refusing to be learned is not protection.

Two rows deserve attention because they change the maths. Perplexity states that a site disallowing PerplexityBot may still have its domain, headline and a brief factual summary indexed, so blocking buys a worse version of being included. And Perplexity-User generally ignores robots.txt, which means the live-fetch path stays open regardless.

The paper is "Strategic Response of News Publishers to Generative AI" by Zhao at Rutgers and Berman at Wharton, first posted 31 December 2025 and revised in April 2026. It uses a causal design over Comscore and SimilarWeb data spanning October 2022 to June 2025, which makes it the strongest evidence available on the blocking decision. The headline number shifted between versions because of the measurement window, not because anything was retracted.

The finding that matters is the second half. Blocking reduced traffic and did not reliably reduce how often the blocked publishers were cited. That asymmetry has a simple mechanism: models also learn about you, and describe you, from what other sites say. Removing your own pages from retrieval removes the version you control and leaves the versions you do not.

For a business selling a product rather than access to an archive, the trade is even more one-sided. The realistic choice is between being described accurately with a link to your page and being described from second-hand sources with no link, and only one of those two involves you.

This is the single most expensive line anyone can put in a robots.txt in 2026, and it gets written by accident. It happens when a blanket disallow is added for "AI bots" during a policy discussion, and OAI-SearchBot is swept up with GPTBot because both have OpenAI in the name. The two do completely different jobs.

Bingbot is the second non-negotiable, for a less obvious reason. Microsoft documents that Copilot Search is grounded on Bing search results, and that Microsoft 365 Copilot generates a search query and sends it to the Bing search service. So a Bing index entry is the gate for Copilot the way an OAI-SearchBot fetch is the gate for ChatGPT search.

Oppira allows every crawler listed in the table above, with no rate limiting at the edge and no user-agent branching in the response. For a site of a few hundred pages the crawl cost is negligible, and the visibility cost of the alternative is not.

The two controls are separate and often confused. Google-Extended is a robots.txt token covering Gemini grounding and model training. The generative AI control in Search Console is a different mechanism that governs whether your pages can appear in AI Overviews and AI Mode at all.

Google is explicit about what the Search Console control does: it stops links and content appearing in those surfaces, and it removes the impressions and clicks that came with them. Google also states it is not a ranking signal, so there is no ranking benefit to trade against the lost visibility. Using it means choosing to be absent from Google generative AI features.

Google-Extended is a milder version of the same trade. Blocking it keeps you out of Gemini grounding while leaving Search ranking untouched, which means the cost is real and the benefit is limited to not being in a training set. For a business that wants to be a known option in its category, that is the wrong side of the trade.

The first case is an infrastructure decision rather than a visibility one. Some crawlers fetch heavily and send no traffic back, and if crawl volume is genuinely costing you money, rate-limiting at the edge is the appropriate response. Do it by measured request volume from your own logs, not by reputation.

Crawl-to-refer ratios circulate widely as the justification, with figures around 10,300 requests per referral for ClaudeBot, roughly 1,276 for GPTBot and roughly 193 for Perplexity. Those numbers are reported second-hand rather than read from a primary publication, so use them as a hint about which agents to look at in your own logs, not as a decision rule.

The second case is genuine and specific. If people pay for access to your archive, every free crawl of it is a substitute for a subscription, and the Zhao and Berman result about lost traffic matters less than the cannibalisation. That is the situation news publishers are in, and it is why their choices should not be copied by businesses in a different one.

For everyone else the answer is the boring one. Allow the crawlers, keep the pages readable without JavaScript, fix the dead links, and spend the effort saved on having something specific worth quoting.