How to Make a React App Visible to Crawlers and AI

The measurement is specific enough to be worth quoting. Across the requests Vercel and MERJ analysed in December 2024, GPTBot accounted for 569 million and ClaudeBot for 370 million. GPTBot downloaded JavaScript files 11.5% of the time and ClaudeBot 23.8%, and neither executed any of them. Downloading a bundle is not running it.

So a single-page application that ships an empty root element and fills it after hydration presents nothing to retrieve. The heading exists, the answer exists, the internal links exist, and none of them are in the bytes the crawler received. This failure is silent, because the page looks perfect in a browser and passes every visual check a marketing team would run.

The same dataset found a second problem that compounds the first. The ChatGPT fetcher hit 404s on 34.8% of its requests and ClaudeBot on 34.2%, against 8.2% for Googlebot. AI crawlers follow stale and malformed links far more often than Googlebot does, so broken internal links waste a much larger share of their crawl.

Prerendering

Running an application once per route at build time and writing the resulting HTML to a file, so the server can return a complete page without executing any JavaScript.

Also known as: Static site generation, SSG, Static prerendering

The decision is per route, not per app. A marketing site with a logged-in dashboard behind it can prerender every public page and leave the application shell entirely client-rendered, because nobody needs a crawler to read a dashboard.

A useful sanity check on the whole pipeline: the build should fail, not warn, when a route in the data has no prerendered output. Silent gaps are the normal failure mode here, because nothing about the site looks broken when a single page is missing its HTML.

Tabs and accordions deserve a specific note because they are the most common accidental hiding place. Content that is present in the HTML and hidden by CSS is retrievable. Content that mounts only when a tab is clicked is not, and the difference is invisible in a browser.

The same rule applies to anything loaded from your own API at runtime, including a list of related pages, a rating, or a table of numbers. If it matters for retrieval, it has to be in the build output rather than fetched on mount.

Google's spam policies define cloaking as presenting different content to users and to search engines with the intent to manipulate rankings. A prerendered page that is byte-identical for every requester is not cloaking, and it is also simpler. A branch that sends bots a fuller page is both a policy risk and a maintenance problem.

The practical failure is more mundane than the policy one. Once two payloads exist, one of them stops being tested. The bot path keeps rendering last quarter's template, or drops a section that was added to the human path, and nobody notices because nobody browses the site as GPTBot.

There is one adjacent trap worth naming: gating the prerendered response on a Googlebot user-agent check. That pattern was common when the only crawler that mattered was Googlebot, and it now excludes OAI-SearchBot, PerplexityBot, Claude-SearchBot and Bingbot from the only readable version of the site.

Do this after every significant template change, not once. Rendering regressions are introduced by ordinary refactors: a component moved behind a lazy import, a section switched to fetch its own data, a route added without a prerender entry. All three look fine in a browser.

Server logs are the confirming evidence. Segment requests by user agent and watch which pages each crawler fetches and what status it gets, verifying against the published IP ranges rather than trusting the user-agent string, because spoofing is common.

Googlebot is the exception among crawlers that matter: it executes JavaScript. That is why client-rendered marketing sites sometimes rank perfectly well and their owners conclude rendering is a solved problem. It is solved for exactly one engine.

Even for Google, prerendering removes a class of risk rather than adding a boost. Rendering happens on a separate pass, which means content can be indexed late or partially, and a script error can cost a page its body copy. Static HTML has none of those failure modes.

The stronger argument is coverage. Google states that eligibility for AI Overviews and AI Mode requires a page to be indexed and eligible to be shown with a snippet, and every other engine reads raw HTML only. One prerender pass satisfies all of them, so there is no scenario where the JavaScript-dependent version is the better choice for a public page.