Most AI Crawlers Don't Run JavaScript. Can They Read Your Site?
What ChatGPT and Claude know about you partly comes from crawlers reading your pages. Words that only appear after a script runs, they miss.
When ChatGPT or Claude answers a question about your business, part of what it knows came from a crawler that visited your site. What that crawler took away depends on how your pages are built.
In December 2024, Vercel and MERJ published what they had seen of AI crawlers across Vercel’s network over several months. None of the major ones ran JavaScript. That covered OpenAI’s GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic’s ClaudeBot, Meta’s crawler and ByteDance’s Bytespider, among others. The ChatGPT and Claude crawlers downloaded JavaScript files, in 11.5% and 23.8% of their requests, and ran none of them (Vercel, The rise of the AI crawler). Googlebot and Apple’s crawler do run it.
These crawlers are busy. In the month before that report, GPTBot made 569 million requests across Vercel’s network and Claude’s crawler 370 million.
The study is almost two years old and crawlers change, so treat it as a snapshot. It is still the clearest public measurement we know of.
What a crawler that doesn’t run JavaScript sees
On a page whose words are in its HTML, it sees everything. On a single-page app it sees the shell: a title, some script tags and an empty element waiting for code to fill it.
Some builders work around this on their own hosting. Lovable serves pre-rendered copies of its older apps’ published pages to verified search engines, social preview bots and AI crawlers (Lovable FAQ). Move the app elsewhere and that copy stays behind.
Check your own site
Open one of your pages, choose View Page Source (not the developer tools, which show the page after scripts have run) and search for a sentence you can see on screen. If it is in the source, any crawler can read it. If the source is mostly script tags, a crawler that doesn’t run JavaScript gets little.
From a terminal, the same test is one line:
curl -s https://example.com/about/ | grep "a sentence from the page"
No output means the sentence isn’t in the HTML.
Static pages pass
Every page on a static site is built as HTML before anyone visits, so the words are in the source for every reader, whether it runs scripts or not. Our analytics counts AI crawlers in a tab of their own, apart from your human visitors, so you can see which ones come.
If you would rather a crawler stayed away, robots.txt is the place to ask it to.
Yesterday’s post: ask your AI tool for a Hugo site, not a single-page app.
Ready to Deploy?
Join Jamstack developers across the UK hosting their static sites with privacy and performance built in. Hosting is free for good, no credit card needed, with Plus features when you want them.
