Which AI crawlers does this check?

Googlebot, Bingbot, Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Claude-User, Applebot, Applebot-Extended, meta-externalagent, CCBot, Bytespider and DuckDuckBot.

If I block GPTBot, will ChatGPT stop citing my site?

Not necessarily. OpenAI uses GPTBot for model training, OAI-SearchBot to index pages for ChatGPT search, and ChatGPT-User when a user asks ChatGPT to open a page. Blocking GPTBot alone keeps you out of training data but still lets ChatGPT search find and cite you. Anthropic splits its crawlers the same way (ClaudeBot, Claude-SearchBot, Claude-User). Perplexity runs PerplexityBot for its search index and Perplexity-User for pages a user asks about.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls whether Google may use your content to train and ground its Gemini models. AI Overviews are part of Google Search and are governed by Googlebot.

The checker says a crawler is allowed. Why does it never visit?

robots.txt is only one layer. A firewall, a CDN bot-protection setting or a hosting provider can block a crawler that robots.txt allows. Check your server or CDN logs for the crawler's user agent.

Free tools

AI crawler checker

See which AI and search crawlers your robots.txt allows or blocks, for your homepage and any page you want to test. We also check whether you publish an llms.txt file.

Why it matters

ChatGPT, Claude, Perplexity and Gemini can only cite a page their systems were allowed to read. Many sites block AI crawlers without meaning to: a security plugin adds a blanket rule, a CDN turns on bot protection, or someone copies a robots.txt that blocked GPTBot in 2023 and never revisits it.

Training crawlers vs search crawlers

Most AI companies now run separate crawlers for separate jobs, and the table above labels each one. Training crawlers and controls (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) decide whether pages can be used to train models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages so the assistant can find and cite them when answering. On-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User) open a page when a user asks about it. You can block training and still allow search, which is the choice many publishers make.

A robots.txt that allows AI search but not training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Every crawler not named falls through to the * group, so OAI-SearchBot, Claude-SearchBot and PerplexityBot stay allowed here. Run the checker again after you change the file.

After robots.txt

Being readable is the first step. To see whether the engines actually mention and cite you, track the questions your buyers ask with XeoRank AI visibility, or start with a free site analysis. If you want a summary file for AI tools too, use the llms.txt generator.

Questions

Which AI crawlers does this check?: Googlebot, Bingbot, Google-Extended, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Claude-User, Applebot, Applebot-Extended, meta-externalagent, CCBot, Bytespider and DuckDuckBot.
If I block GPTBot, will ChatGPT stop citing my site?: Not necessarily. OpenAI uses GPTBot for model training, OAI-SearchBot to index pages for ChatGPT search, and ChatGPT-User when a user asks ChatGPT to open a page. Blocking GPTBot alone keeps you out of training data but still lets ChatGPT search find and cite you. Anthropic splits its crawlers the same way (ClaudeBot, Claude-SearchBot, Claude-User). Perplexity runs PerplexityBot for its search index and Perplexity-User for pages a user asks about.
Does blocking Google-Extended remove me from AI Overviews?: No. Google-Extended controls whether Google may use your content to train and ground its Gemini models. AI Overviews are part of Google Search and are governed by Googlebot.
The checker says a crawler is allowed. Why does it never visit?: robots.txt is only one layer. A firewall, a CDN bot-protection setting or a hosting provider can block a crawler that robots.txt allows. Check your server or CDN logs for the crawler's user agent.

More free tools

llms.txt generator

Reads your sitemap and page titles and drafts an llms.txt file you can edit and download.

Robots.txt generator

Build a robots.txt that allows AI search crawlers and blocks AI training crawlers, with a WordPress preset.

Meta tag checker

Title, meta description, canonical, robots, H1 and Open Graph tags for up to 10 pages, with length checks and duplicates.

Sitemap checker

Finds every sitemap a site declares, counts the URLs, checks lastmod dates and spot-checks that listed URLs return 200.

Redirect checker

Tests http, https, www and non-www versions of a site and shows every redirect hop, so you can see if they all land on one URL.

Schema markup generator

Build JSON-LD for Organization, LocalBusiness, FAQ, Product and Article pages, ready to paste into your site.

Keyword density checker

Word count and the most repeated words and phrases in your text, plus how often a target keyword appears. Runs in your browser.

SERP snippet preview

See how your title and meta description may look in Google on desktop and mobile, and where they get cut off.

Open Graph checker

og:title, og:description, og:image and twitter:card tags for up to 5 pages, with a preview of the shared link card.

H1 checker

The H1 heading of up to 10 pages, flagging pages with none or more than one.

Canonical tag checker

Whether each page's canonical points to itself, is missing, or tells Google to index a different URL.

Hreflang checker

Hreflang tags, language and region codes, x-default and return links between language versions.

Core Web Vitals checker

Real-user LCP, INP and CLS from the Chrome UX Report plus a Lighthouse lab test, on mobile or desktop.