Where do I put robots.txt?

At the root of the host it controls, so it loads at https://yoursite.com/robots.txt. A robots.txt on www.example.com does not apply to shop.example.com; each subdomain needs its own.

Should I block GPTBot?

Block GPTBot if you do not want OpenAI to train models on your pages. It does not remove you from ChatGPT search, which uses OAI-SearchBot. Most businesses that want to be recommended by AI assistants keep the search crawlers allowed.

Does robots.txt stop a page appearing in Google?

No. robots.txt controls crawling, not indexing. Google can still index a blocked URL if other pages link to it, without reading its content. To keep a page out of results, allow crawling and add a noindex meta tag instead.

Do all crawlers obey robots.txt?

Reputable search and AI crawlers say they follow it. It is a request, not access control, so use authentication or firewall rules for anything that must stay private.

Free tools

Robots.txt generator

Pick which AI crawlers may read your site, list the paths to keep private, and copy a robots.txt that search engines and AI assistants will read correctly.

Your domain

AI crawlers

GPTBot OpenAI · training
OAI-SearchBot OpenAI · search
ChatGPT-User OpenAI · user fetch
ClaudeBot Anthropic · training
Claude-SearchBot Anthropic · search
Claude-User Anthropic · user fetch
PerplexityBot Perplexity · search
Perplexity-User Perplexity · user fetch
Google-Extended Google · training
Applebot-Extended Apple · training
meta-externalagent Meta · training
CCBot Common Crawl · training
Bytespider ByteDance · training

Paths to keep every crawler out of (one per line)WordPress site (block /wp-admin/, keep admin-ajax.php open)

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml

The choice most sites now make: AI search yes, training no

OpenAI, Anthropic and Google now separate the crawler that collects pages for model training from the one that lets their assistant find and cite you. The “Allow AI search, block AI training” preset blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot and Bytespider, and leaves OAI-SearchBot, Claude-SearchBot, PerplexityBot and the on-demand fetchers open. Google Search and AI Overviews are governed by Googlebot, which falls under the * group and stays allowed.

How the file is built

Each blocked AI crawler gets its own group with Disallow: /. Every other crawler, including Googlebot and Bingbot, falls through to the User-agent: * group, which carries the private paths you list. Crawlers use the most specific group that names them, so a bot with its own group ignores the * rules. The file ends with a Sitemap line so crawlers can find your URLs.

Mistakes that hide sites from AI

Blocking CSS and JavaScript folders, which stops crawlers rendering the page as people see it.
Copying a 2023 robots.txt that blocked every AI user agent, including the search crawlers added since.
A security plugin or CDN bot setting that blocks AI crawlers the file allows.

After you publish the file, run the AI crawler checker to confirm each bot reads it the way you intended.

Questions

Where do I put robots.txt?: At the root of the host it controls, so it loads at https://yoursite.com/robots.txt. A robots.txt on www.example.com does not apply to shop.example.com; each subdomain needs its own.
Should I block GPTBot?: Block GPTBot if you do not want OpenAI to train models on your pages. It does not remove you from ChatGPT search, which uses OAI-SearchBot. Most businesses that want to be recommended by AI assistants keep the search crawlers allowed.
Does robots.txt stop a page appearing in Google?: No. robots.txt controls crawling, not indexing. Google can still index a blocked URL if other pages link to it, without reading its content. To keep a page out of results, allow crawling and add a noindex meta tag instead.
Do all crawlers obey robots.txt?: Reputable search and AI crawlers say they follow it. It is a request, not access control, so use authentication or firewall rules for anything that must stay private.

More free tools

llms.txt generator

Reads your sitemap and page titles and drafts an llms.txt file you can edit and download.

AI crawler checker

Shows which AI crawlers your robots.txt lets in: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more.

Meta tag checker

Title, meta description, canonical, robots, H1 and Open Graph tags for up to 10 pages, with length checks and duplicates.

Sitemap checker

Finds every sitemap a site declares, counts the URLs, checks lastmod dates and spot-checks that listed URLs return 200.

Redirect checker

Tests http, https, www and non-www versions of a site and shows every redirect hop, so you can see if they all land on one URL.

Schema markup generator

Build JSON-LD for Organization, LocalBusiness, FAQ, Product and Article pages, ready to paste into your site.

Keyword density checker

Word count and the most repeated words and phrases in your text, plus how often a target keyword appears. Runs in your browser.

SERP snippet preview

See how your title and meta description may look in Google on desktop and mobile, and where they get cut off.

Open Graph checker

og:title, og:description, og:image and twitter:card tags for up to 5 pages, with a preview of the shared link card.

H1 checker

The H1 heading of up to 10 pages, flagging pages with none or more than one.

Canonical tag checker

Whether each page's canonical points to itself, is missing, or tells Google to index a different URL.

Hreflang checker

Hreflang tags, language and region codes, x-default and return links between language versions.

Core Web Vitals checker

Real-user LCP, INP and CLS from the Chrome UX Report plus a Lighthouse lab test, on mobile or desktop.