What Is llms.txt? How to Make Your Site Visible to AI Crawlers

AI crawlers are reading your site right now. GPTBot and OAI-SearchBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot, and others are fetching your pages to build the knowledge bases that power AI-generated answers. But unlike traditional search crawlers, these AI systems have no standardized way to understand what your site is about, what content matters most, or how it is organized.

That is the gap llms.txt tries to fill. It is a machine-readable file at the root of your site that summarizes what your site offers and where to find it, like a robots.txt for discovery instead of access control.

What Is llms.txt?

llms.txt is a plain-text file hosted at yoursite.com/llms.txt that provides AI systems with a structured summary of your site. It includes your site's purpose, key content areas, important pages, and any special instructions for AI consumption.

Think of it as a cover letter for AI crawlers. While robots.txt says "here is what you can and cannot access," llms.txt says "here is what we are, what we offer, and where to find our best content."

Jeremy Howard of Answer.AI proposed the format in September 2024 at llmstxt.org. It is not a formal standard. Documentation sites adopted it first, because AI coding assistants read docs on request. Google's AI features guidance says you do not need special AI text files to appear in AI Overviews or AI Mode, and no AI search engine has said it reads the file when choosing which pages to cite, so treat it as cheap documentation for AI tools rather than a ranking factor. You can generate one from your sitemap in under a minute.

Why llms.txt Matters for GEO

Generative Engine Optimization (GEO) is the practice of making your content citable by AI systems. llms.txt is aimed at the discovery part of that problem, though no AI search engine has confirmed it uses the file.

Without llms.txt, an AI crawler lands on your site and has to figure out what you do by reading individual pages. It might index your about page, a random blog post, and your pricing page, but miss your most authoritative content entirely.

With llms.txt, any AI tool that reads the file gets the answer directly: "We are a developer tools company. Our most important content is our API documentation, our technical blog, and our developer guides. Here are the URLs."

This matters because AI systems use the content they index to generate answers. If they index the right content, they cite you accurately. If they index random pages, they might summarize you incorrectly, or not cite you at all.

How AI Crawlers Discover Content

Understanding the AI crawling ecosystem helps you configure your site correctly.

Known AI Crawlers

How They Work

1. Crawling. The bot fetches your pages, starting from your sitemap and following internal links. The major AI crawlers say they follow robots.txt directives.
2. Parsing. The bot extracts text, headings, structured data, and metadata from the HTML. Most AI crawlers do not render JavaScript.
3. Indexing. The extracted content is stored in an index used by the AI system.
4. Retrieval. When a user asks a question, the AI system searches its index for relevant content and synthesizes an answer.

llms.txt is meant to help at step 1 by pointing to your best pages. No AI search crawler has said it uses the file to set priorities, so your sitemap and internal links still do that job.

llms.txt Format and Structure

The llms.txt format is flexible but follows a consistent structure. Here is the basic shape described at llmstxt.org:

# Site Name

> Brief description of what this site is about.

## Key Content Areas

- [Area Name](https://yoursite.com/path): Description of this content area
- [Another Area](https://yoursite.com/other): What this section covers

## Important Pages

- [Page Title](https://yoursite.com/page): Why this page matters
- [Guide Title](https://yoursite.com/guide): What the reader learns

## Optional: Additional Context

Any additional context about how to interpret or cite your content.

Format Rules

• Start with H1 (#): your site name
• Blockquote (>): a one-line site description
• H2 sections (##): content categories
• Markdown links: each important page with a description
• Plain text: any additional context
• An ## Optional section, in the llmstxt.org proposal, holds links a tool can skip when it needs a shorter context
• Keep it concise, under about 100 lines. The point is a short, curated list, not a copy of your sitemap.

Step-by-Step: Creating Your Own llms.txt

Step 1: Identify Your Key Content

List the 10-20 most important pages on your site. Prioritize:

• Pages that define what your product/service does
• Your most authoritative blog posts or guides
• Documentation and API references
• Pricing and comparison pages
• Any page you want AI systems to cite

Step 2: Write the File

Create a file at public/llms.txt (or your static file directory):

# Your Company Name

> One sentence describing what your company does and who it serves.

## Documentation

- [Getting Started](https://yoursite.com/docs/getting-started): Quick start guide for new users
- [API Reference](https://yoursite.com/docs/api): Complete API documentation with examples
- [CLI Guide](https://yoursite.com/docs/cli): Command-line tool usage and configuration

## Blog

- [Best Post Title](https://yoursite.com/blog/best-post): Description of the topic and key insights
- [Another Key Post](https://yoursite.com/blog/key-post): What this post covers

## Product

- [Features](https://yoursite.com/features): Overview of product capabilities
- [Pricing](https://yoursite.com/pricing): Plans and pricing information
- [Changelog](https://yoursite.com/changelog): Recent updates and new features

Step 3: Deploy and Verify

Deploy your site and verify the file is accessible:

curl -s https://yoursite.com/llms.txt | head -20

You should see your llms.txt content. If you get a 404, check your static file configuration.

Step 4: Add to Your Sitemap

This step is optional. Tools that support llms.txt look for it at the root URL, so a sitemap entry is not needed, and there is no evidence it helps. If you want to list it anyway:

<url>
  <loc>https://yoursite.com/llms.txt</loc>
  <changefreq>weekly</changefreq>
</url>

XeoRank's llms.txt as a Working Example

XeoRank has a live llms.txt at xeorank.com/llms.txt. Here is a trimmed excerpt:

# XeoRank

> XeoRank is an AI SEO and AI visibility platform for agencies, consultants and growing businesses. It checks whether ChatGPT, Gemini (grounded in Google Search) and Perplexity mention and cite a brand, audits the site for technical SEO problems, researches and tracks keywords with Google Search Console and GA4 data, and lets an AI assistant such as Claude, ChatGPT, Codex or Cursor prepare fixes through the XeoRank MCP server.

## Product

- [AI visibility](https://xeorank.com/ai-visibility): asks ChatGPT, Gemini (grounded in Google Search) and Perplexity your buyer questions and shows the results per engine.
- [Free site analysis](https://xeorank.com/analyze): run an SEO, AEO and GEO check on any URL.
- [MCP setup](https://xeorank.com/docs/mcp): connect XeoRank to Claude, ChatGPT, Codex, Cursor or VS Code at https://mcp.xeorank.com/mcp.

## Free tools

- [llms.txt generator](https://xeorank.com/tools/llms-txt-generator): free tool. Reads your sitemap and page titles and drafts an llms.txt file you can edit and download.
- [AI crawler checker](https://xeorank.com/tools/ai-crawler-checker): free tool. Shows which AI crawlers your robots.txt lets in.

## Blog

- [The Complete Guide to Answer Engine Optimization (AEO)](https://xeorank.com/blog/complete-guide-to-aeo): ...

Notice the structure: clear site description, categorized content areas, and descriptive link text. Each entry tells an AI crawler not just where to go, but what it will find there.

robots.txt Configuration for AI Crawlers

llms.txt handles discovery. robots.txt handles access control. You need both configured correctly.

Allowing AI Crawlers

If you want AI systems to index your content (recommended for GEO), explicitly allow their user agents:

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /
Disallow: /dashboard/
Disallow: /api/
Disallow: /admin/

Sitemap: https://yoursite.com/sitemap.xml

Blocking Specific AI Crawlers

If you want to block specific AI crawlers (for example, to prevent training but allow search):

# Allow search and user-fetch AI crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Block training-only crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

Selective Access

You can allow AI crawlers to access specific sections while blocking others:

User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /

This lets GPTBot collect your public content (blog, docs) but not your product pages or internal content.

Frequently Asked Questions

Is llms.txt an official standard?

No. It is a convention proposed in September 2024 at llmstxt.org. Documentation platforms and many developer tools publish one, but Google's guidance says no special AI text files are needed to appear in its AI features, and the AI search engines have not said they rely on it. Treat it as low-cost documentation for AI tools, not a ranking signal.

Does llms.txt replace robots.txt?

No. They serve different purposes. robots.txt controls access: what crawlers can and cannot fetch. llms.txt controls discovery: what crawlers should prioritize. Use both together.

How often should I update llms.txt?

Update it whenever you add or remove significant content: new product pages, major blog posts, or documentation sections. A monthly review is a good default. Keep it short: the point is a curated list, not a copy of your sitemap.

Getting Started

Creating an llms.txt file takes a few minutes and costs nothing. It will not move rankings on its own; what moves AI answers is crawlable pages, specific facts and being named on the pages the engines already cite. Publish the file, then spend your time there.

1. List your 10-20 most important pages
2. Write your llms.txt following the format above
3. Deploy it at your site root
4. Configure robots.txt for the AI crawlers you want to allow, and confirm it with the AI crawler checker

Then verify your AI visibility with XeoRank to see how AI engines currently perceive your brand, and track improvements over time.