Guide · Visibility and growth

llms.txt and AI search: does your site need it?

llms.txt is a proposed file that tells AI models a site's most important content in a compact form. It is cheap to make and does no harm, but no major AI search has confirmed using it. AI visibility is affected more by robots.txt settings, indexability and content that answers the buyer's questions directly.

Updated

What is llms.txt?

llms.txt is a text file placed in a site's root (for example example.com/llms.txt). Jeremy Howard proposed it in September 2024, and the specification is at llmstxt.org (opens in a new tab). The idea is to give an AI model a compact map of the site, which is easier to read than heavy HTML.

The file is Markdown, and its structure is simple:

  1. A main heading with the name of the site or company. It is the only required part.
  2. A short summary as a quotation.
  3. Free text, for example what the company does.
  4. Lists of links under subheadings, each link with a short description.
  5. An "Optional" section for secondary links the model can skip.

You will often see llms-full.txt mentioned as well, which puts a site's whole content into a single file. It is not part of the llmstxt.org specification but has become an established practice.

Do AI searches use llms.txt?

At the moment, not demonstrably.

  • Google: Google's guidance on AI features says outright that appearing in AI Overviews or AI Mode needs no new machine-readable files or AI-specific text files. Google's John Mueller said in June 2025 that, to his knowledge, no AI system uses llms.txt.
  • OpenAI, Anthropic and Perplexity: none has officially said it reads llms.txt for search or answers.

The file is mostly useful on software documentation sites, which developers' AI tools read. On a company's marketing site, the effect is currently guesswork. For how AI is used in managing a site, see the guide WordPress MCP: your options compared.

1. robots.txt and AI crawlers

Every AI service says which crawlers it uses to visit sites. An important distinction is whether a crawler collects pages for search or for model training. You can set them separately in robots.txt.

ServiceSearch (appearing in answers)Model training
OpenAI (ChatGPT)OAI-SearchBotGPTBot
Anthropic (Claude)Claude-SearchBotClaudeBot
PerplexityPerplexityBotNo training crawler
Google (AI Overviews, AI Mode)GooglebotGoogle-Extended (Gemini models)

If you want to appear in AI answers, you must not block the search crawlers. You can block the training crawlers if you do not want your content used to train models, and according to Google this does not affect visibility in Google Search. The services also have agents that fetch pages at a user's request (for example ChatGPT-User and Claude-User), and robots.txt rules do not always apply to them.

Also check that a content delivery network or firewall is not blocking these crawlers without you noticing.

2. Indexability and the technical foundation

A page can get into Google's AI features if it is indexed and allowed to appear in search results with a snippet. There are no special technical requirements. The same basics apply to other searches too: the most important content as text, internal links in order, a good page experience and structured data that matches the visible text. We check the technical foundation and indexing in an SEO audit.

3. Content that AI can quote

AI quotes a page that answers the question directly. This is called GEO, or generative engine optimisation. In practice it means:

  • the answer first, the reasoning after
  • concrete facts, such as prices, timelines and quantities
  • the same facts in the same words everywhere, including in your Google Business Profile and other services
  • frequently asked questions as their own short answers.

Does your company show up in AI answers?

A free AI visibility audit shows whether you are mentioned or your competitors are.

Book an audit

Should you make llms.txt anyway?

Yes, if it does not take time from something more important. The file does no harm, and if some service starts using it, it is already in place. Two conditions:

  • Generate it automatically. In WordPress, Yoast SEO creates the file (from version 25.3, June 2025), and WordPress.org has dedicated plugins. An automatic file updates as your content does.
  • Do not expect results from it. If AI visibility is the goal, start with robots.txt, indexability and content.

Where to start

  1. Check robots.txt: does it block AI search crawlers?
  2. Check in Search Console that your important pages are indexed.
  3. Ask ChatGPT, Claude and Perplexity the questions you want to appear as the answer to. Are you mentioned, or your competitors?
  4. Write direct answers on your pages to the questions where you do not appear.
  5. Add llms.txt last, generated automatically.

We carry out steps 1–3 in a free AI visibility audit. The report is yours, whatever you decide to do next.

Frequently asked questions

Does Google use llms.txt?

No. According to Google's guidance, AI Overviews and AI Mode do not require new machine-readable files, AI-specific text files or special markup. A page needs to be indexed and eligible to appear in search results with a snippet. Google's John Mueller said in summer 2025 that, to his knowledge, no AI system uses llms.txt.

Do ChatGPT, Claude or Perplexity use llms.txt?

OpenAI, Anthropic and Perplexity have not officially said they use llms.txt in search results or answers. Their access to your site is governed by robots.txt: each service says which crawler it uses to collect pages for search and which for model training. Check those crawlers first.

Is llms.txt harmful?

No. It is an ordinary text file in the site's root, and it does not affect search rankings or how the site works. The only downside is if the file goes stale and says something different from the site. If you make one, generate it automatically, for example with an SEO plugin.

How do you make llms.txt in WordPress?

The easiest way is an SEO plugin. Yoast SEO added llms.txt generation in summer 2025, and WordPress.org also has dedicated plugins. The plugin updates the file as your content changes, so it does not go stale. A hand-made file is easy to forget to update.

What should you do before llms.txt?

Check that robots.txt does not block AI search crawlers, that your pages are indexed and that your most important content is text rather than hidden behind images or scripts. Write direct answers to buyers' questions, prices and consistent facts on your pages. These demonstrably affect AI visibility, llms.txt does not.

Summary

llms.txt is a harmless and cheap addition, but no major AI search has confirmed using it. AI visibility is built on robots.txt settings, indexability and content that answers the buyer's questions directly. Do those first, and llms.txt only after.

If you want to know whether your company shows up in AI answers, book a free audit.

Sources

Checked 16 November 2026.

Share this page
  • Glossary

    GEO (generative engine optimisation)

    GEO, or generative engine optimisation, means the work that gets your company mentioned in the answers of AI assistants. It rests on clear answers, consistent facts and structured content. GEO does not replace SEO but complements it, because both need content that can be quoted as it stands.

  • Glossary

    llms.txt

    llms.txt is a Markdown file placed in a site's root that lists the site's most important content for AI models in a compact form. Jeremy Howard proposed it in 2024. The file is harmless and cheap to make, but Google does not use it, and no major AI search has confirmed using it in its results.

  • Glossary

    robots.txt

    robots.txt is a file in a site's root that tells the robots of search engines and AI services which pages they may crawl. It can, for example, allow ChatGPT's search robot but block the robot that collects content for model training. It is not a security measure, because robots follow it voluntarily.

Want help with your decision?

Tell us your situation. We will get back to you by the next working day at the latest.

Get a priceTell us what you need

Get a price estimate or tell us what you need

The estimate takes a few minutes and does not ask for your contact details. You can also describe your need. We will get back to you by the next working day at the latest.

Get a priceTell us what you need