llms.txt is a short Markdown file that some websites publish to give AI agents a summary of the business and a list of its most useful pages. Before you add one, or pay anyone to write it, it helps to know who actually reads it, so we checked 30 days of our own server logs.
What llms.txt is, and what our server logs show about who reads it
The idea came from Jeremy Howard of Answer.AI in September 2024, and the llms.txt proposal is now on version 2, updated in August 2026. The file sits at the root of your site (yoursite.co.uk/llms.txt), opens with your business name as a heading, adds a one-line summary, then lists links to your key pages with a short note on each. Only the heading is required, and it is plain Markdown that people and machines can both read.
The proposal itself is modest. Its author expected the file to be read on demand by agents helping someone, rather than used for training, and notes it is used most heavily for software documentation. Nothing in it promises that ChatGPT, Perplexity or Google will read your file.
Our own site publishes an llms.txt file, generated by the Rank Math SEO plugin with a hand-written introduction. For the 30 days from 4 September to 4 October 2026 we counted every request for it in our raw server logs, then compared that with how often the main AI and search crawlers visited the rest of the site.
| Who made the request | Requests for /llms.txt | All requests to the site |
|---|---|---|
| OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User) | 0 | 4,372 |
| Amazon (Amazonbot) | 0 | 2,187 |
| Google (Googlebot) | 0 | 2,084 |
| Anthropic (ClaudeBot, Claude-User) | 0 | 1,128 |
| Apple (Applebot) | 0 | 1,108 |
| Perplexity (PerplexityBot) | 0 | 633 |
| SEO and site-audit tools (SEOJuice, Semrush, RSiteAuditor) | 37 | Not counted |
| Other data crawlers and ordinary browsers | 8 | Not counted |
So the file was fetched 45 times in a month, and not once by a crawler or agent belonging to an AI company. Those six companies' bots made more than 11,500 requests to the rest of the site and never opened the file written for them. Most requests came from SEO tools checking whether the file exists, which is why llms.txt so often appears as a "missing" item in audit reports.
One caveat: our site sits behind Cloudflare, which can serve cached copies of pages without the request reaching our server, so the page counts above are a minimum. The llms.txt file is never cached (its response header reads DYNAMIC), so every request for it does reach our logs. A month on one site is not proof about the whole web, but it matches Google's own guidance on AI features in Search: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."
The irony is that OpenAI and Perplexity both publish llms.txt files for their own developer documentation, and the proposal says Anthropic and Gemini do too. AI companies publish the file for coding tools. Their crawlers did not read ours.
Six things to know before you add an llms.txt file
It is a proposal, not a standard
No standards body sits behind llms.txt. robots.txt, by contrast, is an internet standard, RFC 9309, and it is what every AI company's crawler documentation tells you to use.
It cannot block anything
llms.txt is a reading list, not a permission system. Letting a crawler in or keeping it out happens in robots.txt and in your CDN or firewall settings.
Google Search does not need it
Google says there are no extra requirements and no AI text files needed to appear in AI Overviews or AI Mode. Crawlable, indexable, useful pages are what count.
AI crawlers ignored ours
In 30 days, none of the requests for our llms.txt came from OpenAI, Anthropic, Perplexity, Google, Apple or Amazon, while their bots made over 11,500 requests to the rest of our site.
Coding agents are the real audience
The format works best for software documentation, where developer tools fetch it to find API references. If you run a SaaS product or publish technical docs, it is useful.
A broken file is worse than none
Chrome's Lighthouse now checks llms.txt in its agentic browsing audit. A missing file is marked not applicable, but a server error on the file is flagged.
What actually controls AI crawlers: robots.txt rules by user agent
If your real question is "how do I control what AI does with my site", the answer is still robots.txt, the plain-text file that tells crawlers which parts of a site they may visit. Getting your AI crawler robots.txt rules right matters far more than llms.txt. What has changed is that each AI company now runs separate bots for separate jobs: one collects content for training, one indexes pages for search answers, and one opens a page in real time because a user asked about it. You can treat each job differently.
| Company | Training | Search answers | Live user requests |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None listed (PerplexityBot is not used for model training) | PerplexityBot | Perplexity-User |
| Google-Extended (a control token, not a separate crawler) | Googlebot, which also covers AI Overviews and AI Mode | Not covered here |
That table is built from the OpenAI crawler documentation, Anthropic's crawler policy, Perplexity's crawler page and Google's list of common crawlers. Three details in those pages matter for a small business:
- Blocking training does not remove you from AI search. OpenAI says each setting is independent, so a GPTBot robots.txt rule only covers training, while sites that block OAI-SearchBot "will not be shown in ChatGPT search answers". Google says Google-Extended (which governs Gemini training and grounding in the Gemini apps) "does not impact a site's inclusion in Google Search".
- Live user fetchers may not follow robots.txt. OpenAI says robots.txt rules "may not apply" to ChatGPT-User because a person triggered the visit, and Perplexity says Perplexity-User "generally ignores robots.txt rules". Anthropic says its bots honour robots.txt, and that blocking Claude-User stops Claude retrieving your pages for user questions.
- Changes take about a day. OpenAI quotes around 24 hours for its search to reflect a robots.txt change, Perplexity up to 24 hours, and the standard says crawlers should not rely on a cached copy for longer than 24 hours.
A robots.txt that blocks training but keeps you in AI search
For most UK small businesses the sensible default is to stay visible in AI answers and make a deliberate choice about training. If you decide to opt out of training, this is the shape of it on a WordPress site:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /The search crawlers and live fetchers have no group of their own here, so they fall through to the general rules and can reach everything except the admin area.
The mistake we see most: a named group cancels your general rules
Under RFC 9309, a crawler that finds a group naming it obeys that group and ignores the general * group entirely. So a well-meant "User-agent: OAI-SearchBot, Allow: /" block quietly exempts that bot from every Disallow line written under *. Our own robots.txt names 17 AI bots this way, which is harmless only because our * group covers nothing but the login-protected admin area. If your general group protects anything that matters, such as staging folders, internal search results or filtered category URLs, copy those lines into every named group.
Check your CDN as well as robots.txt
robots.txt is a request, not a lock: the standard says its rules "are not a form of access authorization". A hard block needs your firewall or CDN, and the reverse also applies, because a CDN can block AI bots that robots.txt welcomes. Cloudflare's AI bot documentation says that from 15 September 2026, new domains block bots classified as Training or Agent on pages that display ads by default, while Search stays allowed. Agent means "chat fetch bots" acting for a person, which describes ChatGPT-User and Claude-User. If your site runs ads and moved to Cloudflare recently, check Security Settings, then Configure AI bot policies. The crawler access section of our AI search optimisation playbook covers the rest of the "let them in" checklist.
How to check your own logs for AI crawlers and llms.txt requests
You do not need a tool subscription to repeat our test. Most UK hosts let you download raw access logs from the control panel, usually one compressed file per day. Two commands give you the picture (on Windows, run them in WSL):
zgrep -c "GET /llms.txt" *.gz
zgrep -ohE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User" *.gz | sort | uniq -c- Count requests for llms.txt and note who made them. If the only visitors are SEO tools, the file is being audited, not used.
- Count each AI bot by name. If a search crawler you rely on never appears, check robots.txt and your CDN first.
- Look at the live fetchers. ChatGPT-User and Claude-User hits mean someone asked an AI assistant a question and it opened your page to answer. On our site ChatGPT-User was the busiest AI agent of all, at 3,366 requests in 30 days.
- Verify anything surprising. User-agent strings can be faked. OpenAI, Anthropic and Perplexity publish the IP addresses their bots use, so check a suspicious hit against those lists before acting on it.
- Allow for caching. Pages served from a CDN cache never reach your logs, so treat every count as a minimum.
Crawler visits are not the same as people arriving from AI tools. To measure those, see our guide to tracking AI search traffic in GA4 and your server logs, which separates the two properly.
Should you add llms.txt? A decision guide for UK small businesses
Our view after looking at the evidence: llms.txt is cheap, harmless and occasionally useful, but it is not an AI visibility strategy.
Add one if it costs you nothing
- You run WordPress with Rank Math, Yoast SEO or AIOSEO, all of which can generate the file, or you are on Wix, which creates one for every site.
- You publish software documentation, an API reference or technical help pages that developers and coding agents use.
- You are happy to keep it accurate, which takes a few minutes whenever your services or prices change.
Do not pay for it
If an agency quotes for "llms.txt optimisation" as a route into ChatGPT or AI Overviews, ask to see server logs showing the AI companies' crawlers fetching the file. Ours show none, and Google's documentation says no AI text files are needed. A sensible llms.txt takes under an hour to write.
Spend the time here instead
- Make sure the search crawlers and live fetchers can reach you, in robots.txt and in your CDN.
- Put key facts, prices and answers in the page HTML, not behind scripts, tabs or PDFs.
- Get described accurately on the third-party sites that AI answers already cite for your sector.
- Track crawler visits and AI referrals monthly so you can see what changes.
Our answer engine optimisation audit walks through each of these checks in order. If you would rather hand it over, our answer engine optimisation service covers crawler access, page structure and citation tracking, and our generative engine optimisation service handles the brand and entity work that shapes how ChatGPT and Perplexity describe you. Both build on our SEO services, because the same crawlable, well-linked pages underpin all three.
An llms.txt example: what a good file looks like
If you do publish one, keep it short, factual and written for a reader who knows nothing about you. Here is an example for a fictional UK trade business:
# Hartley Heating
> Gas Safe registered heating engineers covering Taunton and West Somerset. Boiler installation, servicing and repairs for homes and landlords.
We do not cover commercial premises. Emergency call-outs run from 7am to 10pm, seven days a week.
## Services
- [Boiler installation](https://example.co.uk/boiler-installation/): fixed quotes and typical lead times
- [Boiler servicing](https://example.co.uk/boiler-servicing/): annual servicing and landlord gas safety certificates
## Policies
- [Prices and call-out charges](https://example.co.uk/prices/): current rates, VAT included
- [Areas we cover](https://example.co.uk/areas/): postcodes and travel charges
## Optional
- [Blog](https://example.co.uk/blog/): seasonal maintenance advice- The H1 is the only required line. The summary and link lists are optional but make the file useful. The "Optional" section holds links an agent can skip.
- Write facts, not slogans. An agent reading "award-winning and customer-first" learns nothing. Service area, prices, hours and policies are what it needs.
- Keep it in step with the site. An out-of-date price in llms.txt is still an out-of-date price.
- llms-full.txt is a separate extra. Some tools generate a file bundling the full text of your pages. It is not part of the proposal, and a small business site rarely needs one.
FAQ
What is llms.txt?
llms.txt is a plain-text Markdown file at the root of a website, for example yoursite.co.uk/llms.txt, that gives AI agents a short summary of the site and links to its most useful pages. Jeremy Howard of Answer.AI proposed it in September 2024. It is a community convention, not an official internet standard.
Does llms.txt help my site appear in ChatGPT or Google AI Overviews?
There is no evidence that it does. Google's documentation says you do not need AI text files to appear in AI Overviews or AI Mode, and in 30 days of our own server logs no crawler from OpenAI, Anthropic, Perplexity, Google, Apple or Amazon requested our llms.txt file. Being crawlable, clear and accurately described on other sites matters far more.
Should I add llms.txt to my website?
Add one if your platform generates it for free, such as Rank Math, Yoast SEO or AIOSEO on WordPress, or if you publish documentation that coding agents use. It is optional, so do not pay for it as an AI visibility service, and make sure it is accurate and loads without errors.
Is llms.txt the same as robots.txt?
No. robots.txt is an internet standard (RFC 9309) that tells crawlers which pages they may visit. llms.txt is a reading list suggesting which pages are most useful. It cannot allow or block anything, so access decisions still belong in robots.txt and your CDN settings.
How do I block AI crawlers without disappearing from AI search?
Block the training crawlers and leave the search crawlers alone. A Disallow: / group in robots.txt for GPTBot, ClaudeBot and Google-Extended opts you out of model training, while OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot stay allowed, so you can still appear in ChatGPT search, Claude, Perplexity and Google's AI features.
How do I check whether my llms.txt file is working?
Open yoursite.co.uk/llms.txt in a browser to confirm it loads as plain text, then run a Lighthouse report in Chrome, which flags a server error on the file. To see who actually reads it, search your server access logs for requests to /llms.txt and look at the user agents making them.
