LLMs TXT Explained: Who Reads llms.txt and Is It Worth It?

Give an llms txt file a 10-minute budget if you have a concrete use for it, such as helping an agent navigate your documentation. For a small business hoping for more AI citations, it is optional housekeeping: Google Search explicitly says it does not use llms.txt, while the crawler policies checked for OpenAI, Anthropic and Perplexity make no promise to consume it.

Verified against the proposal and the four companies' live documentation on 14 October 2026. This is a comparison of documented behavior, not a measurement of their traffic to your site. The Google Search guidance is explicit; the other companies' narrower statements are set out below.

For a local-services owner, fixing an unreadable service page deserves attention before adding this file. For a software founder whose customers use coding agents, a maintained documentation index has a direct use. That distinction should decide the work.

What Is an llms txt File?

llms.txt is a proposed Markdown guide to a website for AI agents. Its name refers to large language models, the AI models behind many assistants. Agents use models and tools to carry out tasks. Markdown is plain text with simple marks for headings and links. Think of the file as a librarian's reading list: a short introduction and selected routes into useful material, rather than every book in the building.

The llmstxt.org proposal specifies this order:

• A site or project name as an H1 heading, written with #. This is the only required section.
• An optional short summary introduced with >.
• Optional explanatory text without additional headings.
• H2 sections, written with ##, containing lists of links. Each item uses [Page name](URL) and can add a colon and description.

An Optional section can hold secondary links. The file supplies context and navigation; robots.txt expresses crawling preferences, while a sitemap lists URLs for discovery. Adding llms.txt does not replace either.

The current proposal allows more than a root file

The v2 revision, dated August 2026, clarifies that /docs/llms.txt covers pages below /docs/, with the most specific file taking precedence. It also recommends discovery links: rel="describedby" points to the relevant index, and rel="alternate" with type="text/markdown" points to a page's Markdown version.

A developer can supply those relations in HTML or HTTP response headers. They give a willing agent an explicit route. They do not compel an AI service to follow it. In v2, Optional remains a useful convention, without the old context-expansion tooling's special processing meaning.

Do AI Companies Read It? What Their Own Pages Say

Hosting an llms.txt file and using other websites' llms.txt files are different claims. All four companies below publish developer-documentation indexes. Their public crawler instructions do not establish universal consumption of your file.

The linked first-party pages below support each row. Silence means not documented in the page checked. It does not prove that no employee, product, experiment or agent ever requests the file.

OpenAI llms txt: What the Crawler Policy Establishes

OpenAI names OAI-SearchBot for search, GPTBot for potential training collection and ChatGPT-User for certain user-directed visits. Its crawler documentation explains access controls and publishes IP ranges, but makes no commitment to read your llms.txt.

The same documentation links to OpenAI's own index. That is useful evidence of documentation publishing. It is insufficient evidence for a claim that adding the file earns ChatGPT citations. A founder should ask an agency to identify the documented mechanism behind that claim.

Anthropic llms txt: Documentation and Crawling

Anthropic distinguishes ClaudeBot, Claude-SearchBot and Claude-User in its crawler help page. Their roles are potential training collection, search and user-directed retrieval. That page contains no llms.txt guidance.

Anthropic does publish a developer-documentation index, and the Claude Code documentation points readers to its own index. Claude Code is an AI coding assistant. This supports using an index as a route into documentation; it does not establish that Claude's web crawler checks every business site's file.

Google: Search and Lighthouse Answer Different Questions

Google Search's AI optimization guide explicitly says Search does not use llms.txt, including for its generative AI capabilities. It may still crawl a text file as ordinary content. Finding a Googlebot request therefore would not overturn Google's statement about special treatment.

Google's Chrome team separately documents an optional Lighthouse agentic-browsing audit. Lighthouse is a website auditing tool. A missing file returning 404 is marked not applicable; a server error while retrieving it is flagged. That is an agent-accessibility check, not a Search ranking requirement.

Google also publishes a Gemini API documentation index. A marketer can accept all these facts together: an index may help a particular agent navigate docs without being a Google Search optimization.

Perplexity: Its Docs Link Is Not a Crawler Commitment

Perplexity describes PerplexityBot as its search crawler and Perplexity-User as a user-directed fetcher. Its crawler page recommends access configuration and IP verification. It does not state that either agent automatically discovers or uses your llms.txt.

The page does point readers to Perplexity's documentation index. A docs navigation instruction should not be sold to a small-business owner as evidence of a search inclusion program.

Where It Helps Today: Documentation Someone Asks an Agent to Read

The clearest use is a task where you deliberately give an agent the index. A software founder can tell a coding assistant to read the product's llms.txt, locate the current authentication guide and use that source while implementing an integration. The intended consumer and task are explicit.

For an agency, the equivalent is giving an assistant a client's public service and policy index before drafting a proposal. The payoff is a reusable starting point. Whether it saves time depends on the quality of the linked pages and the agent's ability to fetch them; no traffic improvement is implied.

Use an instruction such as: “Start from this documentation index, open the pages relevant to authentication, and cite the exact source pages for the requirements you use.” Check that the answer draws from those pages. An index alone does not contain their detail.

Documentation Platforms Can Maintain the File

Mintlify, a documentation hosting platform, automatically serves llms.txt with links to Markdown pages. It also generates llms-full.txt. Fully authenticated sites require authentication for those files, so publishing on the platform does not make restricted docs publicly readable.

For a SaaS growth lead, check what already exists before commissioning a generator. The useful work may be improving page descriptions and removing stale guidance from the source docs.

GitBook, another documentation platform, automatically creates llms.txt for published docs. That establishes availability, not a measured improvement in citations. An owner still needs to check the generated file and its destinations.

llms-full txt: The Index and the Contents Have Different Jobs

llms-full.txt commonly bundles the documentation itself; Mintlify's implementation combines page content into one file. The smaller index lets an agent select relevant pages. The full export can be useful when you deliberately want a corpus, but sending everything can consume unnecessary context, the text an AI system can consider at once.

For a small service business, start with an index. There is no reason to turn the entire website into an extra document unless a named workflow needs it.

How to Use llms-txt: A Ten-Minute Small-Business Template

Treat ten minutes as a budget, not an implementation guarantee. This exercise assumes you can already publish a public text file. If your website platform makes that difficult, stop and ask the site maintainer; do not let optional housekeeping become a rebuild.

1. 1

   Choose the useful pages: two minutes

   Pick pages that answer customer questions: services, service area, booking, contact details and relevant policies. Use the pages you maintain, not a list of every post. Confirm that your platform has not already generated an index.

2. 2

   Adapt the template: four minutes

   The business below is fictional. Replace its name, description, domain and paths with your own verified information. Copy each displayed line into a plain-text file named llms.txt, preserving the blank lines between sections.

# Northline Plumbing

> Residential plumbing services in our published service area.

Use the linked pages for current services, availability and policies.

## Services

- [Services](https://example.com/services): Plumbing work we offer.
- [Service area](https://example.com/service-area): Locations we serve.

## Booking and Support

- [Book a visit](https://example.com/booking): How to request an appointment.
- [Contact](https://example.com/contact): Contact details and opening hours.
- [Policies](https://example.com/policies): Booking and cancellation terms.

## Optional

- [About](https://example.com/about): Company background.

These are example URLs, not working destinations to retain. Link to clean Markdown versions when they exist. Otherwise use your actual readable public pages; appending .md does not create a page.

1. 1

   Publish and inspect: two minutes

   Serve the file at your chosen path, normally /llms.txt for a small site. Open the URL while signed out. Confirm that it returns the text, rather than the homepage, an HTML error or a login screen. Ask the maintainer to confirm HTTP 200 and an appropriate plain-text response type.

2. 2

   Check links and assign maintenance: two minutes

   Open every destination and check that each description matches it. Put the index on the same maintenance checklist as your service pages. When a page moves or a policy changes, update the source and any affected index description together.

Check Server Logs for Fetches, Then Check the Identity

A server log can establish that a request arrived. It cannot establish that an AI answer used the content. A log is the server's record of requests; ordinary browser analytics may never record these visits.

Ask your host or developer for access logs at the content delivery network, or CDN, if it serves the file from cache. Origin-server logs alone can miss requests handled there. Keep timestamp, hostname, request path, query string, method, status, user-agent and client IP. A user-agent is the client's self-reported software name.

Start with a suggested 30-day observation window. That is a measurement choice, not an expected crawler timetable.

Filter the File Path Before Filtering the Bot

First select the exact normalized path /llms.txt, or the subpath you deployed. Preserve the original request URL so query strings remain visible. Include every user-agent in this first pass: a coding agent may use a browser-like identifier or a general HTTP client.

For downloaded Nginx or Apache logs in a conventional combined format, a developer with ripgrep installed can start with this command:

rg '"(GET|HEAD) /llms\.txt(\?[^ ]*)? HTTP/' access.log

That command assumes the request line appears in quotes. For JSON logs, filter the parsed path and method fields instead. Change the path for /docs/llms.txt; check /llms-full.txt separately if you serve it.

Then narrow the matching requests to the published identifiers:

These tokens come from the company pages cited above and Google's crawler reference. Match names without pinning a browser or bot version.

For the same combined log format, the narrower filter is:

rg '"(GET|HEAD) /llms\.txt(\?[^ ]*)? HTTP/' access.log | rg -i 'OAI-SearchBot|GPTBot|ChatGPT-User|Claude-SearchBot|ClaudeBot|Claude-User|Googlebot|PerplexityBot|Perplexity-User'

Google-Extended is a product-control token used in robots.txt, not a separate identifier to expect in an HTTP log. Counting only that string would miss Google's actual requests.

Verify Who Sent It and What Came Back

A bot name can be imitated. Check client IPs against the current lists linked from the OpenAI crawler page, Anthropic's published list, Perplexity crawler page and Google's crawler reference. Record unverified identities as claims, not confirmed company traffic.

Separate the outcomes:

• A GET with 200 is a successful response. Inspect that the response really is your file, rather than an error page served with a success status.
• A HEAD request checks response metadata without downloading the body.
• A 304 is cache validation; it is not a fresh body transfer.
• A 404 means the requested resource was not found. A server error is a failed retrieval, not successful ingestion.

These distinctions follow HTTP semantics. Keep your own tests and monitoring requests separate. If you explicitly ask an assistant to fetch the file, label that observation as user-triggered; it does not demonstrate automatic discovery.

Report the Observation Without Inflating It

A defensible report says which verified agents requested the file, when, with which outcomes, and whether relevant page requests followed. Even matching requests close together do not establish that the index caused the later visit.

To measure answer visibility, inspect answers separately, recording the question, product, date and cited URL. Our AI visibility tools comparison covers that different measurement job. A before-and-after change in mentions is not, by itself, proof that llms.txt caused it.

No matching requests means none appeared in the logs, paths and period observed. It cannot rule out other identifiers, incomplete logs, caches or future visits. For a small-business owner, it is still a practical reason to avoid spending more time without a concrete use case.

LLM TXT for SEO: What Is Overhyped?

A generated file is not evidence of a ranking or citation benefit. A validator can check formatting. A platform can host an index. A crawler can request it. None of those observations establishes that an answer engine will recommend your business.

Three claims deserve pushback: that the file is mandatory for AI visibility, that all major AI crawlers automatically read it, and that adding it guarantees citations. The first-party pages checked here do not support those claims. Google's Search guidance directly rejects the special-file premise for its own Search features.

Do not pay for an “AI-ready” badge as a substitute for evidence of useful work. Ask what was made readable, which source pages improved, which requests were verified and which answers changed. Our GEO vs SEO guide puts the broader distribution work in context: GEO concerns visibility in generated answers, while SEO concerns search visibility. This file is a small implementation choice within that work.

Who Should Act, Wait, or Skip It?

Builders with developer docs should act when an agent workflow can use the index. Keep it generated from current sources where possible, expose useful Markdown pages and give the agent the starting URL. Check a representative integration question against the linked source.

Operators of small business sites can run the bounded experiment. Use the template if publishing is straightforward, then observe access. If the service page is unclear, your contact information is wrong or your site blocks legitimate retrieval, fix that first.

Buyers commissioning an agency should require a concrete deliverable. A maintained index, valid links and a log report are reviewable. A promise of higher AI rankings from the filename is not.

Your Monday move: open your site's llms.txt URL. If it exists, inspect it before adding anything. If it does not, decide whether a named workflow justifies the ten-minute exercise. Give the rest of the hour to a customer question your website still answers badly.

Questions About the File and Its Name

How to create an llms.txt file for your website?

Use the template above, replace the example details, publish the plain-text file at the path you control, and check the response and every link. Your docs platform may already generate it.

What are the top 5 LLM models?

There is no fixed top five for this task. llms.txt describes website content and does not require choosing a model leaderboard.

What is .txt extension?

It conventionally identifies a plain-text file. The content of llms.txt uses Markdown headings and links within that text.

What does LLM mean in text?

In this technical context, LLM means large language model, an AI model that works with language. The filename refers to those models.

What is LLM in gen z slang?

An isolated acronym in a social message needs context. The website filename discussed here uses the technical AI meaning.

What does "LLM" mean in urban slang?

A slang interpretation cannot be established from the acronym alone. It is not needed to understand this website file.

What does LLM stand for on Snapchat?

The platform alone does not determine a message's meaning; read the surrounding conversation. In a discussion of this file, it refers to large language models.

What is LLM in social media?

In posts about AI, it commonly refers to a large language model. An unrelated post may require a different interpretation from context.

What does LLM stand for on TikTok?

Check the surrounding post rather than assigning a universal platform-specific expansion. For llms.txt, the intended meaning is large language models.

For practical notes on where AI changes business work, join the newsletter.