AI search
What is llms.txt, and does your website need one?
An llms.txt file is a proposed guide to useful pages for AI tools. Learn when to publish one, what to include and which search problems it can actually address.

What the file is for
An llms.txt file is a short, public Markdown guide that directs a reader or AI agent to useful parts of a website. Jeremy Howard proposed the format in 2024. A business can use it as an edited reading list. For example, a software company might point to its current API documentation, its migration guide and its subscription limits, with a sentence explaining who needs each one.
The strongest reason to create the file is a navigation problem you can describe. An agent helping a customer integrate your product might have to choose between three versions of your documentation. A maintained list can identify which version supports new integrations and where developers should look up older accounts. A small five-page business site with a clear navigation menu may have little to add.
| File | What it does | A useful check |
|---|---|---|
| llms.txt | Curates links and context for people or agents using the proposal. | Can a reader find the right guide quickly? |
| robots.txt | Communicates crawl rules to crawlers that follow them. | Can the desired crawler request the public content? |
| XML sitemap | Lists URLs for search engine discovery. | Do listed URLs resolve to the canonical pages? |
Write a file that earns its upkeep
Start with a heading naming the site and a short description. Group a handful of canonical links by purpose. Explain what each destination contains in language that still makes sense when read alone. The proposal specifies a project heading and describes conventions for additional sections. You can start small.
Here is an excerpt for a fictional documentation site. The reserved example domain keeps these sample links separate from real destinations. For a service business, useful categories might be services, work, guides and contact. If the booking rules already live on a service page, link there instead of copying a price or availability window that will need to be updated twice.
# Example Documentation
> Setup and support guides for Example customers.
## Guides
- [Current setup guide](https://example.com/setup): installation steps for new accounts.
- [Migration guide](https://example.com/migrate): steps for customers upgrading an existing account.Test the links and decide what to measure
Open the published file at the root of the site and follow every listed URL. Check the final destination after redirects, the public visibility of the page and whether the description matches the current content. Add this check to the same release process that handles a renamed guide or retired service. If nobody owns the list, its most important link can silently become the least reliable one.
Google does not require an AI text file for AI Overviews or AI Mode. If the actual problem is a blocked page, a wrong canonical or an expired booking link, fix it where it occurs. A useful outcome for llms.txt is that an agent using the file can reach the right manual. If you also want to measure brand recommendations, run a separate, repeatable prompt check.
See how our AI search visibility check tests crawler access and answers
Frequently asked questions
Can I use llms.txt to keep private pages confidential?
Keep private material behind access controls. Anyone can read a public llms.txt file, so include only links and information intended for public use.
Does publishing llms.txt increase AI citations?
There is no guaranteed increase. A file can help systems that choose to use it navigate the site. Measure any specific citation in the answer where it appears.

