What is llms.txt and should a small site use it? llms.txt is a plain-text file written in Markdown that sits at the root of your domain (for example example.com/llms.txt) and gives AI tools a short, hand-picked map of your best pages. For a small site it is optional: it is quick to make and harmless, but no major search engine has confirmed that it uses the file to rank pages or to build AI answers.
That is the short version. Below I explain where the file came from, what goes in it, how it differs from the files you already have, and when it is worth the twenty minutes. If you are new to technical SEO, I define each term as we go.
Where did llms.txt come from?
llms.txt is a proposal, not an official web standard. Jeremy Howard of Answer.AI published it on 3 September 2024 at llmstxt.org. "Proposal" means anyone can follow it, but no browser, search engine or standards body has to support it.
The idea behind it is simple. A large language model (LLM) is the type of AI behind tools like ChatGPT, Claude and Gemini. When one of these tools fetches a web page, it has to dig through menus, ads, cookie banners and scripts to find the actual text. Its context window, which is the amount of text it can look at in one go, is limited. llms.txt tries to help by handing the AI a clean summary plus a list of links to the pages that matter most.
The proposal also suggests offering a Markdown copy of important pages at the same address with .md added to the end, so an AI can read the content without the HTML around it. That part is extra work, and most small sites skip it.
What goes inside an llms.txt file?
The format has a few parts, always in this order:
- An H1 title. One line that starts with
#followed by the name of your site. This is the only required part. - A short summary in a blockquote, which is a line that starts with
>. One or two sentences on what the site is about. - Optional notes. A paragraph or two of plain text with anything an AI should know, such as how often lists are checked.
- Sections with H2 headings (lines that start with
##). Each section holds a list of links written as[Page name](URL): short note. - An "Optional" section. Links an AI can skip when it is short on space.
Here is an example of what the file could look like for a list-based SEO site like this one. Each item below is one line of the file:
# Bloghints> Curated lists of web directories, submission sites and SEO tools for webmasters and small businesses.## Lists- [Free directory list](https://bloghints.com/free-directory-list.php): free general web directories, checked by hand- [Local business directories](https://bloghints.com/local-business-directory.php): citation sites for local SEO## Guides- [What is a web directory](https://bloghints.com/basic/what-is-a-web-directory-guide.php): beginner explainer## Optional- [Webmaster lists hub](https://bloghints.com/webmaster-lists.php): index of every list on the site
Notice what is missing. There are no rules, no "allow" and no "disallow". llms.txt is a reading list, not a set of permissions.
llms.txt vs. robots.txt vs. sitemap.xml
Beginners often mix these three up because they all live in the root folder of a site. They do very different jobs:
| File | What it does | Who reads it | Do you need it? |
|---|---|---|---|
| robots.txt | Tells crawlers which URLs they may or may not crawl | Search engine crawlers and many AI crawlers | Yes, almost every site should have one |
| sitemap.xml | Lists the URLs you want search engines to find, with optional last-changed dates | Google, Bing and other search engines | Yes, especially for new or growing sites |
| llms.txt | Gives AI tools a short summary and a curated list of key pages in Markdown | No confirmed use by the major AI search products | Optional |
The key difference: robots.txt controls access and sitemap.xml helps discovery. llms.txt does neither. If you want to stop an AI crawler from using your content, you still do that in robots.txt, by writing a rule for that crawler's user agent (the name a bot uses to identify itself), such as GPTBot.
Which AI tools read llms.txt today?
This is where honesty matters, because many guides overstate it. Here is what is publicly known as of October 2026:
- Google Search does not use it. In 2025 Google's Gary Illyes said normal SEO is what works for AI Overviews and that Google would not use llms.txt. Google has since said the file will neither help nor harm your search rankings.
- Google's own page on AI features does not ask site owners to create any special AI file. The advice is the same as for regular search: pages that can be crawled, indexed and that genuinely help the reader.
- Chrome's Lighthouse tool now checks llms.txt. Search Engine Journal reported that the audit flags files whose links do not point to Markdown versions of pages. A developer audit is not a ranking signal, but it shows the format is being taken seriously in some tools.
- Coding assistants and documentation tools are where the file is used most. Many software documentation sites publish one so developers can paste it straight into an AI assistant.
- The big AI chat products have not published clear statements that their crawlers fetch llms.txt to choose sources. You may see the odd request for the file in your server logs, but a request alone does not prove it changes anything.
How to create an llms.txt file in five steps
- Open a plain text editor, such as Notepad, VS Code, or TextEdit set to plain text.
- Write the H1 line with your site name, then a one-sentence blockquote summary.
- Add two or three H2 sections and list 5 to 20 of your best pages. Pick the pages a stranger would need first, not every post you have written.
- Save the file as
llms.txtin UTF-8 and upload it to your site's root folder, next to robots.txt. - Open
yoursite.com/llms.txtin a browser to check it loads as plain text. Then set a reminder to update it whenever you add or remove key pages.
If your site runs on a CMS, check whether it can create the file for you. If a tool generates the file for you, check that it doesn't simply dump every URL — that defeats the purpose. A short, curated list is the whole idea.
What is llms.txt and should a small site use it? Our recommendation
Here is how I would decide, in plain terms:
- Add it if you run tutorials, documentation, a tool or a reference site that people might feed into an AI assistant, or if you simply want a tidy index of your key pages. The cost is small.
- Skip it for now if your basic SEO is not done yet. Missing titles, slow pages, thin content or pages Google cannot reach will cost you far more than a missing llms.txt.
- Never treat it as a ranking trick. It will not get you into AI Overviews, and it does not replace backlinks, internal links or a sitemap.
The order of work for a small site is the same as it was before AI search. Make your pages crawlable, get them indexed, and earn a few real links (our beginner guide to backlinks explains how). Get listed where your audience already looks, such as quality free web directories and the main search engines, which our search engine listing guide walks through step by step.
So, what is llms.txt and should a small site use it? It is a curated Markdown reading list for AI tools, and a small site can add one in about twenty minutes with no downside, as long as you treat it as a tidy extra and keep most of your time for content, links and crawlability.
Frequently asked questions
Does llms.txt help with Google rankings or AI Overviews?
No. Google has said the file will neither help nor harm search rankings, and it does not use llms.txt for AI Overviews. Showing up in AI features still depends on normal SEO: indexable, helpful pages that answer the search well.
Can llms.txt block AI crawlers from using my content?
No. llms.txt has no allow or disallow rules at all. To block a specific AI crawler, add a rule for its user agent in robots.txt, such as GPTBot for OpenAI. Remember that robots.txt is a request that well-behaved bots follow, not a lock.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short index of links with notes. llms-full.txt is a convention some sites use to put the full text of their documentation into one large file. For a small blog or list site, the short llms.txt is enough.
