llms.txt: what it does, what it doesn't, and whether to bother

llms.txt is a proposed plain-text Markdown file, placed at the root of a website, that gives large language models a short, curated summary of the site and links to its most useful pages. It is cheap to add and can help AI tools that read it, but no major AI search engine has confirmed it as a ranking or citation signal.

What is llms.txt, exactly?

The idea is simple. A web page is built for browsers: navigation, cookie banners, scripts, footers. A language model trying to understand your business has to dig through all of that. The llms.txt proposal, published in 2024, suggests a single file at /llms.txt written in Markdown that says, in effect, "here is who we are, and here are the pages worth reading". Some versions also point to clean Markdown copies of key pages, and a companion idea, llms-full.txt, bundles the full text of documentation into one file.

It is worth being precise about its status. llms.txt is a community proposal, not a web standard, and it is not the same thing as robots.txt. robots.txt tells crawlers what they may and may not fetch. llms.txt does not grant or block anything; it is a reading guide.

Do ChatGPT, Gemini, Perplexity or Google AI Overviews use it?

This is where most articles on the subject overreach, so we will be careful. At the time of writing, none of the major AI search experiences has publicly stated that llms.txt influences which sources they cite. Google has said its AI features rely on its normal search index and ranking systems, and its representatives have been sceptical of the file. Server logs from various site owners show occasional fetches of llms.txt, but a fetch is not proof of use in an answer.

Where llms.txt clearly has value is in tools that are told to use it: developer assistants, coding agents and documentation platforms that load a project's llms.txt so a model can work with accurate, current docs. If your audience includes developers, that alone can justify the effort.

Our view at SNMRush: treat llms.txt as good housekeeping, not a visibility lever. It takes an hour, it forces useful clarity, and it will not rescue thin content.

How do you write a good llms.txt?

The structure in the proposal is short. An H1 with the site or brand name, a one-paragraph summary in a blockquote, optional notes, then H2 sections containing lists of links with a one-line description each. A section called "Optional" can hold links a model may skip if context is limited.

A worked example for a fictional accounting software company might look like this in outline:

• Title: the brand name, exactly as you want it to be named.
• Summary: two sentences stating what the product is, who it serves and where it operates. Write it as the answer you would want an assistant to give.
• Core pages: product overview, pricing, the three most important help articles, security and compliance.
• Explainers: the guides that answer your audience's most common questions.
• Optional: company history, press, careers.

The summary paragraph is the part that matters most, and it is the same discipline as the answer block we recommend on every page. If you cannot describe your company in two plain, factual sentences, an AI system will struggle too. Our guide to answer-first writing covers that skill in depth.

Which pages should you include?

Choose pages that answer real questions, not pages that you most want to sell. This is where audience research earns its keep. At SNMRush we use SOMIN, an AI audience-research platform, to see which questions and tensions people voice about a category in public conversation, then check whether the site has a clear page for each. The pages that answer those questions go into llms.txt; gaps go onto the content plan. If you want to see how that research feeds strategy more broadly, the SOMIN strategy overview explains the approach.

A short checklist for selection:

1. Does the page answer one question clearly in its first paragraph?
2. Is it current, with accurate prices, features and dates?
3. Is it accessible without login, scripts or interstitials?
4. Would you be happy if an assistant quoted it word for word?
5. Is it the single best page on the site for that question, rather than one of five overlapping ones?

Definitions: llms.txt and its neighbours

• robots.txt: a long-standing standard that tells crawlers which paths they may fetch. This is where you allow or block AI crawlers such as GPTBot or Google-Extended.
• sitemap.xml: a machine-readable list of URLs for search engines to discover.
• llms.txt: a proposed human-readable Markdown guide to your most useful content for language models.
• llms-full.txt: a proposed companion file containing full text, mostly used for documentation.

What matters more than llms.txt?

If you only have a week, spend it elsewhere first. Most AI answer engines that browse the web draw on search indexes, so crawlability, indexing and rankings remain the foundation. Next comes the content itself: direct answers, specific facts, honest comparisons. Then structured data and consistent entity information, which help machines understand who you are. Finally, mentions on credible third-party sites, which shape how assistants describe you.

This ordering is also a useful antidote to the belief that a single file or tag can win AI visibility. The reasoning models discussed by our sister studio GPT5 Marketing are increasingly good at weighing sources; they reward substance, and they are hard to trick.

How do you check whether it is working?

Be modest about attribution. Check your server logs for requests to /llms.txt and note the user agents. Keep a fixed panel of prompts and track whether AI tools cite you, before and after the change, but expect any effect to be mixed in with everything else you are doing. Agencies featured in the KPI Media case study show the value of grounding decisions in evidence rather than assumption, and the same principle applies here: measure, do not assume.

Our verdict is straightforward. Add an llms.txt if you have documentation, a developer audience, or simply want a tidy summary of your site for machines. Write the summary paragraph with care. Then get back to the work that is known to matter: answering the questions your audience is actually asking, better than anyone else.

If you would like a second opinion on your setup, SNMRush includes an llms.txt review in every AI visibility audit, alongside crawler access, schema and citation tracking.

Frequently asked questions

Is llms.txt required for AI search visibility?

No. It is an optional proposal and no major AI search engine has confirmed it as a citation or ranking signal. Crawlability, strong content and credible mentions matter far more.

Does llms.txt block AI crawlers?

No. llms.txt is a reading guide. To allow or block specific AI crawlers, use robots.txt rules for their published user agents.

Where should llms.txt live?

At the root of your domain, for example https://example.com/llms.txt, written in Markdown with an H1, a short summary and lists of links.