llms.txt: What It Is and Whether Your Site Needs One

We build websites and run SEO and AI search work for businesses in Kerala and the Gulf, and our own site publishes an llms.txt. We still tell clients it is optional, and this guide explains why.

Every few weeks someone asks us whether they "need an llms.txt for ChatGPT". The honest answer has two halves. The file is easy to make and does no harm. It is also not a ranking factor anywhere we can verify, and the time it takes is better spent on pages that answer real questions. Below: what the file is, who actually reads it, what goes in a good one, what we found when we checked 58 Indian and Gulf software sites on 4 October 2026, and a template you can copy.

What is llms.txt?#

llms.txt is a proposal, not a standard. Jeremy Howard of Answer.AI published it on llmstxt.org (opens in a new tab) on 3 September 2024 and released a second version on 10 August 2026. The idea is simple: web pages are built for people, wrapped in menus, scripts and adverts, and an AI agent with a limited context window is better served by a short, clean summary in one predictable place.

The proposal defines the format precisely. A file following it contains, in order:

An H1 with the name of the site or project. This is the only required part.
A blockquote with a short summary holding the key facts needed to understand the rest.
Optional paragraphs or lists with more detail, but no headings.
H2 sections of links, each link written as [name](url) with an optional note after a colon.

By convention, a section called "Optional" holds secondary links an agent can skip when it needs a shorter context. Version 2 also spells out what a file at a subpath covers: /docs/llms.txt covers everything under /docs/. And it suggests HTTP link headers that point from a page to its Markdown version and to the llms.txt that covers it.

The proposal says plainly who it is for. Its author expected llms.txt to be useful mainly "for inference rather than training", meaning an assistant reading it while answering someone's question, and says that is how it has been used. And the revised text notes the files "are used most heavily for software documentation, where coding agents follow them to find API references and tutorials". OpenAI, Anthropic and Google's Gemini team all publish one for their own developer docs.

Who reads it in 2026, and does Google?#

Google doesn't. Its guide Optimizing your website for generative AI features on Google Search (opens in a new tab), published on 15 May 2026 and last updated on 10 July 2026, lists "LLMS.txt files and other 'special' markup" among things site owners can ignore:

You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.

It adds that it is "completely fine" to keep one for other systems, and that "doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them". On Google's Search Off the Record podcast, as reported by Search Engine Journal (opens in a new tab) on 15 June 2026, John Mueller explained the deeper problem: language models "by design, can't trust what is here as a way of differentiating between different websites", because the file is the site describing itself.

What about the AI companies' own crawlers? We read the crawler pages of OpenAI (opens in a new tab), Anthropic (opens in a new tab) (updated 7 April 2026) and Perplexity (opens in a new tab) on 4 October 2026. They explain which bots follow robots.txt. None of them says the bot reads a website's llms.txt. OpenAI and Perplexity mention llms.txt only as the index of their own documentation.

Crawler logs point the same way. OtterlyAI ran a 90-day experiment (opens in a new tab) (published 5 February 2026): out of more than 62,100 AI bot visits to a test site, 84 requested /llms.txt, about 0.1%, while an average page drew around 265 visits. A larger study by SE Ranking (opens in a new tab) (7 November 2025) looked at nearly 300,000 domains: 10.13% had the file, and it found no effect on how often a domain was cited by AI models.

Two signals point the other way. Chrome's Lighthouse now has an llms.txt audit (opens in a new tab) (page updated 5 May 2026) among its agentic browsing checks. It flags a site when fetching the file returns a server error, and marks a missing file "Not Applicable", "as providing the file is optional at the moment". And, by the proposal author's account, coding agents use llms.txt files on documentation sites reliably. If you run a software product with public docs, the file earns its keep. If you run a clinic, a builder or a shop, it is housekeeping.

llms.txt, robots.txt and sitemaps: what does each do?#

The three files are often confused, partly because they share a location. They do different jobs.

File	What it tells machines	Who reads it	What happens without it
robots.txt	Which crawlers may fetch which paths	Search engines and most AI crawlers, by name	Everything public may be crawled
sitemap.xml	Every page you want indexed, with optional dates	Google, Bing and other search engines	Pages are found through links instead
llms.txt	A short summary of the site and a curated list of key pages	Some AI tools and coding agents, on demand	Agents read your normal pages

The proposal itself draws the line between the last two: a sitemap lists everything indexable, while llms.txt is "a curated overview", small enough to fit in a model's context. And robots.txt is where AI access is actually decided. OpenAI's page, for example, says sites that block OAI-SearchBot "will not be shown in ChatGPT search answers", and that robots.txt changes take "~24 hours" to apply. It also says its ChatGPT-User agent acts for a user, so "robots.txt rules may not apply". Perplexity says its user-triggered fetcher "generally ignores robots.txt rules". Anthropic says its three bots honour robots.txt, including Crawl-delay.

If you only fix one file this month, make it robots.txt: decide which AI crawlers you allow, then check that your firewall or CDN isn't blocking the ones you meant to let in.

What goes in a good llms.txt?#

The same things an assistant would need to describe you correctly: who you are, where you are, what you do, and the few pages that prove it.

Our file. marketlube.in builds its llms.txt from the site's own data on every release. The code reads the same list of indexable pages that feeds our sitemaps, so a page held back from search, such as an industry page without enough real projects behind it, never appears. When we checked on 4 October 2026 it ran to 121 lines and 81 links in nine sections: Services, Industries, Markets, Case studies, Brand identity studies, Work, Insights, Company and Optional. Above them, a blockquote gives our one-line description, followed by the facts we want repeated exactly: our only office, the founders, the year we started, where we serve, how to contact us and our hours. The office, founders, founding year and contact details come from the same source file as our footer and structured data, so they can't drift apart.

Xale's files. Xale, a CRM product built by the team behind Marketlube, publishes both an llms.txt (opens in a new tab) (about 57 KB) and an llms-full.txt (opens in a new tab) (about 90 KB). Its llms.txt carries a "Last updated" line near the top and a section called "Common misconceptions", correcting outdated descriptions that summaries had repeated, each with a link to the page that shows the fact. Its robots.txt allows AI crawlers by name, as our Xale case study shows.

What we found on 58 Indian and Gulf software sites#

On 4 October 2026 we requested /llms.txt and /llms-full.txt from 58 software and technology companies: 18 Indian IT services firms, 20 Indian software-as-a-service companies and 20 Gulf technology companies. We counted a file only when the server returned it as text with Markdown content. A page served at that address, or a redirect to the home page, didn't count.

Group	Sites checked	Blocked our request	Publish llms.txt
Indian IT services	18	2	11 of 16
Indian SaaS	20	1	14 of 19
Gulf technology	20	3	4 of 17
All	58	6	29 of 52

The Indian files came from HCLTech, LTIMindtree (the request redirected to ltm.com), Mphasis, Persistent, Coforge, Hexaware, Zensar, Birlasoft, Happiest Minds, Mastek and Tata Elxsi among services firms, and Freshworks, Chargebee, Postman, Razorpay, Kissflow, Druva, MoEngage, VWO (served from wingify.com), Darwinbox, Keka, Gupshup (served from gupshup.ai), Whatfix, Capillary and Tally Solutions among product companies. In the Gulf, Foodics, Zid, Kitopi and Help AG published one.

The quality varied more than the adoption:

Six files came from a tool rather than a person: four from Yoast SEO, one from the Wix platform and one from an online generator. Two still opened with the tool's own note rather than the company's name.
Two files had no links at all, which misses the point of the format. Six skipped the blockquote summary or the links.
Two files were over 400 KB, effectively a sitemap pasted into Markdown, far beyond what an agent can use in one go.
Four showed when they were last updated, the simplest signal that someone maintains the file. The oldest of those dates was more than nine months old.
Only one company published a separate llms-full.txt (Razorpay). Postman served the same file at both addresses.

A request from one connection in Kerala is a snapshot, not a survey. Sites can serve different responses to different clients, and the six that blocked us may serve the file to approved crawlers.

A template you can copy#

Replace every line with your own facts, keep it under a few hundred lines, and link only pages you would be happy for an assistant to quote. The first line is your business name as an H1, written # Your Business Name, and the rest follows it:

> One or two sentences: what you do, for whom, where, and since when.

- Office: your full address, written exactly as on your Google Business Profile
- Serving: the cities, states or countries you serve
- Contact: email · phone · WhatsApp
- Hours: Monday to Saturday, 9:30 to 18:30 IST
- Last updated: October 2026

## Services

- [Service name](https://example.com/services/service-name): what it includes, in one line

## Proof

- [Case study name](https://example.com/case-studies/name): the result, with its date and how it was measured

## Guides

- [Guide title](https://example.com/blog/guide-slug): the question it answers

## Optional

- [Privacy policy](https://example.com/privacy-policy)

When does llms-full.txt help?#

When an agent genuinely needs your whole text in one file, which mostly means product documentation. llms-full.txt is not part of the llms.txt proposal, and version 2 doesn't mention it. It came from documentation platforms: Mintlify says it developed the format (opens in a new tab) with its customer Anthropic (post updated 14 March 2026), as a single Markdown file holding a site's full text so a developer can load it into an AI tool with one URL.

For a software product with an API, an SDK or a long manual, that is useful, and Xale publishes one for that reason: its 17 numbered sections cover the product in more depth than any single page. For a typical business site of 30 to 100 pages it adds a second copy of everything to keep accurate, and a stale copy is worse than none. In our check, only one of 52 reachable sites bothered with a separate one.

How do you generate it and keep it current?#

Pick the method that matches how your site is built:

By hand, for a small site that rarely changes. Half an hour with the template above, then a reminder to review it whenever your services, address or phone number change.
With your CMS. The proposal lists tools that generate the file, including Yoast SEO and AIOSEO for WordPress, Wix (which makes one for every Wix site), and Mintlify and GitBook for documentation. Read what the plugin produces before you trust it: in our check, plugin files sometimes led with the tool's note instead of the company's summary.
From your own data, at build time. This is what we do. The route that serves our llms.txt reads the same registry as the sitemap and the SEO checks, plus the one file that holds our name, address and phone. When we add a case study, it appears in llms.txt on the next release with no extra work. Because of that, our own SEO plan lists further work on llms.txt under "not doing": it stays current by itself.

Whichever route you choose, check four things. The file returns 200 and is served as plain text. It doesn't redirect to your home page. Your firewall lets it through. And it gives the same facts as your site, your Google Business Profile and your directory listings. The proposal suggests a good final test: give an AI assistant only your llms.txt and ask it questions about your business.

What it will not do for rankings or AI citations#

It won't rank you higher on Google, which ignores it. It won't make ChatGPT, Claude or Perplexity recommend you: none of their crawler pages say they read it, and SE Ranking found no link between the file and AI citations. And it won't fix a site whose pages can't be crawled or say nothing worth quoting.

What does move AI visibility is what our guide to generative engine optimisation covers: pages crawlers can read without JavaScript, answer-first passages, the same business facts everywhere, and mentions on other reputable sites. Pages built around the real questions your buyers ask do more than any text file, as our post on SEO for educational institutions shows for colleges and schools. And if you want to know whether AI assistants already send you visitors, our guide on tracking ChatGPT and AI traffic in GA4 walks through the set-up we use on our own site.

Frequently asked questions#

Does Google use llms.txt?#

No. Google's guide to generative AI features, published in May 2026 and given a note on llms.txt in June, says Google Search doesn't use llms.txt or other AI text files, and that keeping one will "neither harm nor help" your visibility or rankings. Google may still crawl the file like any other text file.

Is llms.txt an official web standard?#

No. It is a proposal published on llmstxt.org by Jeremy Howard in September 2024, revised in August 2026. No standards body has adopted it, and none of the crawler pages from OpenAI, Anthropic, Perplexity or Google that we read on 4 October 2026 says its crawler reads a website's llms.txt.

What is the difference between llms.txt and llms-full.txt?#

llms.txt is a short summary with links to your key pages. llms-full.txt, which documentation platforms introduced and which the llms.txt proposal doesn't define, puts a site's full text in one Markdown file. The full version mainly suits software documentation.

Where should the llms.txt file go?#

At the root of your domain, for example example.com/llms.txt, served as plain text. The proposal also allows files at subpaths, such as /docs/llms.txt, and version 2 makes clear that such a file covers only the pages under that path.

Will an llms.txt file get my business into ChatGPT answers?#

Not on its own evidence so far. A 90-day experiment by OtterlyAI found AI bots requested the file in about 0.1% of their visits, and an SE Ranking study of nearly 300,000 domains found no effect on AI citations. Crawlable, quotable pages and consistent business facts matter far more.

Should you publish one?#

Publish one if you can keep it accurate with little effort, or if you run a software product with public documentation. Skip it without guilt if your site still has bigger fixes waiting, such as pages that don't load without JavaScript or facts that disagree with your Google Business Profile. If you'd like us to look at how your site reads to search engines and AI assistants, send us a brief, or see how our SEO and AI search programme works. Or ask Althameem Ali, our SEO expert in Kerala, in a consultation.

Changelog#

2026-10-10: The closing paragraph now offers a consultation with Althameem Ali, our SEO expert in Kerala.