GEO / Technical guides
llms.txt: The Complete Guide (With Generator)
What llms.txt is, why AI systems read it, the full syntax reference, common mistakes that break it, and how to test yours. The 10-minute file that improves AI crawlability.
Last updated: October 2026 · Covers the current llms.txt convention as implemented across AI crawlers
What llms.txt is
When an AI crawler visits your site, it faces thousands of pages with no map. llms.txt is the map. It's a markdown file at your domain root that summarizes what your site is, links the important pages, and states usage guidance — all in a format language models parse natively.
Think of it this way: robots.txt = permissions (what crawlers may fetch). llms.txt = guidance (what matters once they're in). You need both. A site with open robots.txt but no llms.txt is like an unlocked library with no catalog.
Why it matters for AI visibility
Crawler efficiency
AI crawlers have per-site budgets. A good llms.txt routes them to your best pages immediately instead of burning budget on nav, tags, and archives.
Accurate representation
Without guidance, AI systems summarize your site from whatever they happen to crawl — often outdated or peripheral pages. llms.txt lets you define the canonical story.
Citation quality
When AI knows which pages are authoritative on which topics, it cites the right page — not a random blog post from 2019.
Zero cost
It's a static text file. No build step, no vendor, no subscription. Ten minutes of work for a permanent visibility asset.
Syntax reference
llms.txt is plain markdown with a conventional structure. Here's a complete template:
# Your Site Name
> One-sentence description of what the site is and who it's for.
## Key pages
- [Homepage](https://yoursite.com/): Main entry point and overview.
- [About](https://yoursite.com/about): Who runs the site, credentials.
- [Pricing](https://yoursite.com/pricing): Plans and pricing details.
## Content
- [Blog](https://yoursite.com/blog): Articles on [topics].
- [Docs](https://yoursite.com/docs): Technical documentation.
- [Guides](https://yoursite.com/guides): How-to guides.
## Optional
- [Status](https://yoursite.com/status): Service status page.
- [Contact](https://yoursite.com/contact): How to reach us.The generator: build yours in 2 minutes
Answer three questions, get a starter file:
1. What's your one-sentence site description?
Example: 'AIClicks.net is a website security and AI visibility auditing platform with 7 free tools.' Write it as you'd explain to a smart stranger.
2. What are your 5–10 most important pages?
Homepage, about, pricing, main product/tool pages, top 3 blog posts or guides. Link each with a one-line description of what it contains.
3. Any usage guidance?
Optional: preferred citation format, contact for commercial use, pages to skip (login, cart, admin). Keep it brief — this is guidance, not a legal document.
Paste the result into a file named llms.txt at your domain root. Done.
5 common mistakes
✕ Wrong location
It must be at /llms.txt on the domain root — not /docs/llms.txt, not a blog post about llms.txt. Crawlers check the root path.
✕ Wrong content type
Serve as text/plain or text/markdown with HTTP 200. A 404 page that says 'llms.txt coming soon' is worse than nothing — it looks like a dead end.
✕ Marketing fluff instead of a map
AI doesn't need your tagline repeated six times. It needs links to pages with one-line descriptions. Every line should help a crawler decide what to fetch next.
✕ Stale links
A llms.txt pointing to deleted pages erodes trust. Update it when your site structure changes — same discipline as your XML sitemap.
✕ Forgetting robots.txt
llms.txt guides crawlers that are allowed in. If your robots.txt blocks GPTBot, your beautiful llms.txt will never be read. Fix permissions first.
How to test yours
1. Manual: curl -I https://yoursite.com/llms.txt — expect HTTP 200 and a text content type.
2. Automated: our free AI crawler check verifies llms.txt presence, parseability, and whether your robots.txt actually lets AI crawlers reach it.
3. Ongoing: re-check after every site restructure. Add it to your deploy checklist next to the XML sitemap.
FAQ
What is llms.txt?
llms.txt is a markdown file at your domain root (yoursite.com/llms.txt) that gives AI systems a structured summary of your site: what it's about, which pages matter, and how the content may be used. Proposed in 2024 by Jeremy Howard, it works like robots.txt but for language models instead of search crawlers.
Which AI systems actually read llms.txt?
Anthropic documents llms.txt support, and adoption spread quickly through 2025–2026 across AI crawlers and agent frameworks. Even where support is partial, the file costs nothing to maintain and helps every system that does read it — and signals machine-readability seriousness to human evaluators too.
Where do I put llms.txt?
At the domain root: https://yoursite.com/llms.txt. Serve it as text/plain or text/markdown with a 200 status. Some sites also publish llms-full.txt (complete content dump), but the standard llms.txt summary file is the priority.
How is llms.txt different from robots.txt?
robots.txt tells crawlers what they're allowed to fetch (permissions). llms.txt tells AI systems what your site contains and which parts matter (guidance). You need both: robots.txt must allow AI crawlers, and llms.txt should then guide them efficiently once they're in.
Can llms.txt hurt my SEO or visibility?
No — it's purely additive guidance for AI systems. It doesn't affect Google rankings, doesn't expose anything not already public, and doesn't override robots.txt permissions. The only risk is writing misleading content in it, which erodes trust.
How do I test if my llms.txt works?
Fetch it with curl to confirm a 200 response and correct content type. Then run AIClicks.net's free AI crawler check, which verifies llms.txt presence, parseability, and whether it's linked sensibly from your site structure.
Test your llms.txt right now
Free check: llms.txt, robots.txt AI rules, and crawler access — 30 seconds, no signup.
Free AI crawler checkFull GEO checklist →Related: GEO audit checklist · Why ChatGPT skips sites · Best GEO tools