llms.txt (AI Crawler Guidance)

What the llms.txt file is, how it guides AI crawlers like GPTBot and ClaudeBot to cite your site accurately, and why AI visibility starts with crawler guidance.

What it is

llms.txt is an emerging convention (like robots.txt, but for AI): a markdown file at your domain root describing your site for large language models. It helps AI assistants cite you accurately instead of guessing.

Why it matters

AI search is eating traditional search share. Sites with clear llms.txt, accurate structured data, and crawlable content get cited; sites without it get summarized from third-party sources — or ignored.

How to fix it

Create

/llms.txt — markdown summary of your site: what you do, key pages, contact

robots.txt

Decide per AI crawler (GPTBot, ClaudeBot, PerplexityBot): allow or disallow explicitly

Structured data

Add JSON-LD (Organization, FAQPage, Product) so AI parses facts correctly

See if your site passes

Check your AI crawler setup free

FAQ

Is llms.txt an official standard?

Not yet — it's a proposed convention gaining adoption. Even so, clear structured content at predictable paths helps every AI crawler today.

Should I block AI crawlers?

Only if you have a reason (content licensing). For most businesses, being cited by AI is free distribution.

More explainers: HSTS (HTTP Strict Transport Security) · Content-Security-Policy (CSP) · DMARC (Domain-based Message Authentication) · Cookie Consent (GDPR/ePrivacy) · SPF (Sender Policy Framework) · DKIM (DomainKeys Identified Mail) · X-Frame-Options / frame-ancestors · all checks