Guide · 6 min · Updated 20/05/2026
robots.txt and llms.txt in 2026
robots.txt for crawlers, llms.txt for AI ingestion — and why most sites get both wrong.
robots.txt — three rules that matter
Disallow /admin and similar admin paths. Disallow parameterised paths that explode crawl frontier. Don't accidentally disallow /static if you serve images from there.
llms.txt — pointer file, not blocker
llms.txt is a content map for LLM crawlers, not an access-control mechanism. Use it to tell AI models WHICH content matters most.
Common questions
Does Google honour llms.txt?
Google does not commit to crawl behaviour based on llms.txt. Other LLM crawlers (Anthropic, OpenAI) increasingly do.