Overview
- A small eight-day server log run by a developer found per-page Markdown twin files were fetched about 30 times by agents claiming to be AI crawlers while the /llms.txt index was fetched only four times and not by confirmed AI crawlers.
- Google has said its Search and AI features do not rely on llms.txt and instead provides per-page Markdown copies for its own developer docs, demonstrating an existing large crawler prefers page-level text.
- Major AI providers including OpenAI and Anthropic have not published documentation committing to parse llms.txt, so its practical value depends on future, explicit adoption by those platforms.
- Practical guidance from the coverage recommends adding llms.txt if it is trivial to maintain but prioritizing server-rendered, fast, crawlable pages and clean semantic markup or per-page Markdown twins so crawlers can actually read and cite your content.
- The current evidence is limited: the log sample is small, user-agent strings can be spoofed, and broader or longer-term tests are needed before llms.txt can be treated as a reliable retrieval mechanism.