LLMs.txt Generator: Build, Validate & Download

llms.txt is a standardized markdown file that lives at the root of your website (e.g., yoursite.com/llms.txt). It was proposed by Jeremy Howard, co-founder of Answer.AI and creator of fast.ai, in September 2024 to solve a fundamental problem: AI tools struggle to understand websites.

Modern websites are built with complex HTML, JavaScript, navigation menus, cookie banners, ads, and interactive elements. When an AI model like ChatGPT, Claude, or Perplexity visits your site, it has to parse through all that noise to find the actual content. With limited context windows, much of your important information gets lost.

llms.txt gives AI a clean, structured summary of your site in plain markdown. It tells AI tools who you are, what you do, and where to find your most important pages. Think of it as a welcome guide specifically designed for AI consumption, separate from your human-facing HTML pages.

The official specification defines a strict structure. Every llms.txt file follows this order:

1. H1 Heading (required): The name of your project or website. This is the only required element. Use a single # prefix.

2. Blockquote (recommended): A brief summary of your site using the > prefix. This gives AI tools a quick understanding of what you do.

3. Additional Details (optional): One or more paragraphs of plain text providing extra context. No headings here, just paragraphs or lists.

4. H2 Sections (recommended): Sections delimited by ## headings, each containing a list of links. Common sections include Docs, Products, Blog, About, and API.

5. Link Format: Each link uses the pattern - [Title](URL): Description. The description is optional but strongly recommended because it helps AI tools decide which pages are relevant to a query.

6. Optional Section: A special ## Optional section at the end marks URLs that AI tools can skip when working with limited context. This section must always be last if present.

These three files serve different purposes and are complementary:

robots.txt controls access permissions. It tells crawlers which URLs they can and cannot visit. It acts as a gatekeeper but provides no content or context about your site.

sitemap.xml lists all indexable URLs on your site with metadata like last modified dates and priority values. It is designed for search engine crawlers to discover your pages, but it does not describe what those pages contain.

llms.txt provides curated content summaries specifically for AI tools. Instead of listing every page, it highlights your 20 to 50 most important pages with human-readable descriptions. It is the only file designed specifically for AI consumption rather than traditional search engines.

For the best AI visibility, use all three files together. robots.txt controls who gets in, sitemap.xml maps your full site, and llms.txt provides the AI-friendly summary.

ChatGPT (with web browsing) can access your llms.txt file when it browses the web. When a user asks about your product or service, ChatGPT can read your llms.txt to quickly understand what you offer and provide accurate answers with proper context.

Claude uses llms.txt through its web search capabilities and integrations. AI assistants and tools built on Claude can reference your llms.txt to understand your documentation structure and find relevant pages.

Perplexity crawls the web to answer questions with citations. A well-structured llms.txt helps Perplexity understand your site hierarchy and cite the correct pages in its answers.

AI coding assistants like Cursor read llms.txt files from documentation sites to understand project structure. This is one of the earliest and most practical uses of the standard.

As more AI tools adopt this standard, having a proper llms.txt file becomes an increasingly important part of your Generative Engine Optimization (GEO) strategy.

WordPress: Download your generated llms.txt file and upload it to your site root directory (the same directory as wp-config.php). Use your hosting file manager or an FTP client like FileZilla. After uploading, verify it works by visiting yourdomain.com/llms.txt in your browser.

Next.js / React: Place the file in your /public directory. Next.js automatically serves files from /public at the root URL. So /public/llms.txt becomes accessible at yourdomain.com/llms.txt with no extra configuration.

Shopify: Go to Settings > Files, upload your llms.txt file, then create a URL redirect from /llms.txt to the uploaded file URL. Alternatively, you can use the Shopify admin to add custom liquid code that serves the file.

Webflow: Currently, Webflow does not natively support serving plain text files at the root domain. You can use a Cloudflare Worker or a redirect rule to serve the file. Some users host the file on a CDN and redirect /llms.txt to it.

Static sites (Hugo, Gatsby, Jekyll): Place the file in your static or public directory. It will be deployed alongside your other static assets and accessible at the root URL.

1. Keep it curated. Do not list every page on your site. Focus on the 20 to 50 most important pages that best represent your product, services, or content. Quality descriptions beat quantity.

2. Write clear descriptions. Each link description should tell AI tools what the page covers in one sentence. Use specific, factual language. Avoid marketing buzzwords that add noise without meaning.

3. Organize by topic. Group related pages into logical sections. Use clear section names like Docs, Products, Blog, API, or About. This helps AI tools navigate your content structure quickly.

4. Use the Optional section. Move supplementary pages (changelog, careers, terms of service) to the Optional section. This signals to AI tools with limited context that these pages can be skipped.

5. Update regularly. Review your llms.txt when you launch new products, publish significant content, or restructure your site. An outdated file can point AI tools to pages that no longer exist or miss your most important new content.

6. Add noindex for search engines. Since llms.txt is designed for AI consumption, you may want to prevent it from appearing in Google search results. Add a noindex header or exclude it from your sitemap.

7. Validate after changes. Use our Validator tab to check your file against the specification after any edits. A malformed file may not be parsed correctly by AI tools.

Since its proposal in September 2024, hundreds of companies have adopted llms.txt. Notable early adopters include:

Anthropic (maker of Claude) hosts llms.txt for their documentation site, helping developers quickly find API references, guides, and model information through AI tools.

Cursor (AI code editor) uses llms.txt extensively, which makes sense given that AI coding assistants are one of the primary consumers of this format for understanding documentation.

Cloudflare, Astro, Deno, and many other developer-focused companies have published llms.txt files for their documentation and product pages.

The adoption is growing beyond developer tools into e-commerce, SaaS, agencies, and content sites as more businesses recognize the importance of AI visibility in their digital strategy.