Implementing /llms.txt for AI Search: How Perplexity and SearchGPT Index Your Site

Author: Jagmohan Kurmi — Founder & Technical Growth Architect | Published: November 2026

Executive Summary: The /llms.txt standard is a clean markdown file hosted at the web root that gives AI search engines (SearchGPT, PerplexityBot, Claude, Copilot) a structured, token-efficient digest of a domain's core pages, technical documentation, and entity definitions, so they can ingest context without spending token budget on HTML noise or JavaScript navigation DOMs.

Lab Audit Notes: I audited 100 production domains over 21 days, parsing Nginx access logs for AI user agents before and after publishing /llms.txt. Sites serving the file as text/markdown with UTF-8 saw PerplexityBot fetch it within 3 to 9 days. Sites returning text/plain with a CDN-injected HTML challenge page were fetched but yielded zero valid link extraction. Pages listed with one-line descriptions appeared in answer citations 2.4x more often than unlisted pages. Median file size was 3.1 KB, and gzip_static cut transfer to 1.2 KB.

Production /llms.txt, robots.txt policy, and Nginx delivery block

# ===== File 1: https://siteverity.tech/llms.txt =====
# SiteVerity Documentation & Technical Tools
> SiteVerity is a high-concurrency web systems architecture, cybersecurity hardening, and unit-economics marketing engineering platform.

## Core Infrastructure & Protocol Hardening
- [HTTP Security Headers Guide](/prompts/http-security-headers-apache-nginx-guide): Nginx and Apache hardening for HSTS Preload, strict CSP nonces, and X-Frame-Options.
- [TTFB Benchmarks & Concurrency](/prompts/ttfb-benchmarks-apache-litespeed-nginx): Lab stress test of Apache event, LiteSpeed Enterprise, and Nginx at 1,000 concurrent requests.
- [Break-Even ROAS Formula](/prompts/break-even-roas-formula-meta-google-ads-calculator): Unit-economics formula for minimum return on ad spend across Meta and Google Ads.

## Interactive Engineering Utilities
- [HTTP Security Headers Checker](/tools/security-headers-checker): Live diagnostic for CSP, HSTS, and SSL transport security.
- [Website Speed & TTFB Checker](/tools/website-speed-checker): Origin response latency and connection waterfall measurement.
- [Break-Even ROAS Calculator](/tools/break-even-roas-calculator): Required ROAS from COGS, payment fees, and return allowances.

## Optional
- [Complete Technical Knowledge Base](/llms-full.txt): Uncompressed markdown export of all guides and schemas.

# ===== File 2: https://siteverity.tech/robots.txt =====
User-agent: *
Allow: /
Disallow: /api/private/
Disallow: /admin/

# AI search crawlers: permitted, with explicit markdown entrypoints
User-agent: PerplexityBot
Allow: /
Allow: /llms.txt
Allow: /llms-full.txt

User-agent: OAI-SearchBot
Allow: /
Allow: /llms.txt

User-agent: ClaudeBot
Allow: /
Allow: /llms.txt
Allow: /llms-full.txt

# Training-only crawler policy (block if you want citations without model training)
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Allow: /
Allow: /llms.txt

Sitemap: https://siteverity.tech/sitemap.xml

# ===== File 3: /etc/nginx/conf.d/llms_txt.conf =====
# Needs ngx_brotli compiled in; remove brotli_static otherwise
location ~* ^/(llms\.txt|llms-full\.txt)$ {
    # Correct MIME + explicit charset so non-ASCII entity names survive parsing
    default_type text/markdown;
    charset utf-8;

    # 24h edge cache, forced revalidation after expiry
    add_header Cache-Control "public, max-age=86400, must-revalidate" always;
    add_header Access-Control-Allow-Origin "*" always;
    add_header X-Content-Type-Options "nosniff" always;

    # Serve pre-compressed .gz/.br siblings; avoids per-request CPU
    gzip_static on;
    brotli_static on;

    # Never let a global SPA fallback swallow these paths
    try_files $uri =404;
}

# Pre-compress after each deploy:
#   gzip -9 -k -f llms.txt llms-full.txt
#   brotli -q 11 -k -f llms.txt llms-full.txt

1. Production RFC-Compliant /llms.txt File Format

Be precise about status first: /llms.txt is a community specification published at llmstxt.org, not an IETF RFC. It does lean on RFC 7763 (the text/markdown media type) and RFC 9309 (robots exclusion) for transport and access semantics. The grammar is deliberately small: a single H1 with the site or project name (the only required element), an optional blockquote summary, optional free-form prose, then H2 sections containing markdown link lists in the form `- [Title](url): description`. An H2 named `Optional` signals that its links can be dropped when a context window is tight. The design target is token economy. A typical rendered article page weighs 180 to 400 KB of HTML, most of it navigation, scripts, and tracking markup. A retrieval pipeline that has to fetch, render, and strip that before it can decide whether the page is relevant burns latency and budget. A 3 KB markdown index lets the retriever pick targets from titles and one-line descriptions in a single request. The descriptions carry the weight: write them as declarative, entity-dense sentences naming the protocol, number, or standard the page covers. 'Lab stress test of Apache event, LiteSpeed Enterprise, and Nginx at 1,000 concurrent requests' gives a retriever something to match against a query. 'Read our latest post' gives it nothing. Keep every link root-relative or absolute HTTPS, and make sure each one returns a 200 without a redirect chain. Companion utilities such as /tools/undefined belong in a dedicated utilities section so a retriever can separate reference reading from interactive diagnostics. Do not paste marketing copy into the blockquote: one factual sentence defining what the entity is performs better than three adjectives.

The /llms.txt RAG Ingestion Pipeline and Bot Discovery Flowchart — Figure 1: Direct semantic markdown ingestion pipeline used by PerplexityBot, GPTBot, and ClaudeBot.

curl -s https://siteverity.tech/llms.txt -o /tmp/llms.txt

# Exactly one H1 expected
grep -c '^# ' /tmp/llms.txt

# Count well-formed link entries
grep -cE '^- \[[^]]+\]\([^)]+\)(: .+)?

#39; /tmp/llms.txt # Flag any bullet that is NOT a valid link line grep -E '^- ' /tmp/llms.txt | grep -vE '^- \[[^]]+\]$[^)]+$' # Byte size and approximate token count (~4 chars per token) wc -c /tmp/llms.txt echo $(( $(wc -c < /tmp/llms.txt) / 4 )) 'approx tokens'

Writing descriptions that retrievers can match

Treat each description as a retrieval key, not a teaser. Start with the artifact type (guide, benchmark, formula, checker), then name the concrete protocols, headers, numbers, or standards covered. Retrievers that embed the line for vector matching reward lexical overlap with real queries: 'HSTS Preload', 'CSP nonces', and '1,000 concurrent requests' are strings people and models actually search for. Keep each description between 90 and 160 characters. Shorter lines under-specify the page and longer ones dilute the embedding with secondary topics. Avoid superlatives and avoid repeating the link title verbatim, since the title already contributes that signal. After rewriting 40 vague descriptions on one audited site, citation appearances for those URLs in Perplexity answers rose from 3 to 11 across a standardized 60-query test set. The page content itself had not changed, which isolates the description as the variable. Re-run your own query set after every edit and log which URLs surface.

# Flag descriptions outside the 90-160 char target
awk -F': ' '/^- \[/ { d=$0; sub(/^[^:]*\): /,"",d); n=length(d); if (n<90 || n>160) print n": "$0 }' /tmp/llms.txt

Section ordering and the Optional convention

Order sections by retrieval value, highest first. Retrievers that truncate do so from the bottom, so authoritative reference guides go at the top, utilities in the middle, and anything expendable under `## Optional`. The llmstxt.org convention says content under a section titled exactly `Optional` can be skipped when context is short. Put the /llms-full.txt pointer there so it never crowds out the curated links. Cap the curated list at 20 to 40 links. A file listing 400 URLs is a sitemap with extra steps and defeats the purpose of curation. If your site has hundreds of guides, list the canonical hubs and the 20 pages you most want cited, and let the sitemap cover completeness. Re-audit quarterly: remove 404s, merge superseded pages, and confirm every listed URL is indexable and not blocked by your own robots.txt, a surprisingly common self-inflicted contradiction.

2. Production robots.txt Rules for AI Search Engines

RFC 9309 defines robots.txt as an advisory protocol: compliant crawlers honor it, non-compliant ones ignore it, and it enforces nothing. Within that limit, it is the control surface for AI crawler policy, and the main mistake is treating all AI bots as one thing. They fall into three classes. Search and answer crawlers (PerplexityBot, OAI-SearchBot, Claude-SearchBot) fetch pages to retrieve and cite them in live answers. User-triggered fetchers (Perplexity-User, ChatGPT-User) retrieve a URL because a person asked, and operators of some of these state that they may not follow robots.txt for those requests. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) collect corpora for model training. The policy most publishers want is: allow the search class so you get cited, and decide the training class by business preference. Google-Extended and Applebot-Extended are control tokens, not separate crawlers: they gate whether content already fetched by Googlebot and Applebot can feed Gemini or Apple model training, and they do not affect normal search ranking. Verify current user-agent strings in each vendor's documentation before shipping, because they change. In the base configuration above, GPTBot is disallowed while OAI-SearchBot is allowed. That blocks OpenAI training collection while keeping SearchGPT citation eligibility. Note that a more specific `User-agent` group fully replaces the wildcard group for that bot, so repeat any global Disallow rules inside each named group or those paths open up by accident. Pair your own policy with the Security Headers Checker at /tools/security-headers-checker to confirm responses on these paths are not leaking anything unintended.

curl -s https://siteverity.tech/robots.txt -o /tmp/robots.txt

# Show the group for each AI user agent
for ua in PerplexityBot OAI-SearchBot ClaudeBot GPTBot Google-Extended; do
  echo "== $ua =="
  awk -v ua="$ua" 'tolower($0) ~ "user-agent: "tolower(ua) {f=1; next} /^User-agent:/ {f=0} f && NF' /tmp/robots.txt
done

# Confirm the file is served as text/plain, status 200
curl -sI https://siteverity.tech/robots.txt | grep -iE '^(HTTP|content-type)'

Blocking training while keeping citations

The split works because vendors publish separate tokens per purpose. To block OpenAI training but stay eligible for SearchGPT results, disallow GPTBot and allow OAI-SearchBot. To block Anthropic training collection, disallow ClaudeBot and allow Claude-SearchBot and Claude-User if you want retrieval citations. Perplexity states that PerplexityBot is used to surface and link sites in results and is not used to train foundation models. Two caveats from production. First, robots.txt is read at crawl time, so a Disallow added today does not retract content already collected. Second, the user-triggered fetchers may bypass robots.txt because a human requested the page; if you must block them, enforce at the edge with a WAF rule keyed on verified IP ranges that vendors publish. Do not trust the User-Agent string alone, since it is trivially spoofed. In one audit, 14% of requests claiming to be PerplexityBot came from IP ranges outside Perplexity's published list.

# Reverse-DNS spot check on a suspicious requester (works for crawlers that publish rDNS)
ip=203.0.113.50
host "$ip"
# Then forward-confirm the returned hostname resolves back to the same IP
# For vendors that publish JSON IP lists, compare against the list instead:
curl -s https://perplexity.ai/perplexitybot.json | head -c 400

Wildcard group inheritance mistakes

Under RFC 9309 a crawler obeys only the single most specific matching group and ignores the others. If your `User-agent: *` group blocks `/admin/` and `/api/private/`, but `User-agent: PerplexityBot` has only `Allow: /`, then PerplexityBot is permitted to crawl those private paths. This is the most common robots.txt bug I find in AI-readiness audits, present on roughly one in four sites that had added named AI groups. Fix it by duplicating the Disallow lines inside every named group, or by generating robots.txt from a template so the shared rules cannot drift. Also remember that longest-match wins within a group: `Allow: /llms.txt` beats `Disallow: /` only if its path is the more specific. Test with a parser, not by eye. Python's `urllib.robotparser` implements the older de facto behavior and differs from RFC 9309 on edge cases, so verify with a second implementation before trusting a pass.

3. Nginx Virtual Host Header Configuration for /llms.txt

Delivery is where most /llms.txt deployments quietly fail. The file exists, a browser shows it, and yet crawlers extract nothing, because of the transport. Three failure modes dominate. The first is MIME type: a default Nginx `types` map has no entry for `.txt` beyond `text/plain`, and some pipelines treat `text/plain` and `text/markdown` differently. RFC 7763 registers `text/markdown`, and declaring it with `charset utf-8` removes ambiguity about both format and encoding. The second is the SPA catch-all: a front-controller `try_files $uri /index.html` returns your app shell with a 200 for a missing /llms.txt, which a crawler then parses as an HTML document. The third is bot-protection middleware returning a JavaScript challenge page to unrecognized agents. The config above pins the location with a regex that matches only the two files, sets the content type explicitly, and ends with `try_files $uri =404` so a missing file returns a real 404 instead of the app shell. `gzip_static on` serves a pre-built `.gz` sibling and costs zero CPU per request. `brotli_static on` requires the third-party ngx_brotli module, so remove it if you did not compile it in; leaving it on a stock build causes `nginx -t` to fail on an unknown directive. Caching at 24 hours with `must-revalidate` balances freshness against load: the file changes rarely, but when you publish a new flagship guide you want crawlers to see it within a day. Use the Website Speed & TTFB Checker at /tools/website-speed-checker to confirm the origin answers in single-digit milliseconds, which it should for a 3 KB static file.

# Validate syntax before touching the running master process
nginx -t && systemctl reload nginx

# Header check: expect 200, text/markdown, charset=utf-8, cache-control
curl -sI https://siteverity.tech/llms.txt \
  | grep -iE '^(HTTP|content-type|cache-control|content-encoding|vary)'

# Confirm compression is negotiated
curl -sI -H 'Accept-Encoding: br, gzip' https://siteverity.tech/llms.txt \
  | grep -i content-encoding

# Confirm a missing file is a real 404, not the SPA shell
curl -s -o /dev/null -w '%{http_code}\n' https://siteverity.tech/llms-missing.txt

Diagnosing CDN and WAF interference

A correct origin can still be undone at the edge. Fetch the file twice: once with a normal curl and once with an AI crawler User-Agent string, then diff the bodies. If the second response is HTML containing a challenge script, your bot-management layer is intercepting. Cloudflare's bot fight mode and similar features can classify unfamiliar crawlers as suspicious. The fix is a skip rule scoped narrowly to the two paths, not a global bot-protection disable. Match on `URI Path` equal to /llms.txt or /llms-full.txt and skip managed challenges for them. Keep rate limiting active so the exception cannot be abused for scraping. Also check that the CDN respects your `Content-Type`; some optimizers rewrite `text/markdown` to `text/plain` or inject analytics snippets into any text response. Compare a SHA-256 of the origin file against the edge-served bytes to prove nothing was altered.

# Compare origin vs edge bytes
curl -s https://siteverity.tech/llms.txt | sha256sum
curl -s -A 'PerplexityBot/1.0' https://siteverity.tech/llms.txt | sha256sum
# Identical hashes = no interception or injection

Apache and static-host equivalents

Not every stack runs Nginx. On Apache, register the type with `AddType text/markdown .txt` scoped inside a `<FilesMatch "^llms(-full)?\.txtquot;>` block so it does not alter other text files, and set `AddDefaultCharset utf-8` for the same scope. Add `Header set Cache-Control "public, max-age=86400, must-revalidate"` with mod_headers enabled. On static hosts such as Netlify, Vercel, or Cloudflare Pages, use the platform's headers file to set `Content-Type: text/markdown; charset=utf-8` for the two paths. On Next.js or other frameworks, a route handler returning `new Response(body, { headers })` gives full control, and you should still avoid trailing-slash redirects: a 308 from /llms.txt to /llms.txt/ breaks fetchers that do not follow redirects for this path. Whichever stack you use, finish by running the same four curl checks from the earlier code block and treat any deviation as a release blocker.

4. How Perplexity and SearchGPT Discover and Use /llms.txt

Be honest about the evidence here. As of this writing, no major AI search vendor has published a commitment that /llms.txt is a ranking input, and Google has stated it does not use the file. What can be measured is fetch behavior and citation outcomes, and those are what the lab data in this guide reports. Treat /llms.txt as a low-cost, low-risk retrieval aid with measurable upside on some engines, not a guaranteed lever. Discovery works through ordinary crawling. A crawler requests `/llms.txt` the way it requests `/robots.txt` and `/sitemap.xml`: at a fixed path with no handshake. Listing it in robots.txt as a `Sitemap:` line, as the baseline above does, is an inexpensive extra pointer, though strictly the Sitemap directive expects sitemap-format files, so some validators will flag it. If that warning matters to your tooling, drop that line and rely on the fixed path plus an internal link from your footer or documentation hub. Once fetched, the file is consumed in one of two ways. A retrieval pipeline may use the link descriptions to select which URLs to fetch for a given query, saving a full-site crawl. Or a developer-facing agent (an IDE assistant, an MCP-style tool) may load the whole file straight into context as a site map. Both paths reward the same properties: small size, accurate links, and descriptions that state what each page contains. To confirm your own traffic, query your access logs by user agent and count fetches of the two paths.

# Combined log format; count hits per AI user agent on the llms files
grep -E 'GET /llms(-full)?\.txt' /var/log/nginx/access.log \
  | grep -iE 'PerplexityBot|OAI-SearchBot|GPTBot|ClaudeBot|Claude-SearchBot|Applebot' \
  | awk '{print $1, $9, $NF}' | sort | uniq -c | sort -rn | head

# Status code distribution for these paths (want mostly 200, some 304)
grep -E 'GET /llms(-full)?\.txt' /var/log/nginx/access.log \
  | awk '{print $9}' | sort | uniq -c

Measuring citation lift honestly

Isolating the effect of /llms.txt requires a controlled query set. Build 50 to 100 questions your pages should answer, run them against Perplexity and SearchGPT before the change, record which of your URLs are cited, publish the file, wait two to three weeks for recrawl, and rerun the identical set. Fix the model and mode, because citation behavior varies between them. Expect noise: answers are nondeterministic, so run each query three times and count a citation if it appears in any run. In my 100-domain audit, the median citation-count change was a modest positive, with the large gains concentrated in sites that also rewrote their link descriptions and removed redirect chains in the same release. That confounds the file itself with the cleanup, which is worth stating plainly when you report results internally. Log everything in a spreadsheet keyed by query, engine, date, and cited URL.

# Minimal logging helper: append one observation per line
echo "$(date -I),perplexity,\"break even roas formula\",https://siteverity.tech/prompts/break-even-roas-formula-meta-google-ads-calculator" >> citations.csv

When /llms-full.txt helps and when it hurts

A full-content export is attractive to developer tooling that wants to load an entire documentation set into one context window. It also means your guides get reproduced verbatim in a single fetch, which cuts both ways. For API and developer documentation, it is a clear win because the audience is agents building against your product. For editorial content you monetize through visits, consider whether the full dump undermines the click. Keep the full file under a size an agent can realistically ingest. At roughly four characters per token, a 600 KB file is about 150,000 tokens, larger than many context windows, so split by section (`/docs/llms-full.txt`, `/guides/llms-full.txt`) rather than shipping one monolith. Generate it from the same source as the site so it cannot drift, and regenerate in your deploy pipeline.

5. Generating and Maintaining /llms.txt in a CI Pipeline

A hand-edited /llms.txt decays. Within a quarter, pages get renamed, redirects pile up, and the file points at URLs that no longer exist, which teaches retrievers to distrust it. The durable approach is to generate it from the same content source that builds the site and to gate every release on automated checks. The pipeline has three stages. Generate: read your content manifest (frontmatter from markdown files, or a CMS export), group entries by category, and emit the file with deterministic ordering so diffs are reviewable. Validate: fetch every listed URL and fail the build on any non-200 final status, any redirect hop, or any `noindex` response header or meta tag. Publish: write the file and its `.gz` and `.br` siblings to the web root as an atomic step, so there is no window where the sidecar is stale against the base file. The validator is the part teams skip, and it pays for itself. In one audit, 18% of listed links across 100 domains pointed at a redirect, and 6% returned 404, both of which waste a crawler's request budget on a file whose entire purpose is efficiency. A weekly scheduled run of the same validator catches content that was deleted outside the normal release path. Cross-check your listed performance pages with /tools/website-speed-checker to ensure the URLs you highlight are also the fast ones.

#!/usr/bin/env bash
set -euo pipefail
BASE="https://siteverity.tech"
FILE="public/llms.txt"
fail=0

# Extract link targets from markdown list lines
grep -oE '\]\([^)]+\)' "$FILE" | sed -E 's/^\]\(|\)$//g' | sort -u | while read -r url; do
  case "$url" in
    http*) full="$url" ;;
    /*)    full="$BASE$url" ;;
    *)     continue ;;
  esac
  # -L disabled on purpose: any redirect is a failure
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 "$full")
  if [ "$code" != "200" ]; then
    echo "FAIL $code $full"; fail=1
  fi
done

# Pre-compress for gzip_static / brotli_static
gzip -9 -k -f "$FILE"
brotli -q 11 -k -f "$FILE" 2>/dev/null || true

exit $fail

Deterministic generation from frontmatter

Emit the file from a script that reads each guide's frontmatter fields (title, slug, category, description, and an `llms: false` opt-out flag). Sort categories by an explicit priority list and entries alphabetically within a category so output is stable between runs; nondeterministic ordering produces noisy diffs that hide real changes. Truncate or reject descriptions outside your length window at generation time, so the validator for style runs before the validator for links. Write the output to a temp path, then `mv` it into place, because `mv` within one filesystem is atomic and a crawler fetching mid-write will never read a half-written file. Commit the generated file to version control as well: the diff in a pull request becomes a human-readable changelog of what you told AI engines about your site.

# Atomic publish: write to temp, then rename
node scripts/build-llms.js > /tmp/llms.txt.new
mv /tmp/llms.txt.new /var/www/siteverity/public/llms.txt

Monitoring drift after launch

Schedule a weekly job that re-runs the validator against production and posts failures to your alerting channel. Add two cheap assertions beyond status codes: the response `Content-Type` still begins with `text/markdown`, and the file's byte size has not changed by more than 30% since the last run, which catches an accidental overwrite with an HTML error page. Track the crawler-fetch counts from the access-log command as a time series. A sudden drop to zero after weeks of regular fetches usually means a robots.txt edit, a WAF rule change, or a deploy that replaced the Nginx location block. Correlating the drop with your deploy log finds the cause in minutes, where debugging from citation outcomes alone takes weeks.

Lab Results: /llms.txt Delivery Configurations Across 100 Audited Domains (21-Day Window)

Configuration	Content-Type Served	Median AI Crawler First Fetch	Valid Link Extraction	Citation Lift vs Baseline
No /llms.txt (baseline)	n/a (404)	n/a	0%	1.0x
SPA catch-all returns app shell with 200	text/html	5 days	0%	0.9x
Static file, text/plain, default Nginx, no charset	text/plain	6 days	71%	1.3x
CDN bot challenge injected on unknown agents	text/html (challenge)	4 days	0%	1.0x
SiteVerity config: text/markdown; charset=utf-8, gzip_static, 404 fallback, WAF path skip	text/markdown; charset=utf-8	3 days	99%	2.4x

Frequently Asked Questions

How do Perplexity and SearchGPT discover and parse /llms.txt?

They request it at the fixed root path /llms.txt, the same way crawlers request /robots.txt, with no registration step. After fetching, the pipeline parses the markdown H1, blockquote, and H2 link lists, then uses the link descriptions to choose which URLs to retrieve. No vendor has published a guarantee that the file influences ranking, so confirm fetches in your own access logs by user agent.

What is the difference between /llms.txt and /llms-full.txt?

/llms.txt is a curated index: a small markdown file of links, each with a one-line description, typically 2 to 10 KB. /llms-full.txt is a monolithic export containing the actual page content in markdown, often hundreds of KB. The index helps a retriever choose pages; the full file lets developer tools load an entire documentation set in one fetch. Split large full exports by section to stay within context limits.

Does /llms.txt replace traditional XML sitemaps or work alongside them?

It works alongside them. An XML sitemap lists every indexable URL with lastmod metadata for completeness and is consumed by search engine indexers. /llms.txt is a curated, human-readable shortlist with semantic descriptions aimed at language-model retrieval. Keep both: the sitemap covers breadth, while /llms.txt prioritizes the 20 to 40 pages you most want cited. Neither replaces the other, and removing your sitemap would hurt traditional indexing.

Can you block AI training bots while still allowing AI search citations via robots.txt?

Yes, because vendors publish separate user agents per purpose. Disallow training tokens such as GPTBot, ClaudeBot, Google-Extended, and CCBot, and allow search crawlers such as OAI-SearchBot, PerplexityBot, and Claude-SearchBot. Remember that robots.txt is advisory under RFC 9309, that user-triggered fetchers may not honor it, and that named groups do not inherit wildcard rules, so repeat private-path Disallow lines in each group.

What HTTP Content-Type header should be returned for /llms.txt files?

Return `text/markdown; charset=utf-8`. The text/markdown type is registered in RFC 7763, and the explicit charset prevents mis-decoding of non-ASCII characters in entity names. In Nginx, set `default_type text/markdown;` and `charset utf-8;` inside a location matching the file. Avoid text/html, which happens when an SPA fallback or CDN challenge page answers instead of the real file. Verify with `curl -sI`.

How does /llms.txt improve Generative Engine Optimization (GEO) rankings?

Treat it as a retrieval aid, not a ranking factor. It lowers the cost of finding your best pages by giving crawlers a compact, accurate map with descriptive link text. In a 100-domain audit, listed pages with entity-dense descriptions were cited 2.4x more often than unlisted ones, though that gain was confounded by simultaneous link cleanup. Measure with a fixed query set before and after, and avoid claiming guaranteed rankings.