Markdown Pages for AI Agents

In February 2026, Cloudflare launched a feature called Markdown for Agents, which automatically converts HTML pages to Markdown and serves them to AI crawlers. The pitch: leaner content costs fewer tokens to ingest, so AI systems will use it more readily. The evidence does not support the claim, and Google’s position is that serving Markdown to bots is unnecessary. The longer answer matters, though, because there are specific contexts where publishing Markdown has genuine value.

What is Cloudflare Markdown for Agents?

Cloudflare Markdown for Agents sits in front of a site and intercepts requests from AI crawlers that signal a preference for text/markdown via content negotiation headers. It converts the HTML response to Markdown in real time before serving it.¹

The efficiency case is real: Cloudflare’s own blog post consumes 16,180 tokens as HTML and 3,150 as Markdown, a reduction of roughly 80%. The argument is that leaner input is cheaper for LLMs to process and therefore more likely to be used.

The feature launched in February 2026 as a Beta at no additional cost for Cloudflare Pro, Business, and Enterprise plans. It is one of several routes to the same outcome: WordPress plugins, static site generators, and manual .md routes can all produce Markdown versions of pages without Cloudflare.

Does serving Markdown improve AI citations?

No evidence supports a citation benefit. Otterly.ai ran a controlled 14-day experiment comparing HTML and Markdown versions of the same pages, giving both equal placement and discoverability on the site. AI crawlers recorded 137 visits to the HTML pages and zero visits to the Markdown pages. No AI platform cited a .md URL across the full test period.²

SALT.agency, analysing behaviour across hundreds of thousands of domains, found no meaningful correlation between serving Markdown to AI crawlers and increased citations in AI-generated answers.³

Google’s Search Advocate John Mueller initially called the approach “a stupid idea”⁴, directing that comment at general sites rather than developer documentation. His later comments draw the distinction explicitly: Markdown is a functional tool for developer tooling, not a discovery signal for most sites.⁵⁶ He has also questioned why publishers would create pages “that no user sees,” and noted that if specialised formats meaningfully improved AI responses, AI companies “would be very vocal about that.”⁷ Serving different content to bots than to human visitors also raises a cloaking concern: Google’s spam policies prohibit serving materially different content depending on the user agent, regardless of intent.

No major AI platform, including Google, Bing, Perplexity, or OpenAI, has published guidance stating a preference for Markdown versions of live pages.

Why does the token efficiency argument appeal?

The argument has surface logic. LLMs work with tokens, and a page that costs 3,000 tokens to process rather than 16,000 is cheaper to handle. The question is whether that saving translates into a search or citation benefit.

For AI search crawlers (GPTBot, ClaudeBot, BingbotLLM), it does not. These systems index pages at crawl time, not at inference time. The LLM generating an answer does not re-fetch your page when a user submits a query: it draws on previously indexed content. Token savings at crawl time do not translate into citation priority.

The argument is more coherent for agentic use cases, where an LLM calls your page live during a session (for example, an agent researching a purchase or reading documentation to write code). Even there, no confirmed citation benefit has been established. Mueller’s framing: “OF COURSE they can read HTML just fine, so this is imo more of a temporary crutch, perhaps to save some tokens.”⁸

Which agents actually request Markdown matters here. Checkly tested seven AI coding tools by checking the Accept headers each sends when fetching a URL. Three send Accept: text/markdown: Claude Code, Cursor, and OpenCode. Four do not: OpenAI Codex, Gemini CLI, GitHub Copilot, and Windsurf.⁹ These are coding tools, not AI search crawlers. The distinction is significant: search crawlers (GPTBot, ClaudeBot, BingbotLLM) build an index at crawl time; coding agents make live page requests at inference time. Content negotiation addresses the coding agent audience, not the search crawler one.

When does publishing Markdown make sense?

The distinction Mueller draws is between discovery and functionality. Discovery is how pages get found via search: a ranking, an AI citation, a link followed by a crawler. Functionality is what happens once someone, or an agent, has already reached the page and is trying to do something with it.⁸

Markdown is a functionality tool. It does not influence discovery. Publishing Markdown versions of pages will not improve your rankings, your AI Overviews presence, or your citation rate in Perplexity. It may improve the experience of users already on your site who are using an AI coding tool to work with your content.

The use cases where this holds:

• Developer documentation and API references. AI coding tools such as Cursor, GitHub Copilot, and Claude Code work directly with reference material at inference time. Clean Markdown reduces noise and improves the accuracy of generated code. This is the use case Mueller cited when explaining why developers.google.com publishes .md.txt versions of its pages: the audience is AI coding assistants, not search crawlers.⁸
• Knowledge bases where agents are the primary audience. If your site is built for direct agent consumption (tool documentation, integration guides, structured reference content), Markdown versions have functional value for that audience.

The use cases where it does not help:

• E-commerce product pages. A Markdown version of a product specification does not improve AI shopping citations. Structured data and catalogue quality are what matter.
• Editorial and news content. AI citation of editorial content is driven by authority, accuracy, and structure. Serving Markdown does not change any of those signals.
• General SEO or GEO. Publishing Markdown does not improve rankings or AI visibility. Accuracy, clear structure, direct answers, and credible authorship are what earn citations.

What are the risks?

Cloaking. The risk depends on how Markdown is delivered. Content negotiation at the same canonical URL via the Accept: text/markdown header is standard HTTP (RFC 9110; text/markdown has been a registered MIME type since RFC 7763 in 2016). Serving the same content in a different format from the same URL is not cloaking. The risk arises with separate .md URLs: when a different URL serves stripped content that human visitors never see, that is cloaking-adjacent regardless of intent. Cloudflare’s header-based conversion operates at the same URL; manually creating parallel .md endpoints does not. Auto-conversion tools that strip navigation and CTAs create a different representation, which is where the Google spam policy concern becomes concrete.

Incorrect plugin implementations. Some WordPress plugins advertise “Markdown for AI” but create separate .md URLs rather than reading the Accept header on the canonical URL. When an agent sends Accept: text/markdown to https://example.com/page/, it receives HTML because the plugin never checks the header. Publishers using these tools should verify their site responds with Markdown at the canonical URL when the header is present, not just that a parallel .md URL exists.¹⁰

Maintenance overhead. Two versions of every page means double the editorial work to keep content in sync, double the edge cases to test, and more surface area for inconsistencies. There is no confirmed return on that investment.

Misallocation. Teams spending time on Markdown conversion are not spending it on the content quality signals that actually influence AI citations: factual accuracy, clear structure, direct answers, and credible authorship.

Frequently asked questions

Does Cloudflare Markdown for Agents improve AI search visibility?
No evidence supports this. In controlled testing, AI crawlers visited HTML pages and ignored the Markdown equivalents, and no AI platform cited a .md URL. Google, Bing, and other major platforms have not published guidance suggesting they prefer Markdown versions of live pages.

Is serving Markdown to bots considered cloaking?
It depends on how Markdown is delivered. Header-based content negotiation (same canonical URL, Accept: text/markdown, same underlying content) is standard HTTP and is not cloaking. Separate .md URLs serving stripped content that human visitors never see is where the cloaking risk is real, regardless of intent. Google’s position is that its systems can read HTML without difficulty and that serving different content to bots raises a spam policy concern.

Should I enable Cloudflare Markdown for Agents?
Only if your site is developer documentation or a knowledge base where AI coding tools are a meaningful part of your audience. For most sites, the feature offers no citation benefit and introduces cloaking risk and maintenance overhead.

Why does Google publish Markdown files on developers.google.com?
Because AI coding tools work more accurately when reading clean reference material. Google publishes these for the benefit of developers using AI coding assistants, not for search visibility. It is a functional publishing decision specific to developer documentation.

How does this relate to llms.txt?
Both are functionality tools rather than discovery tools: they affect what happens once an agent reaches your site, not whether your site gets found. See llms.txt for the index file convention. Neither improves AI search citations.

Footnotes

1. Introducing Markdown for Agents — Cloudflare Blog ↩

2. GEO Experiment: Markdown vs. HTML — Otterly.ai ↩

3. Markdown-only pages for AI crawlers are a waste of time — SALT.agency ↩

4. Google’s Mueller calls Markdown for bots “a stupid idea” — Search Engine Journal ↩

5. Mueller explains why Google uses Markdown on dev docs — Search Engine Journal ↩

6. Google adds Markdown files to help docs — Search Roundtable ↩

7. Google’s Mueller questions need for LLM-only Markdown pages — Search Engine Journal ↩

8. John Mueller on llms.txt and Markdown for agents — Bluesky, via Lily Ray ↩ ↩² ↩³

9. State of AI agent content negotiation — Checkly ↩

10. Some SEO plugins claim Markdown for AI but ignore the Accept header — Roots.io ↩