Serve agents markdown, not your markup

For technical leaders: agent traffic to your content is growing whether you serve it or not, and the alternatives are worse. You can leave it to each vendor's HTML-to-markdown converter (lossy, and different in every pipeline, so no two agents read the same page), buy an "AEO platform" that mirrors your pages into its own index, or hand-maintain a parallel markdown export that drifts from the site within a quarter. This pattern makes markdown a render target of the content you already have, so both formats update on publish. It covers read access only, though: not search, not retrieval, and not a substitute for an API when agents need to write back.

A working implementation of the pattern, twice. The Next.js app and the Astro app render the same Content Lake dataset as HTML pages and as markdown routes, sharing one GROQ query set and one Portable Text serializer package. That duplication is the point: it shows the pattern lives in your content structure, not in a framework feature. Port it to whatever you run.

This guide argues the pattern and quotes the repo as evidence; it is not a step-by-step tutorial. If you want the build walked through file by file, the Markdown Routes with Next.js course on Sanity Learn covers the Next.js half in eight lessons.

Developers running documentation sites, marketing sites, or anything agents cite. Also the person who just watched a coding assistant hallucinate an answer about their product because the real answer was buried in client-rendered HTML.

• Content Lake: structured content queried over an API. Markdown output works because the content lives as data, not as rendered pages.
• GROQ: one query set feeds both apps and both formats, wrapped in defineQuery so TypeGen types every fetch. Navigation routes fetch titles and summaries; article routes fetch full content.
• @portabletext/markdown: converts Portable Text to markdown, with custom renderers for code blocks, images, and callouts.
• Next.js Route Handlers and rewrites: internal /md/ routes serve markdown; next.config.ts rewrites map .md URLs and Accept headers onto them.
• Astro endpoints and middleware: .md.ts endpoint files serve the explicit URLs; middleware handles header negotiation.

Run this against your own docs:

curl -H "Accept: text/markdown" https://yoursite.com/docs/quickstart

You get HTML. Not because the server considered the request and declined, but because nothing is listening. The Accept header is the oldest content negotiation mechanism on the web, and most sites ignore it.

So the agent parses what it gets: the page shell, the cookie banner markup, the nav with every sibling page in it, the footer, the JSON payload the framework embeds for hydration, and somewhere in there, your actual content. On a typical framework-rendered docs page that is roughly a 30-to-1 noise-to-content ratio by bytes [MEASURE]. The agent pays tokens for all of it, and every token spent on your div structure is context it cannot spend on your answer. When an agent decides which of three vendors' docs to cite, the site that costs 40,000 tokens to read loses to the one that costs 1,500.

The better agent pipelines know this, so Claude, ChatGPT, and most fetch tools convert HTML to markdown before the model reads it. That conversion is a heuristic guess at your page, and it loses things you never see it lose: main-content extraction keeps the cookie banner and drops the sidebar you cared about, code blocks come out as bare fences with no language and no filename, callouts flatten into blockquotes indistinguishable from quotations, and a client-rendered page converts to an empty shell because the fetcher never ran your JavaScript. Each vendor's converter guesses differently, so there is no canonical text of your page; there are five. Serving markdown yourself replaces their guess with your render. You decide what the agent reads, and every agent reads the same thing.

The old way to fix this was a parallel export: a script that walks the site and writes markdown files somewhere, on a schedule, until someone changes the page template and nobody updates the script. The content operations way treats markdown as one more render target: generated on request, from the queries that already build the HTML.

                    Content Lake
                    (one dataset)
                          │
              ┌───────────┴───────────┐
              │                       │
        Next.js app              Astro app
              │                       │
   ┌──────────┼──────────┐   (same three surfaces)
   │          │          │
 HTML      markdown   discovery
/docs/...  same URL   /llms.txt
           + .md or   /sitemap.md
           Accept hdr

The monorepo makes the sharing explicit:

agent-ready-content/
├── apps/
│   ├── studio/          # Sanity Studio
│   ├── next/            # Next.js: pages + /md routes + rewrites
│   └── astro/           # Astro: pages + .md endpoints + middleware
└── packages/
    ├── schema/          # section, article, code, callout types
    └── agent-markdown/  # serializers, builders, GROQ queries

Everything that defines the markdown output lives in packages/agent-markdown: the Portable Text serializers, the document builders, and the GROQ queries. The apps contribute routing and nothing else. That is why the Astro app is small enough to read in one sitting, and why porting to Remix or SvelteKit is a routing exercise.

Converting Portable Text to markdown is one dependency, not a project. @portabletext/markdown handles paragraphs, headings, lists, marks, and links without configuration; your custom block types are the part that needs decisions, one renderer each. A type without a renderer drops out of the output silently, so cover every type in your schema. From packages/agent-markdown/src/serializers.ts:

import {portableTextToMarkdown} from '@portabletext/markdown'
import type {TypedObject} from '@portabletext/types'
import imageUrlBuilder from '@sanity/image-url'
import type {SanityClient} from '@sanity/client'
import type {CalloutBlock, CodeBlock, ImageBlock} from './types'

export function createConverter(client: SanityClient) {
  const builder = imageUrlBuilder(client)

  return function convertToMarkdown(blocks: TypedObject[]): string {
    return portableTextToMarkdown(blocks, {
      types: {
        code: ({value}: {value: CodeBlock}) => {
          const {language = '', filename, code} = value
          const lang = filename ? `${language}:${filename}` : language
          return `\`\`\`${lang}\n${code}\n\`\`\``
        },
        image: ({value}: {value: ImageBlock}) => {
          const url = builder.image(value.asset).url()
          const caption = value.caption ? `\n\n*${value.caption}*` : ''
          return `![${value.alt || ''}](${url})${caption}`
        },
        callout: ({value}: {value: CalloutBlock}) => {
          const style = value.style || 'note'
          const content = portableTextToMarkdown(value.content)
          return `> [!${style.toUpperCase()}]\n> ${content.replace(/\n/g, '\n> ')}`
        },
      },
    })
  }
}

Two of these choices are worth defending. Callouts become GitHub Flavored Markdown alerts (> [!WARNING]) because that is the dialect agents see most in training data and in the repos they work with. And code blocks carry the filename in the fence info string (typescript:src/lib/example.ts), so an agent quoting your example knows where the file lives. Structure you modeled in the schema survives into the markdown. That is the whole argument for generating markdown from structured content instead of scraping it from HTML.

Next.js cannot put a dynamic segment inside a filename, so there is no [article].md/route.ts. The starter uses internal /md/ routes and lets rewrites handle the public URLs. From apps/next/src/app/md/[section]/[article]/route.ts:

import {NextRequest} from 'next/server'
import {ARTICLE_WITH_NAV_QUERY, prevNext} from '@agent-ready/markdown'
import {client} from '@/sanity/client'
import {buildArticleMarkdown} from '@/lib/markdown'
import {SITE_URL} from '@/lib/config'

export async function GET(
  request: NextRequest,
  {params}: {params: Promise<{section: string; article: string}>},
) {
  const {section: sectionSlug, article: articleSlug} = await params

  try {
    const data = await client.fetch(
      ARTICLE_WITH_NAV_QUERY,
      {sectionSlug, articleSlug},
      {stega: false, tag: 'md.article'},
    )

    if (!data.article) {
      return new Response('Article not found', {status: 404})
    }

    const markdown = buildArticleMarkdown(data.article, {
      canonicalUrl: `${SITE_URL}/docs/${sectionSlug}/${articleSlug}`,
      ...prevNext(data.allArticles, articleSlug),
    })

    return new Response(markdown, {
      headers: {
        'Content-Type': 'text/markdown; charset=utf-8',
        'Cache-Control': 'public, max-age=60, stale-while-revalidate=300',
      },
    })
  } catch (error) {
    console.error('Markdown route error:', error)
    return new Response('Internal server error', {status: 500})
  }
}

Notice what the fetch call does not have: a type parameter. The queries are wrapped in defineQuery and Sanity TypeGen generates result types from them, so data.article is fully typed here without a manual generic.

Three more details are easy to miss and painful to debug later. stega: false strips the invisible characters that visual editing embeds in strings; leave it on and your markdown carries hidden metadata that confuses diff tools and wastes tokens. The tag rides on requestTagPrefix: 'agent-content' from the client config, so markdown traffic shows up separately in your request logs and you can measure agent adoption instead of guessing at it. And the markdown opens with a canonical URL line, so an agent that got the file through a scrape or a copy-paste can still find its way back.

The public contract is three behaviors on one URL, configured in apps/next/next.config.ts:

const nextConfig: NextConfig = {
  rewrites: async () => ({
    beforeFiles: [
      // Explicit .md URLs, no header required
      {
        source: '/docs/:section/:article.md',
        destination: '/md/:section/:article',
      },
      {
        source: '/docs/:section.md',
        destination: '/md/:section',
      },
      // Accept header negotiation
      {
        source: '/docs/:section/:article',
        destination: '/md/:section/:article',
        has: [{type: 'header', key: 'accept', value: '(.*)text/markdown(.*)'}],
      },
      {
        source: '/docs/:section',
        destination: '/md/:section',
        has: [{type: 'header', key: 'accept', value: '(.*)text/markdown(.*)'}],
      },
    ],
  }),
}

beforeFiles matters: the rewrite runs before Next.js resolves pages, so .md requests never touch the HTML route, and requests that match nothing fall through to the normal page. Verify all three behaviors with curl:

curl http://localhost:3000/docs/getting-started/quickstart.md
curl -H "Accept: text/markdown" http://localhost:3000/docs/getting-started/quickstart
curl http://localhost:3000/docs/getting-started/quickstart   # HTML

Astro makes the explicit URLs almost free. An endpoint file named [article].md.ts serves /docs/[section]/[article].md directly, no rewrite layer needed. From apps/astro/src/pages/docs/[section]/[article].md.ts:

export const prerender = false

export const GET: APIRoute = async ({params}) => {
  const data = await client.fetch(ARTICLE_WITH_NAV_QUERY, {
    sectionSlug: params.section,
    articleSlug: params.article,
  }, {tag: 'md.article'})

  if (!data.article) {
    return new Response('Article not found', {status: 404})
  }

  return new Response(buildArticleMarkdown(data.article, articleOptions(data, params)), {
    headers: {
      'Content-Type': 'text/markdown; charset=utf-8',
      'Cache-Control': 'public, max-age=60, stale-while-revalidate=300',
    },
  })
}

Same builder, same query, imported from the same package the Next.js app uses. Header negotiation takes one middleware, from apps/astro/src/middleware.ts:

export const onRequest = defineMiddleware((context, next) => {
  const {pathname} = context.url
  const accept = context.request.headers.get('accept') ?? ''

  if (
    pathname.startsWith('/docs/') &&
    !pathname.endsWith('.md') &&
    accept.includes('text/markdown')
  ) {
    return next(pathname + '.md')
  }

  return next()
})

Header negotiation requires a server, though. A fully static Astro build can still ship the .md endpoints as prerendered files, which covers the explicit URLs and llms.txt, but Accept header handling needs an SSR adapter. If you deploy static, skip the middleware and let the explicit URLs carry the pattern. They do most of the work anyway, as the next section argues.

Serving markdown is half the pattern. The other half is telling agents it exists, because no agent tries Accept: text/markdown against a random site on faith.

The llms.txt proposal standardizes the front door: a markdown file at /llms.txt with an H1, a blockquote summary, and H2 sections of annotated links. The starter generates it from the same query that builds the sitemap. From packages/agent-markdown/src/llms.ts:

export function buildLlmsTxt(site: SiteInfo, sections: SitemapSection[]): string {
  const lines = [
    `# ${site.title}`,
    '',
    `> ${site.summary}`,
    '',
    'Every page is available as markdown: append `.md` to its URL,',
    'or request it with an `Accept: text/markdown` header.',
    'A full index with per-article summaries lives at `/sitemap.md`.',
    '',
  ]

  for (const section of sections) {
    lines.push(`## ${section.title}`, '')
    for (const article of section.articles) {
      lines.push(
        `- [${article.title}](${site.url}/docs/${section.slug.current}/${article.slug.current}.md): ${article.summary}`,
      )
    }
    lines.push('')
  }

  return lines.join('\n')
}

Both apps expose it. In Next.js the route folder is literally named llms.txt (apps/next/src/app/llms.txt/route.ts), the same trick the sitemap route uses at apps/next/src/app/sitemap.md/route.ts. In Astro it is apps/astro/src/pages/llms.txt.ts. The links point at the .md URLs, not the HTML pages, so an agent that starts at /llms.txt never touches HTML at all: index, follow, read, every hop in markdown.

sitemap.md overlaps with llms.txt on purpose. The llms.txt file is the ecosystem convention, the thing tooling looks for at a known path. The sitemap is the richer in-site index, with access instructions and per-section descriptions, linked from the docs UI so humans find it too. One query feeds both, so keeping two entry points costs one extra route file.

The explicit .md URLs are the load-bearing part of this pattern: any agent that can fetch a URL can use them, today, with no convention required. llms.txt adoption across agent tooling is real but uneven, and Accept header negotiation is the least-used path of the three. Build all three anyway; they share all their code. But if you measure which path gets used first, it will be the .md suffix.

Markdown routes are cheap to cache and you should. The starter ships tiered Cache-Control headers: articles get 60 seconds fresh plus five minutes of stale-while-revalidate; the sitemap, sections, and llms.txt get five minutes plus ten, since structure changes less often than prose.

Header negotiation creates one real problem: a CDN that caches /docs/quickstart without knowing the response depends on the Accept header will serve markdown to a browser. The textbook fix is Vary: Accept, which fragments the cache into one entry per Accept value. The rewrite approach mostly sidesteps this, because the negotiated request is rewritten to a different internal route before the cache key forms, and the explicit .md URLs never vary at all. In practice you end up with about two cached variants per URL, since browsers send */* and agents send text/markdown. Test on your actual CDN before trusting this paragraph; CDN header handling is where this pattern has its sharpest edges.

pnpm create sanity@latest --template sanity-labs/starters/agent-ready-content
cd agent-ready-content
pnpm bootstrap

The bootstrap command creates the dataset, deploys the schema, seeds sample content, and configures CORS. Then pnpm dev runs the Studio, the Next.js app, and the Astro app together, and the three curl commands from this guide work against localhost.

Already have a Sanity project? You can adopt the pattern without the starter. Copy packages/agent-markdown into your repo, point its queries at your own document types (the serializers only assume Portable Text plus whatever custom types you map), and add the route files for your framework: two route handlers and a rewrite block for Next.js, or endpoint files and a middleware for Astro. The repo ships an agent skill (SKILL.md) that walks a coding agent through exactly this port against an existing schema, which is the fastest path if your content model already exists.

This guide is part of AI Content Operations, a series on building AI-powered content workflows with Sanity.

• AI translations that follow your terminology: store glossaries and style guides in Sanity, feed them to the Agent API at translation time
• An AI shopping assistant that actually checks inventory: a storefront agent grounded in live product data
• Automate your content audits with AI: recurring audits that act on findings instead of filing them
• AI email campaigns built from your structured content: campaigns assembled from the content you already model