AEO and GEO for developers: how AI search works and what to build for it

You know the classic pipeline: crawl, index, rank, click. Generative search adds three steps after ranking. It retrieves passages, synthesizes them into one answer, and cites the sources it used. Understanding those steps tells you what to build, so it is worth walking through the retrieval mechanism rather than treating it as a black box.

Classic search matched query terms to documents with lexical scoring like BM25. Retrieval for AI answers leans on dense vectors instead. Each chunk of content is converted by an embedding model into a high-dimensional vector that represents its meaning. The query is embedded the same way. The system then finds the chunks whose vectors point in the most similar direction to the query vector, measured by cosine similarity. The practical consequence: a passage can be retrieved because it means the same thing as the query, even when it shares few exact words. You are writing for semantic match, not keyword match.

An AI search system rarely retrieves for the literal question. It fans the question into several related sub-questions and retrieves for each. Google describes a user asking how to fix a weedy lawn triggering fan-out queries about herbicides, chemical-free removal, and weed prevention. A question like "endpoint security for HIPAA compliance" might fan out to "HIPAA technical safeguards," "endpoint security for healthcare," and "compliance reporting." Each pulls its own sources. This is why single-page-per-keyword targeting is weaker now, and why covering a topic thoroughly across related questions retrieves better than one narrow page.

The retriever pulls passages, not whole pages, and the model attributes an answer to a source passage. The Princeton GEO study formalized this with passage-level metrics like citation recall and precision rather than page rank. The design takeaway is concrete: every key passage should make sense on its own, lifted out of the page, with the answer near the top rather than buried under setup.

Retrieved passages are loaded into the model's context window, the model writes one conversational answer, and it embeds citations back to the source domains. The goal shifts from ranking first to being the clearest, most extractable, best-attributed passage in the evidence pool the model assembled.

There are two distinct ways an AI system can know about you, and only one of them is something you can move this quarter. Some of what an assistant "knows" is baked into its training data from an older crawl, stale by a year or more, and you shift that slowly by staying crawlable and earning references over time. The other route is live retrieval, which is the part you influence now and the focus of everything below. Sanity's field guide on serving content to agents is the most honest published treatment of the split and the uncertainty around both.

The foundational study here is GEO: Generative Engine Optimization (Aggarwal et al., KDD 2024). The team built a benchmark of 10,000 queries and tested nine content modifications. Their headline result: adding citations, quotations, and statistics boosted a source's visibility in AI answers by more than 40% across queries, while keyword stuffing made things worse. Lower-ranked pages benefited most, with pages around position 5 seeing roughly a 115% relative lift, while position-1 pages changed little.

Two things keep the 40% result in perspective. It came from a benchmark under controlled conditions, not from live ChatGPT or Perplexity output. And live signals are volatile. AI citations have shifted by as much as 60% in a single month, and serving markdown instead of HTML has produced no statistically significant change in bot traffic in some measurements. The durable read is that the research rewards the same things good editors already value, real evidence and clear writing, not a markup trick.

A lot of AEO advice is noise. Here is what to stop worrying about, with the reasoning.

• "You need an llms.txt file." Google ignores it for AI Overviews and AI Mode. And adoption is tiny: an analysis of 300,000 domains found about a 10% adoption rate with close to zero correlation to increased AI citations. It costs little to publish one and some non-Google tools will read it, but treat it as optional, not foundational.
• "You have to chunk content into tiny pieces for AI." Google states plainly that you do not. Its systems parse normal long-form pages and surface the relevant section. Write with ordinary headings and paragraphs for people.
• "Structured data is required to appear in AI answers." It is not required for generative AI search per Google, though it remains worth using for rich-result eligibility and it does help non-Google engines. Add schema because it is good practice, not because it unlocks AI citation.
• "Inauthentic mentions build authority." Fabricated reviews and bulk forum posts do not work. Generative features lean on the same quality and spam systems as core Search, which filter exactly this.
• "It is a one-time setup." Roughly two-thirds of optimization targets shift on a monthly cadence in this space. The wiring is one-time, but the content work is ongoing, and chasing week-to-week citation noise is a poor use of effort. Invest in the fundamentals that stay stable.
• "Markup is a magic trick that forces a citation." Whether a passage gets cited is a synthesis step the model owns and changes often. You can make your content the cleanest, best-attributed candidate. You cannot force the outcome. Honesty about that is a credibility asset, not a weakness.

The through-line: optimizing for AI search is mostly doing fundamental SEO and content work well. The myths are shortcuts around work that does not have a shortcut.

These are the things a developer actually controls. They are grouped into four areas, ordered from foundational to advanced. Work top to bottom.

This is the layer that makes everything else moot if you get it wrong. If a crawler cannot read the content, no amount of schema helps.

• Server-render critical content. AI crawlers and many retrieval fetches do not execute JavaScript, and real-time answer generation has tight fetch timeouts. Text, headings, and answers must be in the initial HTML response, not hydrated client-side.
• Test with JavaScript disabled. Load your key pages with JS off. If the content disappears, crawlers and browser agents see the same emptiness.
• Keep load times low. Slow pages raise the risk of incomplete parsing and partial extraction. Treat Core Web Vitals as a parsing concern, not only a ranking one.
• Consider content negotiation. A route handler that checks for an Accept: text/markdown header can serve clean markdown to agents that ask for it, which cuts token waste and improves extraction accuracy. This is optional and benefits non-Google tools specifically.

A minimal content-negotiation handler in a Next.js route, assuming you have a Portable Text to markdown serializer:

// app/[slug]/route.ts
export async function GET(req: Request, {params}: {params: {slug: string}}) {
  const accept = req.headers.get('accept') ?? ''
  if (accept.includes('text/markdown')) {
    const doc = await client.fetch(docBySlugQuery, {slug: params.slug})
    const markdown = serializeToMarkdown(doc.body) // your Portable Text serializer
    return new Response(markdown, {
      headers: {'content-type': 'text/markdown; charset=utf-8'},
    })
  }
  // fall through to your normal HTML response
}

This is where you make passages extractable. Most of it is ordinary good HTML, applied with intent.

• One <h1>, a logical heading tree. Use <h2>, <h3>, and <h4> in order, for structure rather than visual size. AI systems use the hierarchy to map relationships between sections.
• Phrase headings as real questions, answer them first. Frame <h2> and <h3> tags as the natural-language questions people ask, then lead with a direct 40- to 60-word answer before the supporting detail.
• Put comparisons and specs in tables and lists. Use real <table>, <ul>, and <ol> elements. Tabular data is cited noticeably more often than the same facts in prose.
• Write specific anchor text. Descriptive link text helps systems map topic relationships and follow citation chains. Avoid "click here."
• Lead with evidence. Per the GEO research, the highest-impact content changes are citing sources, adding statistics, and quoting named experts. Build those in as a habit, not a final polish.

Schema is not required for AI answers, but it helps non-Google engines and strengthens how systems resolve your brand as an entity. Add it as part of overall SEO.

• Generate granular JSON-LD. FAQPage, HowTo, Article, Product, and Organization cover most needs. Render it server-side as a script tag, not via client JS.
• Link entities with sameAs and @id. Use @id to connect schema blocks without duplication, and sameAs to tie your organization and authors to authoritative knowledge-graph nodes like Wikipedia, LinkedIn, and Crunchbase. This strengthens entity confidence during retrieval.
• Keep mutable data dynamic. Pricing, availability, and ratings in your structured data must reflect the live values. Stale or contradictory schema can get your product dropped from recommendations to avoid hallucinated facts.

// app/products/[slug]/page.tsx
export default async function ProductPage({params}: {params: {slug: string}}) {
  const {slug} = params
  const jsonLd = await client.fetch(productJsonLdQuery, {slug})
  return (
    <>
      <script
        type="application/ld+json"
        dangerouslySetInnerHTML={{__html: JSON.stringify(jsonLd)}}
      />
      {/* page content */}
    </>
  )
}

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://example.com/#org",
  "name": "Example",
  "sameAs": [
    "https://en.wikipedia.org/wiki/Example",
    "https://www.linkedin.com/company/example",
    "https://www.crunchbase.com/organization/example"
  ]
}

The most common own-goal in this whole space: blocking the crawlers you want to cite you.

• Allow the AI bots you want citing you. Check robots.txt for accidental blocks on GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot. Blocking them means those platforms cannot cite you at all.
• Know the dual-crawler split. Some platforms separate training crawlers from live-search fetchers. Perplexity uses PerplexityBot for indexing and Perplexity-User for real-time fetches; Anthropic splits ClaudeBot for training from Claude-SearchBot for live indexing. If you want citation but not training use, this is the lever, so configure deliberately rather than blanket-allowing or blanket-blocking.
• Do not over-invest in llms.txt. Publish one if you like, but as noted above, adoption and measured impact are both low. It is not where your time goes.

You can complete every checklist above on any stack. The reason this work tends to live in structured content is that the items stop being per-page handwork and become a projection of one model. A typed schema maps to JSON-LD properties, so the structured data is a query against content you already maintain rather than markup you hand-write per page. A Portable Text serializer renders the body to clean semantic HTML and to the markdown a content-negotiation route serves. The automatic _updatedAt timestamp gives you an honest dateModified instead of a field someone forgets to update, and a scheduled Function can surface stale content for review:

*[_type == "article" && _updatedAt < dateTime(now()) - 60*60*24*90]{
  _id, title, _updatedAt
}

Two honest notes. The JSON-LD projection is a one-time developer task, and there is no native JSON-LD authoring or AEO preview in the Studio today, so a developer wires it the first time. The payoff is that after that, extending coverage is a field edit, not a template change.

This week, three things you can do.

One, load your top ten pages with JavaScript disabled. If the answer disappears, fix rendering before you fix anything else.

Two, open your robots.txt. Make sure the crawlers you want citing you (OAI-SearchBot, Claude-SearchBot, Perplexity-User, Googlebot) are not blocked by accident.

Three, pick one high-value page and rewrite its opening as a 60-word direct answer. Ship it. Watch what happens over the next month.

The rest of this series will get into entity graphs, measurement, agent delivery, and multi-consumer content modeling. Start with the three above and you will be ahead of most of the industry that is still filing tickets about llms.txt.

This is the first in a planned series. Future articles will go deeper on the pieces this one only introduced:

• Entity graphs and sameAs in practice, on modeling products, solutions, and authors as typed references that compose into a knowledge graph AI systems can resolve.
• Measuring AI visibility honestly, on what you can and cannot track, third-party citation tools, and why there is no AI-specific Search Console report.
• Serving content to agents, a deeper build on content negotiation, clean markdown routes, and the agent-as-buyer use case, extending Sanity's field guide.
• Content modeling for multiple consumers, on designing one schema that serves your website, your apps, and AI systems at once.

Below are some terms you may encounter that are specific to AI search. General SEO vocabulary the article also uses, like JSON-LD, sameAs, semantic HTML, and Core Web Vitals, is left out here on purpose, because those predate the shift to answer engines and are not unique to AEO.

Structuring and formatting content so AI tools can read, trust, and cite it as a direct answer to a question.

An autonomous AI system that visits and acts on a site, comparing options or completing a task, rather than only summarizing it in an answer.

The user agents AI platforms use to fetch web content, including GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot. Blocking them in robots.txt prevents those platforms from citing you.

Google's generative answer surfaces that synthesize and cite sources above or in place of the classic link list.

A system that returns one synthesized answer instead of a ranked list of links. ChatGPT, Perplexity, and Google's AI Mode are answer engines. "Generative engine" is used interchangeably.

Leading a section with a self-contained direct answer, often 40 to 60 words, so a retriever can lift it cleanly without the surrounding context.

How often AI answers cite your source. It replaces click-through as the headline success metric in AI search.

Two metrics from the Princeton GEO study. Recall is the share of your eligible content that gets cited. Precision is the share of citations that are accurate.

Serving a clean markdown version of a page to clients that request it with an Accept: text/markdown header, instead of making them parse HTML.

The amount of text a model can hold at once while it writes an answer. It is why concise, self-contained passages are easier to retrieve and cite than long, padded ones.

Finding content by meaning, using vector embeddings and similarity scoring, rather than by matching exact keywords (the lexical approach behind classic search).

A platform running separate bots for training and for live search, such as PerplexityBot for indexing and Perplexity-User for real-time fetches, or ClaudeBot for training and Claude-SearchBot for live indexing.

The tendency of generative engines to favor independent third-party sources, like reputable publications and forums, over a brand's own pages.

A numeric vector that represents the meaning of a chunk of text. Pages and queries are both embedded so the system can match them by meaning rather than by words.

The preference AI answers show for recently updated content. Stale pages can lose their citation footprint to newer sources with more current data.

The broader practice of getting a brand cited or recommended when an AI system generates an answer. In this field, GEO means generative engine optimization, not geolocation.

The benchmark of 10,000 queries introduced in the Princeton GEO study to measure how specific content changes affect AI citation.

Anchoring a model's answer in retrieved, verifiable sources rather than its training alone. The mechanism is RAG, below.

An LLM producing plausible but unverified or false content. Reducing it is the main reason engines ground answers in retrieved sources.

A proposed markdown file at a site root meant to give AI systems an index of key content. Widely discussed, low in adoption, and ignored by Google Search.

Retrieval and citation operate on individual passages, not whole pages, so the passage is the real unit of visibility.

An engine expanding a single question into several related sub-queries and retrieving sources for each before synthesizing one answer.

The architecture behind AI search. The engine retrieves live passages and feeds them to the model so the answer is grounded in verifiable sources rather than only the model's training weights.

How often you are cited across AI platforms relative to competitors. The cross-platform analog of share of search.

The step where the model combines the retrieved passages into a single written answer and attributes the sources it used.

The two routes to being represented in AI answers. Training data is baked into the model's weights from an older crawl and changes slowly. Retrieval happens live at answer time and is the route you can influence now.