Migrate from the Embeddings Index API to Dataset Embeddings

Replace the deprecated Embeddings Index API with Dataset Embeddings and GROQ: export your index, enable embeddings, rewrite your queries, and cut over.

The Embeddings Index API is deprecated, along with the @sanity/embeddings-index-cli package and the @sanity/embeddings-index-ui Studio plugin. They receive no new features and no fixes. This guide walks through replacing them with Dataset Embeddings and GROQ, from exporting your current index configuration to deleting it once the new path is live.

You'll need:

• A project role with permission to manage the legacy embeddings indexes, for the export in Step 1 and the delete in Step 5. See roles and permissions.
• The Sanity CLI, sanity 5.18.0 or later. The sanity datasets embeddings commands ship in @sanity/cli 6.2.0, which sanity 5.18.0 is the first release to require. Run commands as npx sanity@latest to stay on the current version. To call the legacy endpoints over HTTP instead, you need an API token.
• A server-side application using @sanity/client and a read token. A read token is all your application needs to query.
• Your project ID, dataset name, and the name of every legacy index you're replacing.
• A set of real search queries to test against. Pull these from your application's search logs or analytics.

Also pull your request logs before you start. They show which indexes are still being called, how often, and from where, which is how you catch a caller you forgot about. Self-serve plans can export the last seven days from the Usage section of project settings. That export is capped at 1 GB: a project producing more log data than that within the window gets a truncated file covering less than seven days, so check the range it actually covers before you draw conclusions from it. Enterprise projects can have logs delivered to a storage bucket. Note that the search strings themselves are sent in a POST body and aren't recorded, so your test queries have to come from your own logs.

Today you create a named index, POST a search string to a query endpoint, and get back document IDs and scores. Then you query GROQ a second time to fetch the documents.

Dataset Embeddings replace both requests with one. text::semanticSimilarity() is a GROQ function you call inside score(), so filtering, keyword matching, boosting, ordering, slicing, and your projection all happen in the same query, and you get documents back. There is no index to create, poll, or keep current.

This is not a drop-in swap. Results will not rank identically, because the embedding model, the chunking, and the score scale all differ. Named indexes no longer exist. Reference expansion stops working inside the embedding projection, though -> still works everywhere else in your queries. Check what ports cleanly and what doesn't before you start.

Each legacy behavior falls into one of three groups: it ports directly, it ports with a code or content change, or it doesn't port at all. Check where yours lands before you plan the work.

The examples in this guide all use the same index: public-articles, filtered to _type == "article" && searchable == true, projecting {title, body, "categoryTitle": category->title}.

Fetch every index on the dataset and save the filter, projection, and indexName for each one. You need all three to rebuild the behavior. Store the response as JSON in version control or wherever your team keeps infrastructure config, not just in a scratch file. You'll want it again in step 5. Two things about the response: Sanity stores your projection with _type auto-prepended, so the saved config won't be byte-identical to what you created; and the projection string may contain an unescaped newline that trips strict JSON parsers. Save the raw text, or parse tolerantly.

curl https://YOUR_PROJECT_ID.api.sanity.io/vX/embeddings-index/production \
  -H "Authorization: Bearer $SANITY_API_TOKEN"

Then search your codebase for the query URL, /embeddings-index/query/, and note every caller and everything downstream that reads the response. Anything reading value.documentId or comparing score to a number needs to change. Cross-check the list against your request logs: an index that's taking traffic but doesn't appear in your codebase means there's a caller somewhere you haven't accounted for.

You get one projection per dataset. The index in the running example covers one document type, so the projection does too. If you're replacing several indexes, merge their fields into a single conditional projection. See type-specific projections.

Type-level scoping from your old filters can move into the projection, because document types you don't list are not embedded. Value-level conditions, like searchable == true, cannot. Those stay at query time.

{
  _type == "article" => {
    title,
    body
  }
}

Keep the projection tight. Every field you add grows each document's embedding, slows generation and recomputation, and adds noise that competes with the signal your users are searching for. Leave out fields that change often but carry no meaning for search, because each change triggers a recomputation.

Field names carry semantic weight. {"musicalGenre": category} tells the model to read "classical" as music rather than engineering.

Documents are chunked before embedding, and there's a cap of 10 chunks per document (subject to change). Content past the cap is dropped. If you're embedding long body fields, scope the projection.

Expanding references does not work in embedding projections. Only what's in the document. In testing, the legacy index did embed category->title, but as a weak signal—a few hundredths of cosine similarity, outweighed by the document's own title and body. Losing it mostly reshuffles the tail of your results, not the top hit. Materialize the field only if that referenced value is genuinely important to how users search.

npx sanity@latest datasets embeddings enable production \
  --projection '{_type == "article" => {title, body}}' \
  --wait

pnpm dlx sanity@latest datasets embeddings enable production \
  --projection '{_type == "article" => {title, body}}' \
  --wait

yarn dlx sanity@latest datasets embeddings enable production \
  --projection '{_type == "article" => {title, body}}' \
  --wait

bunx sanity@latest datasets embeddings enable production \
  --projection '{_type == "article" => {title, body}}' \
  --wait

--wait blocks until the initial generation finishes. Without it, the command returns immediately and generation continues in the background. On a large dataset this takes a while (for a few dozen documents it's under a minute; budget much more for large datasets).

Check the status at any point:

npx sanity@latest datasets embeddings status production

pnpm dlx sanity@latest datasets embeddings status production

yarn dlx sanity@latest datasets embeddings status production

bunx sanity@latest datasets embeddings status production

The status is updating, ready, or error. Don't send production traffic until it reads ready. Querying a dataset without embeddings enabled returns an error.

To do this over HTTP instead:

PUT /projects/:projectId/datasets/:name/settings/embeddings HTTP/1.1
Content-Type: application/json

{
  "enabled": true,
  "projection": "{_type == \"article\" => {title, body}}"
}

The endpoint returns 202 Accepted and generates asynchronously.

The POST to the index and the follow-up query that fetched the documents collapse into a single GROQ query.

Before:

const results = await client.request({
  url: '/embeddings-index/query/production/public-articles',
  method: 'POST',
  body: {query: searchText, maxResults: 10, filter: {type: ['article']}},
})

const ids = results.map((result) => result.value.documentId)
const documents = await client.fetch(`*[_id in $ids]{_id, title, slug}`, {ids})

After:

*[_type == "article" && searchable == true]
  | score(text::semanticSimilarity($searchText))
  [0...10] {
    _id, _type, title, slug, _score
  }

The index filter becomes the GROQ filter, filter.type becomes _type, and maxResults becomes the slice.

Query changes has the full client code, the new response shape, a compatibility adapter for callers that still expect the old format, and when to add keyword matching.

curl -X DELETE https://YOUR_PROJECT_ID.api.sanity.io/vX/embeddings-index/production/public-articles \
  -H "Authorization: Bearer $SANITY_API_TOKEN"

Sanity removes the webhook it created for the index automatically when you delete the index. If you added any webhooks of your own to keep the index current, remove those.

Two requests: one to the embeddings index, one to fetch the documents.

const results = await client.request({
  url: '/embeddings-index/query/production/public-articles',
  method: 'POST',
  body: {
    query: searchText,
    maxResults: 10,
    filter: {type: ['article']},
  },
})

// [{score: 0.83, value: {documentId: 'abc123', type: 'article'}}]

const ids = results.map((result) => result.value.documentId)
const documents = await client.fetch(`*[_id in $ids]{_id, title, slug}`, {ids})

One request. The filter, the scoring, the ordering, the slice, and the projection all live in the same query.

import {createClient} from '@sanity/client'

const client = createClient({
  projectId: process.env.SANITY_PROJECT_ID,
  dataset: 'production',
  apiVersion: '2026-08-21',
  token: process.env.SANITY_API_READ_TOKEN,
  useCdn: false,
  perspective: 'published',
})

export async function searchArticles(searchText: string, maxResults = 10) {
  const limit = Math.min(Math.max(Math.trunc(maxResults) || 10, 1), 50)

  return client.fetch(
    `*[_type == "article" && searchable == true]
      | score(text::semanticSimilarity($searchText))
      [0...$limit] {
        _id, _type, title, slug, _score
      }`,
    {searchText, limit},
  )
}

The old index filter, searchable == true, is now in the GROQ filter. Nothing carries it over for you.

text::semanticSimilarity() is only valid as an argument to score(). Using it anywhere else returns an error. score() already sorts results by _score descending, so you don't need an explicit order().

maxResults becomes the upper bound of a slice. There is no default limit in GROQ, so always bound your results. Note that .. is inclusive and ... is exclusive, which matters when you convert a maxResults integer into a slice bound.

You get documents back, not pointers to documents:

[
  {
    "_id": "article-auth-guide",
    "_type": "article",
    "title": "Authentication guide",
    "slug": {"current": "authentication-guide"},
    "_score": 8.341205
  }
]

_score is a unitless, opaque ranking value. It orders results within one query and nothing more. It is not comparable across queries, and it is not on the same scale as the score the old API returned. If your code has a line like results.filter(r => r.score > 0.75), delete it and control the result count with the slice instead.

Semantic queries automatically include an _embeddings field on each result, carrying the text fragments that drove the match along with their source fields and character offsets. If your query uses an explicit projection, add _embeddings to it to keep the field. Good for highlighting search results, and good for working out why something ranked where it did:

*[_type == "article" && searchable == true]
  | score(text::semanticSimilarity($searchText))
  [0...$limit] {
    _id, _type, title, slug, _score, _embeddings
  }

{
  "_embeddings": [
    {
      "fragments": ["OAuth 2.0 provides a secure delegation protocol..."],
      "fields": ["body"],
      "startPositions": [0],
      "endPositions": [74],
      "score": 8.341205
    }
  ]
}

If several callers read the legacy format and you'd rather not change them all at once, rebuild the old shape in the projection. No adapter code needed:

*[_type == "article" && searchable == true]
  | score(text::semanticSimilarity($searchText))
  [0...$limit] {
    "score": _score,
    "value": {
      "documentId": _id,
      "type": _type
    }
  }

The shape matches, but score no longer means what it did. Treat this as a stepping stone, not a destination, and make sure nothing downstream is thresholding on that number.

Semantic search alone handles conceptual queries well. It's weaker on proper nouns, product codes, brand names, and part numbers, where the exact string is the point. Add a match expression alongside the semantic one inside score(), wrapped in boost() to set how much weight the keyword signal carries:

*[_type == "article" && searchable == true]
  | score(
      boost([title, body] match text::query($searchText), 0.5),
      text::semanticSimilarity($searchText)
    )
  [0...10] {
      _id, _type, title, slug, _score
    }

Each expression contributes to _score independently, and a document doesn't need to match both to appear. boost() sets the balance. Keyword matches on short fields like title can outweigh strong semantic matches on long fields like body, because each matching term is a bigger share of a short field, so start the keyword weight low and tune from there.

Start semantic-only. Add keyword matching when you can point at queries it fixes.

Your new results will not match the old ones exactly. The goal is to confirm they're as good or better, not that they're identical.

Use the queries you gathered in before you start. Twenty to thirty is enough. Include the high-volume ones, the long-tail ones, queries that should return nothing, and any query containing an exact name, code, or brand. Those last ones are where semantic-only search tends to fall short, and where keyword matching helps most.

Run each query through both paths and look at the top few results side by side. You're checking for two things: results that got worse, and results that appeared out of nowhere. The second one usually means a filter didn't make it across.

• Content appearing that the old index excluded. The legacy filter is missing from your GROQ query. Check your perspective too. On API versions from 2025-02-19 onward the default is published, but on older versions it's raw, which returns drafts alongside published documents. Set perspective: 'published' explicitly if the old index only covered published documents.
• Exact names and codes ranking poorly. Add keyword matching.
• Recently edited content returning stale matches. Embedding updates are asynchronous and debounced. The lag is usually under a minute but can be longer on large or busy datasets. If your product needs fresher results than that, measure the real lag before you commit.
• Everything returning an error. Check that the embeddings status is ready, and that you're calling text::semanticSimilarity() inside score().

Run both paths against production traffic for a short period if you can, comparing outputs without acting on the new one. Then switch, keeping the old index in place so a rollback is a config change rather than a rebuild. On a large or business-critical search integration, talk to your account team before you cut over.

After the switch, watch error rate, latency, and your zero-result rate. Give it a week at full traffic, then confirm in your request logs that the legacy index is taking no traffic before you delete it. Check that your log export actually covers the whole week before you treat quiet logs as evidence: a truncated export looks exactly like an idle index.

Yes, if you want your search to keep working. The Embeddings Index API is deprecated and should be replaced with Dataset Embeddings. It receives no fixes or new features.

The Sanity CLI. See sanity datasets embeddings.

There's no replacement for reference population in AI Assist and Agent Actions yet, and none for the similar-document search in the Studio plugin. If you depend on either, contact us or post in the Sanity Community so we can factor your use case into what comes next.

Embeddings generation is included on all plans. Embeddings queries are metered, with a monthly allowance that varies by plan and overage pricing above it. Semantic search itself is included. See pricing for current allowances, and if you're on an enterprise plan, ask your account team for a forecast based on your actual query volume.

• Dataset Embeddings. Enabling, projections, chunking, querying, and result metadata.
• Search text content with GROQ. Filtering, scoring, BM25, boosts, and pagination.
• Datasets CLI reference. The sanity datasets embeddings commands.
• Embeddings Index API reference. The deprecated API, for reference while you migrate.