> For AI agents: the complete Sanity documentation index is available at [https://www.sanity.io/docs/llms.txt](https://www.sanity.io/docs/llms.txt). # Migrate from the Embeddings Index API to Dataset Embeddings Replace the deprecated Embeddings Index API with Dataset Embeddings and GROQ: export your index, enable embeddings, rewrite your queries, and cut over. The Embeddings Index API is deprecated, along with the `@sanity/embeddings-index-cli` package and the `@sanity/embeddings-index-ui` Studio plugin. They receive no new features and no fixes. This guide walks through replacing them with Dataset Embeddings and GROQ, from exporting your current index configuration to deleting it once the new path is live. > [!NOTE] > Does this guide cover you? > It covers search: your application sends a query and renders the results. It does not cover using the index to populate reference fields through AI Assist or Agent Actions, or similar-document search in the Studio plugin. **At this time, we do not have a replacement solution available for using references with Agent Actions.** ## Before you start You'll need: - A project role with permission to manage the legacy embeddings indexes, for the export in [Step 1](https://www.sanity.io/docs/content-lake/migrate-from-embeddings-index-api) and the delete in [Step 5](https://www.sanity.io/docs/content-lake/migrate-from-embeddings-index-api). See [roles and permissions](https://www.sanity.io/docs/user-guides/roles). - The Sanity CLI, `sanity` 5.18.0 or later. The `sanity datasets embeddings` commands ship in `@sanity/cli` 6.2.0, which `sanity` 5.18.0 is the first release to require. Run commands as `npx sanity@latest` to stay on the current version. To call the legacy endpoints over HTTP instead, you need an API token. - A server-side application using [@sanity/client](https://www.sanity.io/docs/apis-and-sdks/js-client-getting-started) and a [read token](https://www.sanity.io/docs/content-lake/http-auth). A read token is all your application needs to *query*. - Your project ID, dataset name, and the name of every legacy index you're replacing. - A set of real search queries to test against. Pull these from your application's search logs or analytics. Also pull your [request logs](https://www.sanity.io/docs/platform-management/request-logs) before you start. They show which indexes are still being called, how often, and from where, which is how you catch a caller you forgot about. Self-serve plans can export the last seven days from the Usage section of [project settings](https://www.sanity.io/manage). That export is capped at 1 GB: a project producing more log data than that within the window gets a truncated file covering less than seven days, so check the range it actually covers before you draw conclusions from it. Enterprise projects can have logs delivered to a storage bucket. Note that the search strings themselves are sent in a POST body and aren't recorded, so your test queries have to come from your own logs. ## What is actually changing Today you create a named index, POST a search string to a query endpoint, and get back document IDs and scores. Then you query GROQ a second time to fetch the documents. Dataset Embeddings replace both requests with one. `text::semanticSimilarity()` is a GROQ function you call inside `score()`, so filtering, keyword matching, boosting, ordering, slicing, and your projection all happen in the same query, and you get documents back. There is no index to create, poll, or keep current. This is not a drop-in swap. Results will not rank identically, because the embedding model, the chunking, and the score scale all differ. Named indexes no longer exist. Reference expansion stops working inside the embedding projection, though `->` still works everywhere else in your queries. Check [what ports cleanly and what doesn't](https://www.sanity.io/docs/content-lake/migrate-from-embeddings-index-api) before you start. ### Concept mapping | Embeddings Index API | Dataset Embeddings | | --- | --- | | One or more named indexes per dataset | One embeddings configuration per dataset | | Index `filter` selects which documents get embedded | Projection selects which document types get embedded. Value-level filtering moves to query time | | Index `projection` selects which fields get embedded, and supports reference expansion | Dataset projection selects which fields get embedded. No reference expansion | | `POST /embeddings-index/query/:dataset/:indexName` | `text::semanticSimilarity()` inside `score()` in any GROQ query | | `maxResults` in the request body | A GROQ slice, for example `[0...10]` | | `filter.type` in the request body | `_type == $type` or `_type in $types` in the GROQ filter | | Response: `[{score, value: {documentId, type}}]` | Documents, with `_score` and `_embeddings` fields | | `score` is a per-result similarity value on a fixed scale | `_score` is opaque and only ranks results within a single query | | Sanity creates a webhook to keep the index current | Updates are automatic, asynchronous, and debounced. No webhook involved | | Create, poll, and delete an index | Enable embeddings once per dataset | ### The endpoints you are replacing | Legacy endpoint | Replacement | | --- | --- | | `GET /vX/embeddings-index/:dataset` | `GET /projects/:projectId/datasets/:name/settings/embeddings`, or `sanity datasets embeddings status ` | | `POST /vX/embeddings-index/:dataset` | `PUT /projects/:projectId/datasets/:name/settings/embeddings`, or `sanity datasets embeddings enable ` | | `GET /vX/embeddings-index/:dataset/:indexName` | `GET /projects/:projectId/datasets/:name/settings/embeddings`. There is one configuration per dataset, so there is no index name | | `DELETE /vX/embeddings-index/:dataset/:indexName` | Nothing. See the warning about `sanity datasets embeddings disable` in step 5 | | `POST /vX/embeddings-index/query/:dataset/:indexName` | A GROQ query using `text::semanticSimilarity()` | ### What ports cleanly and what doesn't Each legacy behavior falls into one of three groups: it ports directly, it ports with a code or content change, or it doesn't port at all. Check where yours lands before you plan the work. #### Ports directly | Legacy behavior | What to do | | --- | --- | | A projection of fields on the document itself | Add the same fields to the dataset projection | | `filter.type` in the query request | Use `_type == $type` or `_type in $types` in the GROQ filter | | `maxResults` | Use a GROQ slice, for example `[0...10]` | | Returning document IDs and types | Project `_id` and `_type`, or project the fields you render and skip the second query entirely | | Fixed RAG retrieval | Same as site search. Query with GROQ and pass the results to your model | #### Ports with a code or content change | Legacy behavior | What to do | | --- | --- | | The index `filter` | Copy it into every replacement query. Enabling embeddings does not carry it over | | A named index | Replace each `indexName` with a query function in your application | | Several indexes over different document types | Merge the fields into one type-conditional projection. Keep a separate query function per index | | Reference expansion in the projection, like `category->title` | Expanding references does not work in embedding projections. Only what's in the document. To keep a referenced value in the embedding, materialize it into a real field first (see Step 2) | | Webhook-driven index updates | Delete the webhook after cutover. Sanity handles updates | | Passing a JSON document as the query input, as the deprecated CLI's `query --text` argument allowed | `text::semanticSimilarity()` takes a string. Serialize or summarize structured input in your application before you query | #### Doesn't port | Legacy behavior | What to do | | --- | --- | | Several indexes with different projections over the *same* documents | A dataset holds one embeddings configuration, so each document has a single projected representation. Embed the union of fields and separate the behaviors with query-time filters and scoring, or model separate searchable documents | | Numeric score thresholds, like `score > 0.8` | Remove them. `_score` is opaque and only meaningful within one query. Bound results with a slice instead | | Identical result ranking | Different model, different chunking, different scoring. Expect the ordering to shift and test accordingly | > [!WARNING] > Copy the legacy filter into every replacement query > Enabling embeddings does not carry your index filter forward. If that filter excluded unpublished, private, market-specific, or expired content, leaving it out changes the result set and can surface content you meant to hide. ## Steps to execute the migration The examples in this guide all use the same index: `public-articles`, filtered to `_type == "article" && searchable == true`, projecting `{title, body, "categoryTitle": category->title}`. ### Step 1: Export your index configuration Fetch every index on the dataset and save the `filter`, `projection`, and `indexName` for each one. You need all three to rebuild the behavior. Store the response as JSON in version control or wherever your team keeps infrastructure config, not just in a scratch file. You'll want it again in [step 5](https://www.sanity.io/docs/content-lake/migrate-from-embeddings-index-api). Two things about the response: Sanity stores your projection with `_type` auto-prepended, so the saved config won't be byte-identical to what you created; and the projection string may contain an unescaped newline that trips strict JSON parsers. Save the raw text, or parse tolerantly. **CLI** ```bash curl https://YOUR_PROJECT_ID.api.sanity.io/vX/embeddings-index/production \ -H "Authorization: Bearer $SANITY_API_TOKEN" ``` Then search your codebase for the query URL, `/embeddings-index/query/`, and note every caller and everything downstream that reads the response. Anything reading `value.documentId` or comparing `score` to a number needs to change. Cross-check the list against your request logs: an index that's taking traffic but doesn't appear in your codebase means there's a caller somewhere you haven't accounted for. ### Step 2: Translate the projections into one dataset projection You get one projection per dataset. The index in the running example covers one document type, so the projection does too. If you're replacing several indexes, merge their fields into a single conditional projection. See [type-specific projections](https://www.sanity.io/docs/content-lake/dataset-embeddings). Type-level scoping from your old filters can move into the projection, because document types you don't list are not embedded. Value-level conditions, like `searchable == true`, cannot. Those stay at query time. **Projection** ```groq { _type == "article" => { title, body } } ``` > [!WARNING] > Do not project a reference alias like categoryTitle > In the legacy index, `categoryTitle` was `"categoryTitle": category->title` — reference expansion. Dataset embeddings cannot dereference, and there is no `categoryTitle` field on the document, so projecting `categoryTitle` here embeds nothing at all, with no error. If you need the referenced value in the embedding, first materialize it into a real string field on the document (for example, an `article.categoryTitle` field kept in sync by a [content migration](https://www.sanity.io/docs/content-lake/schema-and-content-migrations) or a [Sanity Function](https://www.sanity.io/functions)), then project that real field. It's a weak signal either way, so weigh whether it earns the extra field. Keep the projection tight. Every field you add grows each document's embedding, slows generation and recomputation, and adds noise that competes with the signal your users are searching for. Leave out fields that change often but carry no meaning for search, because each change triggers a recomputation. Field names carry semantic weight. `{"musicalGenre": category}` tells the model to read "classical" as music rather than engineering. Documents are chunked before embedding, and there's a cap of 10 chunks per document (subject to change). Content past the cap is dropped. If you're embedding long body fields, scope the projection. Expanding references does not work in embedding projections. Only what's in the document. In testing, the legacy index did embed `category->title`, but as a **weak** signal—a few hundredths of cosine similarity, outweighed by the document's own `title` and `body`. Losing it mostly reshuffles the tail of your results, not the top hit. Materialize the field only if that referenced value is genuinely important to how users search. ### Step 3: Enable embeddings on the dataset **npm** ```shell npx sanity@latest datasets embeddings enable production \ --projection '{_type == "article" => {title, body}}' \ --wait ``` **pnpm** ```shell pnpm dlx sanity@latest datasets embeddings enable production \ --projection '{_type == "article" => {title, body}}' \ --wait ``` **yarn** ```shell yarn dlx sanity@latest datasets embeddings enable production \ --projection '{_type == "article" => {title, body}}' \ --wait ``` **bun** ```shell bunx sanity@latest datasets embeddings enable production \ --projection '{_type == "article" => {title, body}}' \ --wait ``` `--wait` blocks until the initial generation finishes. Without it, the command returns immediately and generation continues in the background. On a large dataset this takes a while (for a few dozen documents it's under a minute; budget much more for large datasets). Check the status at any point: **npm** ```shell npx sanity@latest datasets embeddings status production ``` **pnpm** ```shell pnpm dlx sanity@latest datasets embeddings status production ``` **yarn** ```shell yarn dlx sanity@latest datasets embeddings status production ``` **bun** ```shell bunx sanity@latest datasets embeddings status production ``` The status is `updating`, `ready`, or `error`. Don't send production traffic until it reads `ready`. Querying a dataset without embeddings enabled returns an error. To do this over HTTP instead: **HTTP** ```http PUT /projects/:projectId/datasets/:name/settings/embeddings HTTP/1.1 Content-Type: application/json { "enabled": true, "projection": "{_type == \"article\" => {title, body}}" } ``` The endpoint returns `202 Accepted` and generates asynchronously. > [!NOTE] > Write performance > Depending on system load, write speeds may be slower on datasets with embeddings enabled, and Sanity may apply rate limits to manage resource usage. These behaviors are subject to change. If your dataset takes heavy write traffic, watch it during your test window. See [performance considerations](https://www.sanity.io/docs/content-lake/dataset-embeddings). ### Step 4: Replace the query call The POST to the index and the follow-up query that fetched the documents collapse into a single GROQ query. Before: **Before** ```typescript const results = await client.request({ url: '/embeddings-index/query/production/public-articles', method: 'POST', body: {query: searchText, maxResults: 10, filter: {type: ['article']}}, }) const ids = results.map((result) => result.value.documentId) const documents = await client.fetch(`*[_id in $ids]{_id, title, slug}`, {ids}) ``` After: **After** ```groq *[_type == "article" && searchable == true] | score(text::semanticSimilarity($searchText)) [0...10] { _id, _type, title, slug, _score } ``` The index filter becomes the GROQ filter, `filter.type` becomes `_type`, and `maxResults` becomes the slice. [Query changes](https://www.sanity.io/docs/content-lake/migrate-from-embeddings-index-api) has the full client code, the new response shape, a compatibility adapter for callers that still expect the old format, and when to add keyword matching. ### Step 5: Delete the legacy index > [!WARNING] > Destructive operations > Deleting an index is permanent, and the legacy API is deprecated, so recreating one is not a path you want to depend on. Delete only after you have cut over and run at full traffic long enough to notice a problem. Until then, keep the index so a rollback stays available. Keep the JSON you saved in [step 1](https://www.sanity.io/docs/content-lake/migrate-from-embeddings-index-api) as well: it holds the `filter`, `projection`, and `indexName` you would need to rebuild. And don't reach for `sanity datasets embeddings disable` as cleanup: that command turns off your new dataset embeddings, not your old index. Disabling is destructive too. The computed embedding data may be deleted immediately, and re-enabling triggers a full recompute of every document. **CLI** ```bash curl -X DELETE https://YOUR_PROJECT_ID.api.sanity.io/vX/embeddings-index/production/public-articles \ -H "Authorization: Bearer $SANITY_API_TOKEN" ``` Sanity removes the webhook it created for the index automatically when you delete the index. If *you* added any webhooks of your own to keep the index current, remove those. ## Query changes ### Before Two requests: one to the embeddings index, one to fetch the documents. **Before** ```typescript const results = await client.request({ url: '/embeddings-index/query/production/public-articles', method: 'POST', body: { query: searchText, maxResults: 10, filter: {type: ['article']}, }, }) // [{score: 0.83, value: {documentId: 'abc123', type: 'article'}}] const ids = results.map((result) => result.value.documentId) const documents = await client.fetch(`*[_id in $ids]{_id, title, slug}`, {ids}) ``` ### After One request. The filter, the scoring, the ordering, the slice, and the projection all live in the same query. **searchArticles.ts** ```typescript import {createClient} from '@sanity/client' const client = createClient({ projectId: process.env.SANITY_PROJECT_ID, dataset: 'production', apiVersion: '2026-08-21', token: process.env.SANITY_API_READ_TOKEN, useCdn: false, perspective: 'published', }) export async function searchArticles(searchText: string, maxResults = 10) { const limit = Math.min(Math.max(Math.trunc(maxResults) || 10, 1), 50) return client.fetch( `*[_type == "article" && searchable == true] | score(text::semanticSimilarity($searchText)) [0...$limit] { _id, _type, title, slug, _score }`, {searchText, limit}, ) } ``` The old index filter, `searchable == true`, is now in the GROQ filter. Nothing carries it over for you. `text::semanticSimilarity()` is only valid as an argument to `score()`. Using it anywhere else returns an error. `score()` already sorts results by `_score` descending, so you don't need an explicit `order()`. `maxResults` becomes the upper bound of a [slice](https://www.sanity.io/docs/content-lake/query-cheat-sheet). There is no default limit in GROQ, so always bound your results. Note that `..` is inclusive and `...` is exclusive, which matters when you convert a `maxResults` integer into a slice bound. ### What the response looks like now You get documents back, not pointers to documents: **Response** ```json [ { "_id": "article-auth-guide", "_type": "article", "title": "Authentication guide", "slug": {"current": "authentication-guide"}, "_score": 8.341205 } ] ``` `_score` is a unitless, opaque ranking value. It orders results within one query and nothing more. It is not comparable across queries, and it is not on the same scale as the score the old API returned. If your code has a line like `results.filter(r => r.score > 0.75)`, delete it and control the result count with the slice instead. Semantic queries automatically include an `_embeddings` field on each result, carrying the text fragments that drove the match along with their source fields and character offsets. If your query uses an explicit projection, add `_embeddings` to it to keep the field. Good for highlighting search results, and good for working out why something ranked where it did: **Projection** ```groq *[_type == "article" && searchable == true] | score(text::semanticSimilarity($searchText)) [0...$limit] { _id, _type, title, slug, _score, _embeddings } ``` **_embeddings** ```json { "_embeddings": [ { "fragments": ["OAuth 2.0 provides a secure delegation protocol..."], "fields": ["body"], "startPositions": [0], "endPositions": [74], "score": 8.341205 } ] } ``` ### Keeping the old response shape temporarily If several callers read the legacy format and you'd rather not change them all at once, rebuild the old shape in the projection. No adapter code needed: **GROQ** ```groq *[_type == "article" && searchable == true] | score(text::semanticSimilarity($searchText)) [0...$limit] { "score": _score, "value": { "documentId": _id, "type": _type } } ``` The shape matches, but `score` no longer means what it did. Treat this as a stepping stone, not a destination, and make sure nothing downstream is thresholding on that number. ### Add keyword matching when exact terms matter Semantic search alone handles conceptual queries well. It's weaker on proper nouns, product codes, brand names, and part numbers, where the exact string is the point. Add a `match` expression alongside the semantic one inside `score()`, wrapped in `boost()` to set how much weight the keyword signal carries: **GROQ** ```groq *[_type == "article" && searchable == true] | score( boost([title, body] match text::query($searchText), 0.5), text::semanticSimilarity($searchText) ) [0...10] { _id, _type, title, slug, _score } ``` Each expression contributes to `_score` independently, and a document doesn't need to match both to appear. `boost()` sets the balance. Keyword matches on short fields like `title` can outweigh strong semantic matches on long fields like `body`, because each matching term is a bigger share of a short field, so start the keyword weight low and tune from there. Start semantic-only. Add keyword matching when you can point at queries it fixes. > [!NOTE] > Building an agent rather than a search feature? > If a model decides what to look up, inspects your schema, and writes its own queries, [Sanity Context](https://www.sanity.io/docs/ai/sanity-context) is likely a better fit than querying GROQ directly. It reads from the same dataset embeddings, so enable them either way. Passing search results to a model does not by itself mean you need Context. ## Validation and launch Your new results will not match the old ones exactly. The goal is to confirm they're as good or better, not that they're identical. ### Compare before you switch Use the queries you gathered in [before you start](https://www.sanity.io/docs/content-lake/migrate-from-embeddings-index-api). Twenty to thirty is enough. Include the high-volume ones, the long-tail ones, queries that should return nothing, and any query containing an exact name, code, or brand. Those last ones are where semantic-only search tends to fall short, and where keyword matching helps most. Run each query through both paths and look at the top few results side by side. You're checking for two things: results that got worse, and results that appeared out of nowhere. The second one usually means a filter didn't make it across. ### Watch for the known traps - **Content appearing that the old index excluded.** The legacy filter is missing from your GROQ query. Check your [perspective](https://www.sanity.io/docs/content-lake/perspectives) too. On API versions from 2025-02-19 onward the default is `published`, but on older versions it's `raw`, which returns drafts alongside published documents. Set `perspective: 'published'` explicitly if the old index only covered published documents. - **Exact names and codes ranking poorly.** Add keyword matching. - **Recently edited content returning stale matches.** Embedding updates are asynchronous and debounced. The lag is usually under a minute but can be longer on large or busy datasets. If your product needs fresher results than that, measure the real lag before you commit. - **Everything returning an error.** Check that the embeddings status is `ready`, and that you're calling `text::semanticSimilarity()` inside `score()`. ### Cut over Run both paths against production traffic for a short period if you can, comparing outputs without acting on the new one. Then switch, keeping the old index in place so a rollback is a config change rather than a rebuild. On a large or business-critical search integration, talk to your account team before you cut over. After the switch, watch error rate, latency, and your zero-result rate. Give it a week at full traffic, then confirm in your request logs that the legacy index is taking no traffic before you [delete it](https://www.sanity.io/docs/content-lake/migrate-from-embeddings-index-api). Check that your log export actually covers the whole week before you treat quiet logs as evidence: a truncated export looks exactly like an idle index. ## FAQs ### Do I have to migrate? Yes, if you want your search to keep working. The Embeddings Index API is deprecated and should be replaced with Dataset Embeddings. It receives no fixes or new features. ### What replaces the `@sanity/embeddings-index-cli` package? The Sanity CLI. See [sanity datasets embeddings](https://www.sanity.io/docs/cli-reference/cli-datasets). ### What about the Studio plugin, AI Assist, and Agent Actions references? There's no replacement for reference population in AI Assist and Agent Actions yet, and none for the similar-document search in the Studio plugin. If you depend on either, [contact us](https://www.sanity.io/contact) or post in the [Sanity Community](https://snty.link/community) so we can factor your use case into what comes next. ### Does this cost more? Embeddings generation is included on all plans. Embeddings queries are metered, with a monthly allowance that varies by plan and overage pricing above it. Semantic search itself is included. See [pricing](https://www.sanity.io/pricing) for current allowances, and if you're on an enterprise plan, ask your account team for a forecast based on your actual query volume. ## Next steps - [Dataset Embeddings](https://www.sanity.io/docs/content-lake/dataset-embeddings). Enabling, projections, chunking, querying, and result metadata. - [Search text content with GROQ](https://www.sanity.io/docs/content-lake/search-content-with-groq). Filtering, scoring, BM25, boosts, and pagination. - [Datasets CLI reference](https://www.sanity.io/docs/cli-reference/cli-datasets). The `sanity datasets embeddings` commands. - [Embeddings Index API reference](https://www.sanity.io/docs/http-reference/embeddings-index). The deprecated API, for reference while you migrate.