Create and query an embeddings index
To get started with using the embeddings index API, you first need to create an index.
You can create an embeddings index in one of the following ways:
- With the Embeddings Index CLI.
- With the Embeddings Index UI for Sanity Studio.
- With the Embeddings Index HTTP API.
This guide walks you through configuring an embeddings index for a Sanity project using the Embeddings Index CLI.
Prerequisites
- The Sanity CLI. The CLI ships with the main Sanity package.
You need it to log in to Sanity, which enables consuming the Embeddings Index CLI. - The example assumes that the CLI is run from within a local Sanity project.
Gotcha
In its current state, the embeddings-index API does not support dataset aliases. This means that you have to use the real dataset name in all requests.
Creating an embeddings index
To create an embeddings index, open a terminal session, and then the command that matches how you plan to create an index:
# Create an embeddings index by passing arguments npx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection> # Alternatively, create an embeddings index by passing a JSON manifest npx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json
# Create an embeddings index by passing arguments pnpm dlx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection> # Alternatively, create an embeddings index by passing a JSON manifest pnpm dlx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json
# Create an embeddings index by passing arguments yarn dlx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection> # Alternatively, create an embeddings index by passing a JSON manifest yarn dlx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json
# Create an embeddings index by passing arguments bunx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection> # Alternatively, create an embeddings index by passing a JSON manifest bunx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json
Creating an index can take time, depending on the number of existing documents and the indexer load.
The commands in this guide use npx to run the library, but you can also install the CLI globally and use the embeddings-index command as shown in the CLI readme.
You can define the configuration of an embeddings index in one of the following ways:
- By passing configuration arguments when you create the index in the CLI.
- By storing the configuration details in a JSON manifest file.
Defining the index in the CLI
To define a new embeddings index in the root directory of a Sanity project, pass the following required arguments with the embeddings-index create command:
--indexName: assign a descriptive name to the index.--dataset: specify the name of an existing dataset. This is the target dataset to index.--filter: specify the filtering criteria to include in the index only the selected subset of documents from the database.
The filter must be a valid GROQ filter without the square brackets that wrap the value assigned to_type.
Example:_type=='tutorial'--projection: specify the projection criteria to include in the index only the selected subset of properties from the filtered documents.
The projection must be a valid GROQ projection, including curly brackets.
Example:{title, author}
Alternatively, you can create an embeddings index by passing a JSON manifest file with the --manifest argument:
--manifest <manifest-file-name>.json
Example
# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
npx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
npx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
pnpm dlx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
pnpm dlx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
yarn dlx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
yarn dlx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
bunx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
bunx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.jsonDefining the index in a JSON manifest
To store, reuse, and manage embeddings indexes with source code control and versioning, define their configuration in a JSON manifest file. Save the embeddings indexes manifest.json file to the root directory of a Sanity project.
A JSON manifest file defining an embeddings index must contain the following required fields:
{
indexName: string,
dataset: string,
filter: string,
projection: string
}Example
{
"indexName": "my-embeddings-index",
"dataset": "production",
"filter": "_type=='myType'", // No '[]' square brackets
"projection": "{...}" // Keeps '{}' square brackets
}To create a JSON manifest file, invoke the manifest command:
npx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
pnpm dlx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
yarn dlx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
bunx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
To replace/update an existing index configuration, you'll need to first run the delete command, followed by the create process again.
Checking an embeddings index status
You can check the status of your embeddings indices to monitor the creation progress or the completeness of the indexes.
To check the status of all embeddings indexes in a Sanity project, run:
npx @sanity/embeddings-index-cli listpnpm dlx @sanity/embeddings-index-cli list
yarn dlx @sanity/embeddings-index-cli list
bunx @sanity/embeddings-index-cli listTo check the status of a specific embeddings index in a Sanity project, run:
npx @sanity/embeddings-index-cli get --indexName <name-of-the-index>
pnpm dlx @sanity/embeddings-index-cli get --indexName <name-of-the-index>
yarn dlx @sanity/embeddings-index-cli get --indexName <name-of-the-index>
bunx @sanity/embeddings-index-cli get --indexName <name-of-the-index>
Query an index
To query an index, make a request with the Embeddings Index HTTP API.
import { createClient } from '@sanity/client'
const client = createClient({
projectId: '<your-project-id>',
dataset: '<dataset-name>',
apiVersion: 'vX', // vX is required for embeddings API calls
token: process.env.SANITY_API_TOKEN,
});
const dataset = '<dataset-name>'
const indexName = '<index-name>'
await response = client.request({
url: `/embeddings-index/query/${dataset}/${indexName}`,
method: 'POST',
body: {
query: 'your search query',
maxResults: 15,
}
})curl --request POST 'https://<project-id>.api.sanity.io/<api-version>/embeddings-index/query/<dataset>/<index-name>' \
--header 'Authorization: Bearer <bearer-token>' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"query": "sci-fi adventure with cowboys and aliens",
"maxResults": 10,
"filter": {
"type": ["summary", "synopsis", "userReview"]
}
}'This example uses the query endpoint to search against an index and filter by document type. Learn more about querying the API.
Was this page helpful?