Embeddings (Experimental)
Was this page helpful?
To get started with using the embeddings index API, you first need to create an index.
You can create an embeddings index in one of the following ways:
This guide walks you through configuring an embeddings index for a Sanity project using the Embeddings Index CLI.
In its current state, the embeddings-index API does not support dataset aliases. This means that you have to use the real dataset name in all requests.
To create an embeddings index, open a terminal session, and then the command that matches how you plan to create an index:
# Create an embeddings index by passing arguments npx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection> # Alternatively, create an embeddings index by passing a JSON manifest npx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json
# Create an embeddings index by passing arguments pnpm dlx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection> # Alternatively, create an embeddings index by passing a JSON manifest pnpm dlx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json
# Create an embeddings index by passing arguments yarn dlx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection> # Alternatively, create an embeddings index by passing a JSON manifest yarn dlx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json
# Create an embeddings index by passing arguments bunx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection> # Alternatively, create an embeddings index by passing a JSON manifest bunx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json
Creating an index can take time, depending on the number of existing documents and the indexer load.
The commands in this guide use npx to run the library, but you can also install the CLI globally and use the embeddings-index command as shown in the CLI readme.
You can define the configuration of an embeddings index in one of the following ways:
To define a new embeddings index in the root directory of a Sanity project, pass the following required arguments with the embeddings-index create command:
--indexName: assign a descriptive name to the index.--dataset: specify the name of an existing dataset. This is the target dataset to index.--filter: specify the filtering criteria to include in the index only the selected subset of documents from the database._type._type=='tutorial'--projection: specify the projection criteria to include in the index only the selected subset of properties from the filtered documents.{title, author}Alternatively, you can create an embeddings index by passing a JSON manifest file with the --manifest argument:
--manifest <manifest-file-name>.jsonExample
# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
npx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
npx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
pnpm dlx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
pnpm dlx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
yarn dlx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
yarn dlx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
bunx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
bunx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.jsonTo store, reuse, and manage embeddings indexes with source code control and versioning, define their configuration in a JSON manifest file. Save the embeddings indexes manifest.json file to the root directory of a Sanity project.
A JSON manifest file defining an embeddings index must contain the following required fields:
{
indexName: string,
dataset: string,
filter: string,
projection: string
}Example
{
"indexName": "my-embeddings-index",
"dataset": "production",
"filter": "_type=='myType'", // No '[]' square brackets
"projection": "{...}" // Keeps '{}' square brackets
}To create a JSON manifest file, invoke the manifest command:
npx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
pnpm dlx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
yarn dlx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
bunx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
To replace/update an existing index configuration, you'll need to first run the delete command, followed by the create process again.
You can check the status of your embeddings indices to monitor the creation progress or the completeness of the indexes.
To check the status of all embeddings indexes in a Sanity project, run:
npx @sanity/embeddings-index-cli listpnpm dlx @sanity/embeddings-index-cli list
yarn dlx @sanity/embeddings-index-cli list
bunx @sanity/embeddings-index-cli listTo check the status of a specific embeddings index in a Sanity project, run:
npx @sanity/embeddings-index-cli get --indexName <name-of-the-index>
pnpm dlx @sanity/embeddings-index-cli get --indexName <name-of-the-index>
yarn dlx @sanity/embeddings-index-cli get --indexName <name-of-the-index>
bunx @sanity/embeddings-index-cli get --indexName <name-of-the-index>
To query an index, make a request with the Embeddings Index HTTP API.
import { createClient } from '@sanity/client'
const client = createClient({
projectId: '<your-project-id>',
dataset: '<dataset-name>',
apiVersion: 'vX', // vX is required for embeddings API calls
token: process.env.SANITY_API_TOKEN,
});
const dataset = '<dataset-name>'
const indexName = '<index-name>'
await response = client.request({
url: `/embeddings-index/query/${dataset}/${indexName}`,
method: 'POST',
body: {
query: 'your search query',
maxResults: 15,
}
})curl --request POST 'https://<project-id>.api.sanity.io/<api-version>/embeddings-index/query/<dataset>/<index-name>' \
--header 'Authorization: Bearer <bearer-token>' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"query": "sci-fi adventure with cowboys and aliens",
"maxResults": 10,
"filter": {
"type": ["summary", "synopsis", "userReview"]
}
}'This example uses the query endpoint to search against an index and filter by document type. Learn more about querying the API.
# Create an embeddings index by passing arguments
npx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
# Alternatively, create an embeddings index by passing a JSON manifest
npx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json# Create an embeddings index by passing arguments pnpm dlx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection> # Alternatively, create an embeddings index by passing a JSON manifest pnpm dlx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json
# Create an embeddings index by passing arguments yarn dlx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection> # Alternatively, create an embeddings index by passing a JSON manifest yarn dlx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json
# Create an embeddings index by passing arguments
bunx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
# Alternatively, create an embeddings index by passing a JSON manifest
bunx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json# Create an embeddings index by passing arguments
npx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
# Alternatively, create an embeddings index by passing a JSON manifest
npx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json# Create an embeddings index by passing arguments
pnpm dlx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
# Alternatively, create an embeddings index by passing a JSON manifest
pnpm dlx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json# Create an embeddings index by passing arguments
yarn dlx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
# Alternatively, create an embeddings index by passing a JSON manifest
yarn dlx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json# Create an embeddings index by passing arguments
bunx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
# Alternatively, create an embeddings index by passing a JSON manifest
bunx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
npx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
npx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
pnpm dlx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
pnpm dlx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
yarn dlx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
yarn dlx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
bunx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
bunx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
npx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
npx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
pnpm dlx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
pnpm dlx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
yarn dlx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
yarn dlx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
bunx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
bunx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json{
indexName: string,
dataset: string,
filter: string,
projection: string
}{
"indexName": "my-embeddings-index",
"dataset": "production",
"filter": "_type=='myType'", // No '[]' square brackets
"projection": "{...}" // Keeps '{}' square brackets
}npx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>pnpm dlx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>yarn dlx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>bunx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
npx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>pnpm dlx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>yarn dlx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>bunx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>npx @sanity/embeddings-index-cli listpnpm dlx @sanity/embeddings-index-cli listyarn dlx @sanity/embeddings-index-cli listbunx @sanity/embeddings-index-cli listpnpm dlx @sanity/embeddings-index-cli listyarn dlx @sanity/embeddings-index-cli listnpx @sanity/embeddings-index-cli get --indexName <name-of-the-index>pnpm dlx @sanity/embeddings-index-cli get --indexName <name-of-the-index>yarn dlx @sanity/embeddings-index-cli get --indexName <name-of-the-index>bunx @sanity/embeddings-index-cli get --indexName <name-of-the-index>npx @sanity/embeddings-index-cli get --indexName <name-of-the-index>pnpm dlx @sanity/embeddings-index-cli get --indexName <name-of-the-index>yarn dlx @sanity/embeddings-index-cli get --indexName <name-of-the-index>bunx @sanity/embeddings-index-cli get --indexName <name-of-the-index>import { createClient } from '@sanity/client'
const client = createClient({
projectId: '<your-project-id>',
dataset: '<dataset-name>',
apiVersion: 'vX', // vX is required for embeddings API calls
token: process.env.SANITY_API_TOKEN,
});
const dataset = '<dataset-name>'
const indexName = '<index-name>'
await response = client.request({
url: `/embeddings-index/query/${dataset}/${indexName}`,
method: 'POST',
body: {
query: 'your search query',
maxResults: 15,
}
})curl --request POST 'https://<project-id>.api.sanity.io/<api-version>/embeddings-index/query/<dataset>/<index-name>' \
--header 'Authorization: Bearer <bearer-token>' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--data '{
"query": "sci-fi adventure with cowboys and aliens",
"maxResults": 10,
"filter": {
"type": ["summary", "synopsis", "userReview"]
}
}'# Create an embeddings index by passing arguments
pnpm dlx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
# Alternatively, create an embeddings index by passing a JSON manifest
pnpm dlx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json# Create an embeddings index by passing arguments
yarn dlx @sanity/embeddings-index-cli create --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>
# Alternatively, create an embeddings index by passing a JSON manifest
yarn dlx @sanity/embeddings-index-cli create --manifest <manifest-file-name>.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
yarn dlx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
yarn dlx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.json# Create embeddings index with arguments
# 'filter' has no '[]' square brackets
# 'projection' keeps '{}' curly brackets
bunx @sanity/embeddings-index-cli create --indexName my-embeddings-index --dataset production --filter _type=='myDocumentType' --projection {...}
# Create embeddings index with JSON manifest
# The JSON manifest is in the project root directory
bunx @sanity/embeddings-index-cli create --manifest embeddings-index-manifest.jsonbunx @sanity/embeddings-index-cli manifest --out manifest.json --indexName <name-of-the-index> --dataset <name-of-the-dataset> --filter <GROQ-filter> --projection <GROQ-projection>