Create images with Agent Actions

Generate and transform images with Agent Actions, either by targeting an image asset directly or by configuring image prompt fields for AI Assist.

This guide takes you through the steps required to enable image generation with Generate or Transform.

Prerequisites:

• Complete the Generate quick start or Transform quick start.
• If using the AI Assist plugin approach, you'll need access to your Studio codebase.
• Each Agent Action request consumes 1 AI credit.

There are two ways to generate images with Agent Actions. You can either explicitly target the image's asset with the target property, or you can enable the AI Assist plugin along with image prompts in your schema. Target the image asset directly to start, since that approach needs no schema change.

This guide assumes you have a configured Sanity client. The examples for both approaches use the following configuration and reference client:

import { createClient } from "@sanity/client";

export const client = createClient({
  projectId: 'YOUR_PROJECT_ID',
  dataset: 'production',
  apiVersion: 'vX',
  token: process.env.SANITY_API_TOKEN
})

Generating images with the explicit targets approach requires instructions that directly target an image asset, but doesn't require a schema change and limits generation to Agent Actions.

Both Generate and Transform use the target property to narrow instructions down to a specific field or fields.

To allow the actions to create or update an image, your request needs to target the image's asset field directly. Transform changes an image that already exists, while Generate creates one where the asset is empty. In this example, the target provides a direct path to the asset:

await client.agent.action.generate({
  documentId: 'someDocumentId',
  schemaId: 'your-schema-id',
  instruction: 'Create an image about cats wrangling project managers.',
  target: {path: ['image', 'asset']}
})

await client.agent.action.transform({
  documentId: 'someDocumentId',
  schemaId: 'your-schema-id',
  instruction: 'Change the image to cats wrangling project managers.',
  target: {path: ['image', 'asset']}
})

You can also target related fields at the same time, such as the image alt text:

await client.agent.action.generate({
  documentId: 'someDocumentId',
  schemaId: 'your-schema-id',
  instruction: 'Create an image about cats wrangling project managers.',
  target: [
    {path: ['image', 'alt']},
    {path: ['image', 'asset']}
  ]
})

await client.agent.action.transform({
  documentId: 'someDocumentId',
  schemaId: 'your-schema-id',
  instruction: 'Change the image to an image about cats wrangling project managers.',
  target: [
    {path: ['image', 'alt']},
    {path: ['image', 'asset']}
  ]
})

This approach doesn't require you to write image-only instructions. You can provide instructions that apply to multiple field types. In this example, the instruction is more generic and uses include alongside the asset path in target:

await client.agent.action.generate({
  documentId: 'someDocumentId',
  schemaId: 'your-schema-id',
  instruction: 'Create content about cats wrangling project managers.',
  target: [
    {include: ['title', 'description', 'body', 'image']},
    {path: ['image', 'asset']},
  ]
})

await client.agent.action.transform({
  documentId: 'someDocumentId',
  schemaId: 'your-schema-id',
  instruction: 'Change this content to be about cats wrangling project managers.',
  target: [
    {include: ['title', 'description', 'body', 'image']},
    {path: ['image', 'asset']},
  ]
})

Transform can perform path-level instructions. Path-level instructions let you apply specific image updates when transforming a document:

await client.agent.action.transform({
  documentId: 'someDocumentId',
  schemaId: 'your-schema-id',
  instruction: 'Create content about cats wrangling project managers.',
  target: {
    path: ['image'],
    include: [
      {path: 'asset', instruction: 'Make it a blue dog.'},
      'alt',
    ]
  }
})

See additional target examples in the common patterns guide.

The AI Assist plugin is available for projects on the Growth plan and up.

The AI Assist method lets you write less specific instructions, but requires adding an image prompt field to your studio's schema. Installing the AI Assist plugin is optional: it adds type completion for the schema option and renders AI presence in the studio.

If you have previously set up the AI Assist plugin and used it to generate images within Sanity Studio, you can skip the setup and configuration steps.

While Generate doesn't require the AI Assist plugin to operate, the plugin provides type completion and adds AI Assist to presence, the avatars that show who is currently editing a document or field:

npm install sanity@latest @sanity/assist@latest

pnpm add sanity@latest @sanity/assist@latest

yarn add sanity@latest @sanity/assist@latest

bun add sanity@latest @sanity/assist@latest

Next, import and add the plugin to your studio config's plugins array:

import { defineConfig } from 'sanity'
import { assist } from '@sanity/assist'
/* other imports */

export default defineConfig({
  /* other config */
  plugins: [
    /* other plugins */
    assist(),
  ]
})

Image generation in schemas works by having Generate write an image prompt to a text field, then using the field's contents to generate the image. Having an explicit field for the prompt lets content editors view it and make changes. One way to set this up is to create a new field as part of your images. For example:

import {defineType, defineField} from 'sanity'

export default defineType({
  type: 'document',
  name: 'movie',
  fields: [
    defineField({
      name: 'image',
      type: 'image',
      fields: [
        defineField({
          type: 'text',
          name: 'instruction',
          title: 'Image prompt',
        })
      ],
      options: {
        hotspot: true,
        aiAssist: {
          imageInstructionField: 'instruction',
        }
      },
    }),
  ]
})

This code creates a new instruction text field that Generate uses to write an image prompt. It also configures the AI Assist plugin and Generate to recognize that field and associate it with the parent image.

You must implement this pattern for any images you'd like AI Assist to interact with.

To make the new field available to Generate, deploy your studio to Sanity with the sanity deploy command, or deploy just the schema with the sanity schema deploy command.

npx sanity@latest schema deploy

pnpm dlx sanity@latest schema deploy

yarn dlx sanity@latest schema deploy

bunx sanity@latest schema deploy

Note the resulting schemaId if you haven't previously used this workspace/dataset combination with Generate.

With your schema deployed, write a script that sends an instruction to Generate to create a document with a generated image.

First, set up your client:

import { createClient } from "@sanity/client";

export const client = createClient({
  projectId: 'YOUR_PROJECT_ID',
  dataset: 'production',
  apiVersion: 'vX',
  token: process.env.SANITY_API_TOKEN
})

Next, create a new instruction:

// ...client setup

await client.agent.action.generate({
  schemaId: 'your-schema-id',
  targetDocument: {operation: 'create', _type: 'movie'},
  instruction: `
    Come up with an idea for a movie.
    Give it a title and overview.
    Generate a poster image based on the overview and title.
  `,
})

This instruction doesn't explicitly call out the image fields, but that's okay. Generate is good at finding fields and intuiting what you mean. To be more explicit, set a target path. The following example reads an existing movie document and targets the poster image field to generate the image:

// ...client setup

const docId = 'your-movie-id'
await client.agent.action.generate({
  schemaId: 'your-schema-id',
  documentId: docId,
  instruction: `
    Add a poster image for this movie.
    Use the information in $background to come up with the image.
  `,
  instructionParams: {
    background: {
      type: 'document'
    },
  },
  target: {
    path: 'poster'
  }
})

The code in this example does the following:

• It uses documentId instead of targetDocument to update an existing document.
• It sets the path to poster, which is the image in the movie schema. Setting the path tells Generate to apply the instruction to that field.
• It uses a document-type instruction parameter to query the details of the existing document.

Generate writes the image prompt to the text field named by imageInstructionField, but this example targets poster. The example works because Agent Actions can navigate to children of the supplied path and use the fields they need to generate the image.