sanity dataset import: 'API rate limit exceeded' at 'Setting asset references to documents'

Late in an import, the CLI sends a separate patch mutation for each document that references assets or other documents. Mutations are limited to 25 requests per second per client IP, and anything over that returns HTTP 429. A large import with many asset-bearing documents can reach that limit during the patch step even when the document and asset uploads went through fine. The fix is to retry the same file with --replace or --missing, keep other traffic from your IP low while it runs, and split the import into smaller batches if it keeps failing.

What the step does

In your NDJSON, assets are written as _sanityAsset values (for example "image@file:///abs/path/poster.jpg") in the place where a _ref would normally go. The CLI uploads those files and connects them to your documents. According to the import docs, all references go in as weak first and are made strong once every document exists. To do that, the CLI submits patch mutations for the documents that contain references, each in its own transaction. That's also why those documents end up with an _updatedAt set to the time of the patch.

The docs don't describe the internals of the "Setting asset references to documents" label. They do say this phase is a burst of per-document mutations, and the technical limits are:

The limit applies per IP, so other scripts, migrations or imports running from the same machine or network share the budget.

Options that help

The datasets import reference lists these relevant flags:

• --asset-concurrency=<n>: the number of parallel asset imports. A lower value reduces upload pressure. The reference doesn't list a flag for how fast the reference-patching mutations are sent.
• --replace: replaces documents that have the same IDs.
• --missing: skips documents that already exist.
• --replace-assets: skips reuse of existing assets. Leave it off on a retry, so assets that already uploaded are reused and not uploaded again.
• --allow-failing-assets: skips assets that can't be fetched or uploaded.

The reference doesn't list a resume flag. Retrying with --replace or --missing is how you continue after a partial import.

Retry safely

By default an import fails if an incoming document already exists. After a partial run, rerun the same file with one of these flags:

# Overwrite whatever the failed run already wrote
npx sanity@latest datasets import my-data.tar.gz -d production --replace

# Or only create documents that aren't there yet
npx sanity@latest datasets import my-data.tar.gz -d production --missing --asset-concurrency 2

Both flags match documents by _id, so put a stable _id on every document in your NDJSON. The docs note that _id is what lets you re-import and replace data from an earlier run.

Use --replace if the first run may have left documents half-finished, for example still holding weak references. Use --missing if you only want to fill in what's missing. Also check that nothing else is sending mutations from your IP while the import runs. The docs also suggest disabling webhooks before large imports, and making sure functions won't run on imported documents.

If you keep hitting the limit

• Split the file. NDJSON has one document per line, so you can cut it into chunks and import them one after another, waiting between chunks:

  split -l 2000 data.ndjson chunk_
  for f in chunk_*; do
    npx sanity@latest datasets import "$f" -d production --missing
  done

  This works because references are imported weak first, which means documents can arrive in any order. Use absolute file:/// paths in _sanityAsset (or import a tarball), so each chunk can still find its asset files.

• Upload assets separately. The docs' approach for script-based imports is to upload assets first, then write documents whose _ref points to the returned asset document ID. Send the requests through a low-concurrency queue that stays under 25 per second:

  const {default: PQueue} = require('p-queue')
  const queue = new PQueue({concurrency: 1, interval: 1000 / 25})

  @sanity/client retries rate-limited queries automatically but not mutations, so your script has to handle 429 on writes. Batch several mutations per transaction, keeping each request under the mutation body size limit, and use visibility: 'deferred' for large imports.