Index
Edit

Importing Data

There are two ways to import data into your Sanity project. The recommended way of importing data is to use our Command Line Interface sanity dataset import. The other option is to use one of our client libraries and handle the import yourself. The latter approach has some common pitfalls which are outlined below.

Import using Sanity CLI

The Sanity import tool operates on newline-delimited JSON (ndjson) files. Basically, each line in a file is a valid JSON-object containing a document you want to import.

Documents should follow the structure of your data model - most importantly, the requirement of a _type attribute. During import, all references are automatically set to weak, then flipped to strong after all documents are in place. This ensures that you can import documents that reference other documents in any order you like.

Assets (images and files) are stored using references in Sanity. To make it easy to import these and refer to them within your documents, you can use a special _sanityAsset-property where you would normally put a _ref. For instance, lets say you want your document to end up like this:

{
  "_id": "movie_123",
  "_type": "movie",
  "title": "Rogue One",
  "poster": {
    "_type": "image",
    "asset": {
      "_ref": "image_234",
      "_type": "reference"
    }
  }
}

This is what your import ndjson-file should look like:

{
  "_id": "movie_123",
  "_type": "movie",
  "title": "Rogue One",
  "poster": {
    "_sanityAsset": "image@file:///local/path/to/rogue-one-poster.jpg",
  }
}

Note that you need to prefix the asset URL with a type declaration, either image@ or file@.

If your asset is on the Internet use image@https://example.com/path/to/rogue-one-poster.jpg instead of image@file:///local/path/to/rogue-one-poster.jpg.

Once you have prepared your ndjson-file, you can run the import using the Sanity CLI:

sanity dataset import <file.ndjson> <targetDataset>

E.g.:

sanity dataset import my-data-dump.ndjson production

Note: This import fails if an incoming document already exists. A couple of options allow you to amend this:

--replace Overwrite existing documents. If you specify _id in the imported data, this flag can be very useful. It will let you reimport stuff that you got wrong in an earlier pass.
--missing Only create documents which don't exist, leave the rest alone.

Import using a client library

If you prefer not to use our CLI import tool, you may of course do the import yourself with help from one of our client libraries.

There are some common pitfalls to keep in mind:

  • Concurrency. While you may have thousands of documents to import, you shouldn't trigger thousands of requests in parallel. This is going to exceed API rate limits and fail. We advise you to use a queue with a reasonably low concurrency.
  • API usage limits. Importing large data sets can quickly cause a lot of requests, especially if you import a single document per request. It is usually a good idea to send multiple mutations within a single transaction.
  • Mutation size limits. While it's a good idea to do multiple mutations per transaction, you need to make sure that the size of the request is within our limits, in terms of byte size.
  • Mutation visibility. Clients use the visibility mode of sync by default, which means that it will wait for the documents to be searchable before returning. This should not be necessary when importing large data sets. Instead, use deferred. If you have a lot of documents, it can take a little while for them to be searchable, but the import job will move along much faster.
  • References. If you are referring to one document from another, they either need to be imported in the right order, or the reference needs to be flagged as weak by setting the _weak property to true. After importing, you probably want to remove the weak property in order to prevent referenced documents from being deleted.
  • Assets. Since assets (files, images) in Sanity are stored using references, you'll need to upload the assets first and put the returned document ID in your reference.

With these things in mind, check out our client libraries documentation to see how to perform mutations.

Previous: Project StructureNext: Localization