Index
Edit

Migrating Data

Adding new fields and data structures to your project is super easy, but sometimes you get it wrong and need to make changes. If it's early days and you don't have much data it's easy, you just make the required changes and fix a few fields by hand.

For projects with considerable amounts of information this quickly becomes impractical though. You then need to write code to reshape your data. Sanity provides two workflows for doing this. Migrating using the API or just using the CLI to export and import datasets.

Migrate using the API

This is really the only way to go about this if you have a live installation and want to avoid stopping the world for your editors.

The example below shows how you might go about renaming a field called name to fullname for a the author document type. It works in batches of 100 and continues patching until no more documents are returned.

It can also be safely be run run even if people are in the middle of editing. If a document has been edited in right after the document was fetched the migration will fail, but can be safely re-run until it passes.

import client from 'part:@sanity/base/client'

// Run this script with: `sanity exec --with-user-token migrations/renameField.js`
//
// This example shows how you may write a migration script that renames a field (name => fullname)
// on a specific document type (author).
// This will migrate documents in batches of 100 and continue patching until no more documents are
// returned from the query.
//
// This script can safely be run, even if documents are being concurrently modified by others.
// If a document gets modified in the time between fetch => submit patch, this script will fail,
// but can safely be re-run multiple times until it eventually runs out of documents to migrate.

// A few things to note:
// - This script will exit if any of the mutations fail due to a revision mismatch (which means the
//   document was edited between fetch => update)
// - The query must eventually return an empty set, or else this script will continue indefinitely

// Fetching documents that matches the precondition for the migration.
// NOTE: This query should eventually return an empty set of documents to mark the migration
// as complete
const fetchDocuments = () =>
  client.fetch(`*[_type == 'author' && defined(name)][0...100] {_id, _rev, name}`)

const buildPatches = docs =>
  docs.map(doc => ({
    id: doc._id,
    patch: {
      set: {fullname: doc.name},
      unset: ['name'],
      // this will cause the transaction to fail if the documents has been
      // modified since it was fetched.
      ifRevisionID: doc._rev
    }
  }))

const createTransaction = patches =>
  patches.reduce((tx, patch) => tx.patch(patch.id, patch.patch), client.transaction())

const commitTransaction = tx => tx.commit()

const migrateNextBatch = async () => {
  const documents = await fetchDocuments()
  const patches = buildPatches(documents)
  if (patches.length === 0) {
    console.log('No more documents to migrate!')
    return null
  }
  console.log(
    `Migrating batch:\n %s`,
    patches.map(patch => `${patch.id} => ${JSON.stringify(patch.patch)}`).join('\n')
  )
  const transaction = createTransaction(patches)
  await commitTransaction(transaction)
  return migrateNextBatch()
}

migrateNextBatch().catch(err => {
  console.error(err)
  process.exit(1)
})

Export, reshape, import

Like it's been noted a previously this method requires that you stop the world for editors - at least for the data types that you are going to be re-importing.

There are three steps involved:

1. Export

sanity dataset export [NAME] [DESTINATION]

Running sanity dataset export --help will get you an overview over the commands available. Note that you can run it with --types if you only want to export certain document types.

2. Reshape

Use whatever tooling you like to reshape your data. This could be as simple as running find-replace-all to rename a type, using a CLI command like ndjson-cli or writing a small script.

3. Import

sanity dataset import [FILE | FOLDER | URL] [TARGET_DATASET]

Again sanity dataset import --help will provide commands for import

Gotcha

Image assets are tied to data-sets. Importing your data to a completely new dataset after reshaping is a good idea as you can verify that everything worked out as planned before switching over your front-ends. Make sure though to bring your assets along for the ride if this is the workflow you have chosen.

Previous: Importing DataNext: Access Control