Best practices for creating/updating multiple documents in Sanity using JS client and p-throttle

8 replies
Last updated: Apr 7, 2020
Let's say I need to create/update 100+ documents. Is there a best practice? Tried a naive approach to just loop through the data, and do a
client.create()
for each entry, but then things quickly starts to fail. Using the JS client.
Apr 7, 2020, 5:22 PM
Works with 10-20 requests at a time (hence the
slice()
, but starts to fail when increased to about 30 simultaneous requests.
Apr 7, 2020, 5:55 PM
Yeah, that'll be the rate limiting kicking in (max 25 requests per second)!
Apr 7, 2020, 6:51 PM
I'd suggest using a throttler such as `p-throttle`:
npm install p-throttle


const pThrottle = require('p-throttle')

const persistSpeaker = pThrottle(
  // Define the function to be called when ready
  speaker => client.createOrReplace(speaker, {visibility: 'async'}),
  // Max 20 requests
  20,
  // Within a 1 second window
  1000
)

Promise.all(
  speakers
    .map(transformSpeaker)
    .map(persistSpeaker)
)
  .then(console.log)
  .catch(console.error)
Apr 7, 2020, 6:51 PM
You could also use a transaction, but if you've got 100+ speakers I'm not sure I would suggest it - one should try to keep the size of the transaction payload below a reasonable size (< 500kB perhaps?)
So kind of depends on the size and number of those documents.
Apr 7, 2020, 6:52 PM
// Use a transaction (not great for a large number of documents)
speakers
  .map(transformSpeaker)
  .reduce(
    (trx, speaker) => trx.createOrReplace(speaker),
    client.transaction()
  )
  .commit()
  .then(console.log)
  .catch(console.error)
Apr 7, 2020, 6:53 PM
Even better, combine the two approaches, batching the speakers up into groups of a certain size (10 in this case) and doing transactions for them:

const pThrottle = require('p-throttle')

const persistSpeakerBatch = pThrottle(
  // Define the function to be called when ready
  batch => batch.reduce(
    (trx, speaker) => trx.createOrReplace(speaker),
    client.transaction()
  ),
  // Max 20 requests
  20,
  // Within a 1 second window
  1000
)

let batch = []
for (let i = 0; i < speakers.length; i++) {
  batch.push(speakers[i])

  if (batch.length === 10 || (i === speakers.length - 1 && batch.length > 0)) {
    persistSpeakerBatch(batch)
    batch = []
  }
}
Apr 7, 2020, 7:07 PM
Thanks for clarifying! πŸ˜„ I did manage to get it to work with the
throttled-queue
library, but
p-throttle
looks a bit nicer, and has more users so think I will replace it. πŸ™‚
This is for a importer (importing from
Sessionize.com ), so speed is not that important. Manually executed once in a while.
But really like the combined example.
πŸ˜„
Apr 7, 2020, 7:09 PM
I also looked at the
transaction()
API to see if I could use that, but didn't figure out how to reduce the list into it, so really cool to see that example. πŸ™‚
Apr 7, 2020, 7:11 PM

Sanity– build remarkable experiences at scale

Sanity is a modern headless CMS that treats content as data to power your digital business. Free to get started, and pay-as-you-go on all plans.

Was this answer helpful?