Best practices for creating/updating multiple documents in Sanity using JS client and p-throttle
8 replies
Last updated: Apr 7, 2020
F
Let's say I need to create/update 100+ documents. Is there a best practice? Tried a naive approach to just loop through the data, and do a
client.create()for each entry, but then things quickly starts to fail. Using the JS client.
Apr 7, 2020, 5:22 PM
F
Works with 10-20 requests at a time (hence the
slice(), but starts to fail when increased to about 30 simultaneous requests.
Apr 7, 2020, 5:55 PM
I'd suggest using a throttler such as `p-throttle`:
npm install p-throttle
const pThrottle = require('p-throttle') const persistSpeaker = pThrottle( // Define the function to be called when ready speaker => client.createOrReplace(speaker, {visibility: 'async'}), // Max 20 requests 20, // Within a 1 second window 1000 ) Promise.all( speakers .map(transformSpeaker) .map(persistSpeaker) ) .then(console.log) .catch(console.error)
Apr 7, 2020, 6:51 PM
You could also use a transaction, but if you've got 100+ speakers I'm not sure I would suggest it - one should try to keep the size of the transaction payload below a reasonable size (< 500kB perhaps?)
So kind of depends on the size and number of those documents.
So kind of depends on the size and number of those documents.
Apr 7, 2020, 6:52 PM
// Use a transaction (not great for a large number of documents) speakers .map(transformSpeaker) .reduce( (trx, speaker) => trx.createOrReplace(speaker), client.transaction() ) .commit() .then(console.log) .catch(console.error)
Apr 7, 2020, 6:53 PM
Even better, combine the two approaches, batching the speakers up into groups of a certain size (10 in this case) and doing transactions for them:
const pThrottle = require('p-throttle') const persistSpeakerBatch = pThrottle( // Define the function to be called when ready batch => batch.reduce( (trx, speaker) => trx.createOrReplace(speaker), client.transaction() ), // Max 20 requests 20, // Within a 1 second window 1000 ) let batch = [] for (let i = 0; i < speakers.length; i++) { batch.push(speakers[i]) if (batch.length === 10 || (i === speakers.length - 1 && batch.length > 0)) { persistSpeakerBatch(batch) batch = [] } }
Apr 7, 2020, 7:07 PM
F
Thanks for clarifying! π I did manage to get it to work with the
This is for a importer (importing from
Sessionize.com ), so speed is not that important. Manually executed once in a while.
But really like the combined example.
π
throttled-queuelibrary, but
p-throttlelooks a bit nicer, and has more users so think I will replace it. π
This is for a importer (importing from
Sessionize.com ), so speed is not that important. Manually executed once in a while.
But really like the combined example.
π
Apr 7, 2020, 7:09 PM
F
I also looked at the
transaction()API to see if I could use that, but didn't figure out how to reduce the list into it, so really cool to see that example. π
Apr 7, 2020, 7:11 PM
Sanityβ build remarkable experiences at scale
The Sanity Composable Content Cloud is the headless CMS that treats content as data to power your digital business. Free to get started, and pay-as-you-go on all plans.