Best practices for creating/updating multiple documents in Sanity using JS client and p-throttle
8 replies
Last updated: Apr 7, 2020
F
Let's say I need to create/update 100+ documents. Is there a best practice? Tried a naive approach to just loop through the data, and do a
client.create()for each entry, but then things quickly starts to fail. Using the JS client.
Apr 7, 2020, 5:22 PM
F
Works with 10-20 requests at a time (hence the
slice(), but starts to fail when increased to about 30 simultaneous requests.
Apr 7, 2020, 5:55 PM
Yeah, that'll be the rate limiting kicking in (max 25 requests per second)!
Apr 7, 2020, 6:51 PM
I'd suggest using a throttler such as `p-throttle`:
npm install p-throttle
const pThrottle = require('p-throttle')
const persistSpeaker = pThrottle(
// Define the function to be called when ready
speaker => client.createOrReplace(speaker, {visibility: 'async'}),
// Max 20 requests
20,
// Within a 1 second window
1000
)
Promise.all(
speakers
.map(transformSpeaker)
.map(persistSpeaker)
)
.then(console.log)
.catch(console.error)Apr 7, 2020, 6:51 PM
You could also use a transaction, but if you've got 100+ speakers I'm not sure I would suggest it - one should try to keep the size of the transaction payload below a reasonable size (< 500kB perhaps?)
So kind of depends on the size and number of those documents.
So kind of depends on the size and number of those documents.
Apr 7, 2020, 6:52 PM
// Use a transaction (not great for a large number of documents)
speakers
.map(transformSpeaker)
.reduce(
(trx, speaker) => trx.createOrReplace(speaker),
client.transaction()
)
.commit()
.then(console.log)
.catch(console.error)Apr 7, 2020, 6:53 PM
Even better, combine the two approaches, batching the speakers up into groups of a certain size (10 in this case) and doing transactions for them:
const pThrottle = require('p-throttle')
const persistSpeakerBatch = pThrottle(
// Define the function to be called when ready
batch => batch.reduce(
(trx, speaker) => trx.createOrReplace(speaker),
client.transaction()
),
// Max 20 requests
20,
// Within a 1 second window
1000
)
let batch = []
for (let i = 0; i < speakers.length; i++) {
batch.push(speakers[i])
if (batch.length === 10 || (i === speakers.length - 1 && batch.length > 0)) {
persistSpeakerBatch(batch)
batch = []
}
}Apr 7, 2020, 7:07 PM
F
Thanks for clarifying! 😄 I did manage to get it to work with the
This is for a importer (importing from
Sessionize.com ), so speed is not that important. Manually executed once in a while.
But really like the combined example.
😄
throttled-queuelibrary, but
p-throttlelooks a bit nicer, and has more users so think I will replace it. 🙂
This is for a importer (importing from
Sessionize.com ), so speed is not that important. Manually executed once in a while.
But really like the combined example.
😄
Apr 7, 2020, 7:09 PM
F
I also looked at the
transaction()API to see if I could use that, but didn't figure out how to reduce the list into it, so really cool to see that example. 🙂
Apr 7, 2020, 7:11 PM
Sanity– build remarkable experiences at scale
The Sanity Composable Content Cloud is the headless CMS that treats content as data to power your digital business. Free to get started, and pay-as-you-go on all plans.