Batch Pipeline
The bulk workflow follows a three-phase pipeline from keywords to published content:Prerequisites
- An API key for each site (with
ideaclouds:writeandcontent:writescopes) - A webhook configured on each site to receive
content.completedandcontent.failedevents - Completed IdeaClouds for each article you want to generate
Step 1: Prepare Batch Payloads
Group your IdeaCloud IDs into batches of up to 10 (the maximum for Create content (batch)):Step 2: Submit Batches with Throttling
- Single Site
- Multi-Site (Parallel)
Send each batch with a pause between requests to stay well under the 60 requests/minute default limit:At 2-second intervals, 15 batches (150 articles) complete in about 30 seconds — using only 15 of your 60 requests/minute budget.
Step 3: Handle Rate Limit Errors
If you receive a429, use the Retry-After header to wait and retry with exponential backoff:
Step 4: Track Progress with Webhooks
Each site’s webhook receives events as articles generate. Log the article IDs from the batch response to correlate with incoming webhook payloads:content.progress events as well. See Content Progress Events for details.
Tracking Completion
Keep a simple counter of submitted vs. completed articles per site:Capacity Planning
All sites run concurrently, so total submission time stays constant regardless of how many sites you have.
Submission time is how long it takes to send all requests. Actual content generation runs asynchronously and typically takes 1-3 minutes per article depending on type and complexity.
Checklist
1
Create IdeaClouds
Use the batch endpoint to research all topics. Wait for
ideacloud.completed webhooks before proceeding.2
Prepare batch files
Group completed IdeaCloud IDs into JSON files of up to 10 items each.
3
Submit with throttling
Run all sites in parallel, each sending batches sequentially with 2-second gaps.
4
Handle 429 errors
Respect
Retry-After headers and retry with exponential backoff.5
Monitor via webhooks
Track
content.completed and content.failed events to know when everything is done.Bulk generation checklist complete. You’ve set up IdeaClouds, prepared batches, implemented throttling with retry logic, and configured webhook-based progress tracking.
