Scaling & Monitoring
From the course AI Webhooks & Event-Driven Pipelines
Scaling Event-Driven AI Systems
Horizontal Scaling
// Run multiple workers
const WORKER_COUNT = parseInt(process.env.WORKERS || "4");
for (let i = 0; i < WORKER_COUNT; i++) {
processQueue().catch(console.error);
}
Auto-Scaling Strategies
- Queue depth: Scale workers when queue length exceeds threshold
- Processing time: Add workers when average processing time increases
- Time-based: Scale up during business hours, down at night
- Cost-based: Cap worker count to stay within budget
Monitoring Dashboard
Track these metrics:
- Queue depth: Current items waiting
- Processing rate: Tasks completed per minute
- Error rate: Failed tasks as percentage
- Latency: Time from enqueue to completion
- Cost: AI API spend per task and total
Circuit Breaker Pattern
class AICircuitBreaker {
private failures = 0;
private lastCheck = Date.now();
async process(task: AITask) {
if (this.failures > 10 && Date.now() - this.lastCheck < 60000) {
throw new Error("Circuit open — AI service unavailable");
}
try {
const result = await callAI(task);
this.failures = 0;
return result;
} catch (error) {
this.failures++;
this.lastCheck = Date.now();
throw error;
}
}
}
End-to-End Observability
- Assign correlation IDs to trace events through the entire pipeline
- Log queue operations, worker processing, and AI API calls
- Set up alerts for queue buildup, error spikes, and latency increases
- Build dashboards showing the full pipeline health at a glance
Common Pitfall: Not Tracking Queue Depth Over Time
Monitoring current queue depth tells you there is a problem. Tracking it over time tells you when problems happen and how fast they resolve. Store queue depth as a time-series metric and alert when it grows faster than workers can process:
setInterval(async () => {
const depth = await redis.llen("ai-tasks");
const processingRate = counters.completedLastMinute;
metrics.gauge("queue.depth", depth);
metrics.gauge("queue.processing_rate", processingRate);
if (depth > 1000 && processingRate < 10) {
await pagerDuty.alert("AI queue growing faster than processing rate");
}
}, 60000);
Best Practice: Use Graceful Shutdown for Workers
Workers that are killed mid-task leave partial results in the database and corrupt the pipeline. Always implement graceful shutdown:
process.on("SIGTERM", async () => {
console.log("Received SIGTERM — finishing current task before shutdown...");
isShuttingDown = true;
await currentTask?.complete(); // Finish the task in progress
await redis.quit();
process.exit(0);
});
Docker and Kubernetes send SIGTERM before force-killing a container. With graceful shutdown, workers complete their current task before stopping.
Named Product Reference: Temporal for Long-Running AI Workflows
Temporal is a workflow orchestration platform that handles multi-step, long-running AI tasks with built-in durability. If your AI pipeline involves 10+ steps that can take hours (e.g., ingest → analyze → research → generate → review → publish), Temporal ensures the workflow resumes from the last completed step after any failure — no custom checkpointing code required. Used in production by Stripe, Hashicorp, and Doordash.
Key Takeaways
- •Scale workers horizontally based on queue depth and processing time
- •Circuit breakers prevent cascading failures when AI services are down
- •Monitor queue depth, processing rate, errors, latency, and cost
- •Correlation IDs enable end-to-end tracing through event pipelines
Progress is kept in this browser only.