AI Model Wiki

    Skip to main content
    40 min
    Lesson 4 of 5

    Scaling & Monitoring

    From the course AI Webhooks & Event-Driven Pipelines

    Scaling Event-Driven AI Systems

    Horizontal Scaling

    // Run multiple workers
    const WORKER_COUNT = parseInt(process.env.WORKERS || "4");
    
    for (let i = 0; i < WORKER_COUNT; i++) {
      processQueue().catch(console.error);
    }
    

    Auto-Scaling Strategies

    • Queue depth: Scale workers when queue length exceeds threshold
    • Processing time: Add workers when average processing time increases
    • Time-based: Scale up during business hours, down at night
    • Cost-based: Cap worker count to stay within budget

    Monitoring Dashboard

    Track these metrics:

    • Queue depth: Current items waiting
    • Processing rate: Tasks completed per minute
    • Error rate: Failed tasks as percentage
    • Latency: Time from enqueue to completion
    • Cost: AI API spend per task and total

    Circuit Breaker Pattern

    class AICircuitBreaker {
      private failures = 0;
      private lastCheck = Date.now();
    
      async process(task: AITask) {
        if (this.failures > 10 && Date.now() - this.lastCheck < 60000) {
          throw new Error("Circuit open — AI service unavailable");
        }
        try {
          const result = await callAI(task);
          this.failures = 0;
          return result;
        } catch (error) {
          this.failures++;
          this.lastCheck = Date.now();
          throw error;
        }
      }
    }
    

    End-to-End Observability

    • Assign correlation IDs to trace events through the entire pipeline
    • Log queue operations, worker processing, and AI API calls
    • Set up alerts for queue buildup, error spikes, and latency increases
    • Build dashboards showing the full pipeline health at a glance

    Common Pitfall: Not Tracking Queue Depth Over Time

    Monitoring current queue depth tells you there is a problem. Tracking it over time tells you when problems happen and how fast they resolve. Store queue depth as a time-series metric and alert when it grows faster than workers can process:

    setInterval(async () => {
      const depth = await redis.llen("ai-tasks");
      const processingRate = counters.completedLastMinute;
      metrics.gauge("queue.depth", depth);
      metrics.gauge("queue.processing_rate", processingRate);
    
      if (depth > 1000 && processingRate < 10) {
        await pagerDuty.alert("AI queue growing faster than processing rate");
      }
    }, 60000);
    

    Best Practice: Use Graceful Shutdown for Workers

    Workers that are killed mid-task leave partial results in the database and corrupt the pipeline. Always implement graceful shutdown:

    process.on("SIGTERM", async () => {
      console.log("Received SIGTERM — finishing current task before shutdown...");
      isShuttingDown = true;
      await currentTask?.complete();  // Finish the task in progress
      await redis.quit();
      process.exit(0);
    });
    

    Docker and Kubernetes send SIGTERM before force-killing a container. With graceful shutdown, workers complete their current task before stopping.

    Named Product Reference: Temporal for Long-Running AI Workflows

    Temporal is a workflow orchestration platform that handles multi-step, long-running AI tasks with built-in durability. If your AI pipeline involves 10+ steps that can take hours (e.g., ingest → analyze → research → generate → review → publish), Temporal ensures the workflow resumes from the last completed step after any failure — no custom checkpointing code required. Used in production by Stripe, Hashicorp, and Doordash.

    Key Takeaways

    • •Scale workers horizontally based on queue depth and processing time
    • •Circuit breakers prevent cascading failures when AI services are down
    • •Monitor queue depth, processing rate, errors, latency, and cost
    • •Correlation IDs enable end-to-end tracing through event pipelines
    0 of 5 lessons marked complete

    Progress is kept in this browser only.