AI Model Wiki

    Skip to main content
    40 min
    Lesson 3 of 5

    Queue Patterns & Retry Logic

    From the course AI Webhooks & Event-Driven Pipelines

    Reliable AI Task Processing

    Queue-Based Processing

    // Producer: Add tasks to queue
    async function enqueueAITask(task: AITask) {
      await redis.lpush("ai-tasks", JSON.stringify({
        id: crypto.randomUUID(),
        ...task,
        createdAt: Date.now(),
        attempts: 0,
      }));
    }
    
    // Consumer: Process tasks from queue
    async function processQueue() {
      while (true) {
        const item = await redis.brpop("ai-tasks", 30); // Block 30s
        if (!item) continue;
    
        const task = JSON.parse(item[1]);
        try {
          const result = await processAITask(task);
          await storeResult(task.id, result);
          await notifyClient(task.callbackUrl, result);
        } catch (error) {
          await handleFailure(task, error);
        }
      }
    }
    

    Retry with Exponential Backoff

    async function handleFailure(task: any, error: Error) {
      task.attempts++;
      if (task.attempts >= 5) {
        await moveToDeadLetterQueue(task, error);
        return;
      }
      const delay = Math.pow(2, task.attempts) * 1000; // 2s, 4s, 8s, 16s
      await redis.lpush("ai-tasks-delayed", JSON.stringify({ ...task, processAfter: Date.now() + delay }));
    }
    

    Dead Letter Queue

    Failed tasks that exceed max retries go to a dead letter queue for:

    • Manual investigation
    • Pattern analysis (are certain tasks always failing?)
    • Bulk reprocessing after fixing the underlying issue

    Priority Queues

    // High-priority tasks processed first
    await redis.lpush("ai-tasks:high", task);    // Premium users
    await redis.lpush("ai-tasks:normal", task);  // Free users
    await redis.lpush("ai-tasks:low", task);     // Background jobs
    

    Common Pitfall: Setting Max Retries Too High on Rate-Limited Tasks

    If the AI provider is rate-limiting requests (HTTP 429), retrying with exponential backoff is correct. But if you set maxRetries=10, a batch of 1,000 tasks can each retry up to 10 times — suddenly creating 10,000 API requests and amplifying the rate-limit problem. Distinguish between retryable errors (network failures, 5xx) and non-retryable ones (rate limits need coordinated backoff):

    async function handleFailure(task: any, error: any) {
      const isRateLimit = error.status === 429;
      const maxRetries = isRateLimit ? 3 : 5;
    
      if (task.attempts >= maxRetries) {
        await moveToDeadLetterQueue(task, error);
        return;
      }
      const baseDelay = isRateLimit ? 30000 : 1000;  // 30s for rate limits, 1s for others
      const delay = baseDelay * Math.pow(2, task.attempts);
      await scheduleRetry(task, delay);
    }
    

    Best Practice: Add Jitter to Retry Delays

    When many tasks fail simultaneously (e.g., during an API outage), fixed exponential backoff causes all retries to fire at the same moment — creating a "thundering herd" that overwhelms the recovering service. Add random jitter:

    const jitter = Math.random() * 1000;  // 0-1000ms random delay
    const delay = Math.pow(2, task.attempts) * 1000 + jitter;
    

    Named Product Reference: Inngest for Event-Driven AI Workflows

    Inngest is a cloud service for event-driven workflows that handles retry logic, rate limiting, concurrency, and scheduling out of the box. You define workflow steps as TypeScript functions and Inngest manages the execution infrastructure. For AI workflows that need reliable multi-step execution with retries and human-in-the-loop steps, Inngest is a strong alternative to building a custom queue system.

    Key Takeaways

    • •Queues decouple producers from consumers for reliable processing
    • •Retry with exponential backoff handles transient failures
    • •Dead letter queues capture permanently failed tasks for investigation
    • •Priority queues ensure important tasks are processed first
    0 of 5 lessons marked complete

    Progress is kept in this browser only.