Production Audio Pipelines

File Processing Pipeline

async function processAudio(file: File) {
  // 1. Validate format and size
  const supported = ["audio/mp3", "audio/wav", "audio/webm", "audio/mp4"];
  if (!supported.includes(file.type)) throw new Error("Unsupported format");

  // 2. Upload to storage
  const audioUrl = await uploadToStorage(file);

  // 3. Transcribe
  const transcript = await transcribe(audioUrl, {
    speakerLabels: true,
    timestamps: true,
  });

  // 4. Analyze with LLM
  const analysis = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{
      role: "user",
      content: `Summarize this transcript and extract key points:\n${transcript.text}`
    }],
  });

  // 5. Store results
  await saveTranscript({ audioUrl, transcript, analysis });
  return { transcript, analysis };
}

Cost Optimization

• Compress audio before uploading (reduce file size without quality loss)
• Use cheaper models for simple transcription, premium for complex audio
• Cache transcriptions — don't re-process the same audio
• Batch process when real-time isn't needed

Quality Tips

• Use the highest quality audio source available
• Specify the language when known (improves accuracy)
• Post-process transcripts: fix common errors, format properly
• Allow users to edit and correct transcriptions

Accessibility Applications

• Live captions: Real-time subtitles for video calls and presentations
• Podcast transcripts: Full text of audio content for search and accessibility
• Voice commands: Speech-to-text for hands-free interface control
• Language learning: Transcribe learner speech for pronunciation feedback

Common Pitfall: Not Caching Transcription Results

Re-transcribing the same audio file wastes API credits. Cache transcription results keyed by a hash of the audio file:

import { createHash } from "crypto";

async function transcribeWithCache(audioBuffer: Buffer) {
  const hash = createHash("sha256").update(audioBuffer).digest("hex");
  const cached = await redis.get(`transcript:${hash}`);
  if (cached) return JSON.parse(cached);

  const result = await transcribe(audioBuffer);
  await redis.setex(`transcript:${hash}`, 86400, JSON.stringify(result));  // 24h TTL
  return result;
}

Best Practice: Store Word-Level Timestamps for Downstream Use

Even if you only display the plain transcript now, store word-level timestamps with the result. These are invaluable for future features like audio search, transcript highlighting synchronized with audio playback, or identifying when specific keywords were spoken:

const transcript = await openai.audio.transcriptions.create({
  file: audioStream,
  model: "whisper-1",
  response_format: "verbose_json",
  timestamp_granularities: ["word"],  // Request word-level timestamps
});

// Store everything — you can always discard timestamps later
await db.saveTranscript({ text: transcript.text, words: transcript.words });

Named Product Reference: OpenAI Realtime API

For two-way voice conversations (voice assistants, phone bots), OpenAI's Realtime API provides WebSocket-based audio streaming with built-in speech-to-text, LLM processing, and text-to-speech in a single connection. This eliminates the need to chain three separate APIs (STT → LLM → TTS) and reduces round-trip latency from ~2 seconds to under 500ms for voice interactions.