Live Speech Recognition

WebSocket-Based Streaming

// Deepgram real-time transcription
const deepgram = createClient(process.env.DEEPGRAM_API_KEY);

const connection = deepgram.listen.live({
  model: "nova-2",
  language: "en",
  smart_format: true,
  interim_results: true,
});

connection.on("open", () => {
  // Start streaming audio
  mediaRecorder.ondataavailable = (event) => {
    connection.send(event.data);
  };
  mediaRecorder.start(250); // Send every 250ms
});

connection.on("transcriptReceived", (data) => {
  const transcript = data.channel.alternatives[0];
  if (data.is_final) {
    console.log("Final:", transcript.transcript);
  } else {
    console.log("Interim:", transcript.transcript);
  }
});

Browser Audio Capture

const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
const mediaRecorder = new MediaRecorder(stream, {
  mimeType: "audio/webm;codecs=opus",
});

Interim vs. Final Results

• Interim results: Show text as it's being spoken (updates frequently)
• Final results: Confirmed transcription (won't change)
• Display interim results in gray, final in black for UX clarity

Handling Real-Time Challenges

• Buffer audio to avoid gaps from network jitter
• Implement reconnection logic for WebSocket drops
• Handle silence detection to save bandwidth
• Show a visual indicator when the system is listening

Common Pitfall: Not Handling Silence Gracefully

Real-time transcription systems often send empty interim results during silence periods. If you display these as-is, the UI will flicker with empty text. Filter out empty transcripts:

connection.on("transcriptReceived", (data) => {
  const transcript = data.channel.alternatives[0].transcript;
  if (!transcript.trim()) return;  // Ignore empty/whitespace-only transcripts

  if (data.is_final) {
    setFinalText(prev => prev + " " + transcript);
  } else {
    setInterimText(transcript);
  }
});

Best Practice: Use Voice Activity Detection (VAD)

Sending continuous audio to the API wastes bandwidth and incurs unnecessary cost during silence. Use VAD to detect speech presence before streaming:

// Deepgram's API handles VAD server-side when endpointing is enabled
const connection = deepgram.listen.live({
  model: "nova-2",
  endpointing: 500,   // Consider utterance complete after 500ms silence
  vad_events: true,   // Receive speech_started and utterance_end events
});

Named Product Reference: AssemblyAI Universal-2

AssemblyAI's Universal-2 model offers the best accuracy of any commercially available speech-to-text API as of 2025, particularly for noisy environments and accented speech. Its best model tier approaches human-level accuracy on clean audio. AssemblyAI also provides the most comprehensive post-transcription analysis features — sentiment analysis, chapter detection, entity recognition, and PII redaction — all in a single API call.