How Speech-to-Text Works

The Pipeline

1. Audio capture: Microphone input or file upload
2. Pre-processing: Noise reduction, normalization, format conversion
3. Transcription: Convert speech to text using AI models
4. Post-processing: Punctuation, capitalization, formatting

API Options

OpenAI Whisper API

const transcription = await openai.audio.transcriptions.create({
  file: fs.createReadStream("recording.mp3"),
  model: "whisper-1",
  language: "en",
  response_format: "verbose_json",
  timestamp_granularities: ["word", "segment"],
});

console.log(transcription.text);
// Access word-level timestamps
for (const segment of transcription.segments) {
  console.log(`[${segment.start}s] ${segment.text}`);
}

Audio Formats

• Supported: mp3, mp4, mpeg, mpga, m4a, wav, webm
• Best quality: WAV (uncompressed) or FLAC
• Best size: MP3 or Opus for compressed
• Max file size: 25MB (Whisper API)

Common Pitfall: Sending Unsupported Audio Formats

The Whisper API accepts mp3, mp4, mpeg, mpga, m4a, wav, and webm. Browser-recorded audio is often in audio/ogg format (from Firefox) which Whisper does not support. Always convert before sending:

// In a Next.js API route, convert OGG to MP3 using ffmpeg
import { exec } from "child_process";
import { promisify } from "util";
const execAsync = promisify(exec);

await execAsync(`ffmpeg -i input.ogg -codec:a libmp3lame output.mp3`);

Or, on the client side, specify a supported MIME type when creating the MediaRecorder:

const mimeType = MediaRecorder.isTypeSupported("audio/webm;codecs=opus")
  ? "audio/webm;codecs=opus"
  : "audio/mp4";
const recorder = new MediaRecorder(stream, { mimeType });

Best Practice: Split Long Audio Into Chunks

The Whisper API has a 25MB file size limit. For long recordings, split audio into 10-minute segments using ffmpeg before sending, then concatenate the transcripts:

ffmpeg -i long_recording.mp3 -f segment -segment_time 600 -c copy chunk_%03d.mp3

Named Product Reference: Deepgram Nova-2

For real-time transcription where latency matters, Deepgram's Nova-2 model delivers under 300ms latency via WebSocket streaming and consistently outperforms Whisper on English accuracy benchmarks. Deepgram also offers a free tier of 12,000 minutes per year — sufficient for most prototype projects. The Node.js SDK (@deepgram/sdk) makes integration straightforward.