Audio and video AI has reached production quality for transcription, understanding, and generation. These modalities unlock applications that text alone cannot serve.

Audio AI

Speech-to-Text (ASR):

• Whisper (OpenAI): Open-source, multilingual, robust. The default choice for most applications.
• Deepgram: Low-latency streaming ASR for real-time applications.
• AssemblyAI: Production API with speaker diarization, sentiment, and topic detection.

Text-to-Speech (TTS):

• ElevenLabs: Highest quality, voice cloning, emotional control
• OpenAI TTS: Good quality, simple API, multiple voices
• Coqui/XTTS: Open-source, self-hostable

Audio Understanding: GPT-4o and Gemini can directly process audio — understanding tone, emotion, music, and environmental sounds.

Video AI

Video Understanding:

• Gemini 1.5/2.5 processes up to hours of video natively
• GPT-4o can process video through frame extraction
• Applications: content moderation, highlight extraction, search within videos

Frame Extraction Strategy:

# Extract 1 frame per second for a 60-second video
# Send key frames to a VLM for analysis
import cv2

cap = cv2.VideoCapture("video.mp4")
fps = cap.get(cv2.CAP_PROP_FPS)
frames = []

while True:
    ret, frame = cap.read()
    if not ret:
        break
    frame_number = int(cap.get(cv2.CAP_PROP_POS_FRAMES))
    if frame_number % int(fps) == 0:  # 1 frame per second
        frames.append(frame)

Video Generation:

• Sora (OpenAI): High-quality, physics-aware video generation
• Runway Gen-3: Production video generation and editing
• Kling, Pika, Veo: Competitive alternatives with different strengths

Real-Time Multimodal

The frontier is real-time multimodal interaction:

• GPT-4o's voice mode processes speech, vision, and text simultaneously
• Gemini Live offers similar real-time multimodal conversation
• Applications: AI tutoring, accessibility tools, hands-free assistance, robotics