Computer Vision with AI APIs

Vision Through LLMs

Modern multimodal LLMs provide powerful vision capabilities through simple API calls:

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{
    role: "user",
    content: [
      { type: "image_url", image_url: { url: imageUrl } },
      { type: "text", text: "Classify this image. Return the category and confidence." }
    ],
  }],
});

Structured Image Analysis

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{
    role: "user",
    content: [
      { type: "image_url", image_url: { url: imageUrl } },
      { type: "text", text: "Analyze this image and return JSON with: objects, scene, colors, mood, text_content" }
    ],
  }],
  response_format: { type: "json_object" },
});

Dedicated Vision APIs

Choosing Your Approach

• LLM Vision: Best for general understanding, reasoning about images, flexible queries
• Specialized APIs: Best for specific tasks (OCR, face detection) with structured output
• Custom models: Best when you need domain-specific detection (defects, medical imaging)

Common Pitfall: Sending Full-Resolution Images to Vision APIs

A 12-megapixel phone photo sent to GPT-4o or Google Cloud Vision consumes enormous tokens and bandwidth — and produces no better results than a 1024px-wide resize. Most vision APIs tile large images internally anyway, so you pay for multiple tiles. Resize to a maximum of 1024–2048px on the longest edge before sending:

import sharp from "sharp";

async function prepareForVision(filePath: string): Promise<Buffer> {
  return sharp(filePath)
    .resize({ width: 1536, height: 1536, fit: "inside", withoutEnlargement: true })
    .jpeg({ quality: 85 })
    .toBuffer();
}

For GPT-4o specifically, the "low" detail mode processes images as a single 512×512 tile for a flat cost of 85 tokens — sufficient for most classification and moderation tasks.

Best Practice: Use Structured Output Prompts for Consistent Parsing

Free-form vision responses ("the image shows a cat sitting on a red couch") are hard to parse reliably. Always ask for structured JSON in the prompt and combine it with response_format set to json_object mode. Define the exact schema in the prompt so downstream code can depend on it:

const ANALYSIS_PROMPT = `Analyze this image and return ONLY valid JSON matching this schema:
{
  "primary_subject": string,
  "scene_type": "indoor" | "outdoor" | "close-up" | "abstract",
  "dominant_colors": string[],
  "contains_text": boolean,
  "estimated_text": string | null,
  "confidence": "high" | "medium" | "low"
}`;

Named Product Reference: Google Cloud Vision API

Google Cloud Vision (cloud.google.com/vision) provides purpose-built detectors for OCR, label detection, landmark recognition, safe search, and face detection — all in a single API call. For high-volume document processing, its Document AI product offers specialized parsers for invoices, contracts, W-2 forms, and US driver's licenses that return structured entity extractions rather than raw text, saving significant prompt engineering effort.