Detecting Objects and Extracting Text

Object Detection with Vision LLMs

const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{
    role: "user",
    content: [
      { type: "image_url", image_url: { url: imageUrl } },
      { type: "text", text: `Identify all objects in this image. Return JSON array:
[{ "object": string, "location": "top-left|center|bottom-right|etc", "confidence": "high|medium|low" }]` }
    ],
  }],
  response_format: { type: "json_object" },
});

OCR (Optical Character Recognition)

Extract text from images — receipts, documents, signs, screenshots:

// Using GPT-4o for OCR
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{
    role: "user",
    content: [
      { type: "image_url", image_url: { url: receiptImage } },
      { type: "text", text: "Extract all text from this receipt. Return structured JSON with: store_name, date, items (name, quantity, price), subtotal, tax, total." }
    ],
  }],
  response_format: { type: "json_object" },
});

Document Processing Pipeline

1. Capture: Accept image uploads or camera captures
2. Pre-process: Resize, enhance contrast if needed
3. Extract: Use OCR to pull text and structure
4. Validate: Check extracted data against expected schemas
5. Store: Save structured data to database

OCR Comparison

Common Pitfall: Prompting for Bounding Boxes with LLM Vision

LLM vision models (GPT-4o, Gemini) do not return pixel coordinates for detected objects — they describe spatial position in natural language ("upper-left", "center"). If you need actual bounding boxes for downstream tasks (cropping, annotation, highlighting), use a dedicated detector like Google Cloud Vision, AWS Rekognition, or Roboflow. Attempting to parse natural language coordinates ("the cat is roughly in the top 30% of the image") for programmatic use is unreliable.

Best Practice: Cache OCR Results by Image Hash

OCR is expensive and deterministic — the same image will always produce the same text. Before sending an image to an OCR API, compute a SHA-256 hash of the image bytes and check a cache (Redis, database) for a stored result. This pattern eliminates redundant API calls when users resubmit the same receipt or document:

import { createHash } from "crypto";

async function getCachedOrExtract(imageBuffer: Buffer): Promise<string> {
  const hash = createHash("sha256").update(imageBuffer).digest("hex");
  const cached = await redis.get(`ocr:${hash}`);
  if (cached) return cached as string;

  const extracted = await callOcrApi(imageBuffer);
  await redis.setex(`ocr:${hash}`, 86400 * 30, extracted); // Cache 30 days
  return extracted;
}

Named Product Reference: AWS Textract

AWS Textract goes beyond basic OCR to understand document structure — it distinguishes between headers, form key-value pairs, table cells, and signature fields. For processing structured business documents (invoices, tax forms, insurance claims), Textract's AnalyzeDocument API returns a typed document model with bounding boxes and confidence scores for each element, enabling fully automated document processing pipelines with validation against expected schemas.