Put what you have learned in this course into practice with the structured exercise below. Working through it end-to-end is the single best way to consolidate the concepts.

Concrete Starter Project: Build an AI Receipt Scanner

This project covers the full pipeline from image upload to structured data — combining the OCR, validation, and storage patterns from all four lessons.

What You'll Build

A web form where users upload receipt images. The app extracts structured purchase data (store, date, items, totals) using GPT-4o vision and stores the results in a database.

Step 1 — Accept the Upload and Pre-Process

import sharp from "sharp";
import { createHash } from "crypto";

export async function POST(req: Request) {
  const formData = await req.formData();
  const file = formData.get("receipt") as File;

  // Validate type and size
  if (!["image/jpeg", "image/png", "image/webp"].includes(file.type)) {
    return Response.json({ error: "Unsupported format" }, { status: 400 });
  }
  if (file.size > 10 * 1024 * 1024) {
    return Response.json({ error: "File too large (max 10MB)" }, { status: 400 });
  }

  // Resize to reduce tokens
  const buffer = Buffer.from(await file.arrayBuffer());
  const resized = await sharp(buffer)
    .resize({ width: 1536, height: 1536, fit: "inside", withoutEnlargement: true })
    .jpeg({ quality: 85 })
    .toBuffer();

  const imageHash = createHash("sha256").update(resized).digest("hex");
  const base64 = resized.toString("base64");
  return extractReceipt(base64, imageHash);
}

Step 2 — Extract with GPT-4o Vision

async function extractReceipt(base64Image: string, imageHash: string) {
  // Check cache first
  const cached = await db.query("SELECT data FROM receipt_cache WHERE image_hash = $1", [imageHash]);
  if (cached.rows.length) return Response.json(cached.rows[0].data);

  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{
      role: "user",
      content: [
        { type: "image_url", image_url: { url: `data:image/jpeg;base64,${base64Image}`, detail: "high" } },
        { type: "text", text: 'Extract receipt data. Return JSON: { "store_name": string, "date": "YYYY-MM-DD", "items": [{ "name": string, "qty": number, "price": number }], "subtotal": number, "tax": number, "total": number, "currency": string }' },
      ],
    }],
    response_format: { type: "json_object" },
  });

  const data = JSON.parse(response.choices[0].message.content!);
  await db.query("INSERT INTO receipt_cache (image_hash, data) VALUES ($1, $2)", [imageHash, JSON.stringify(data)]);
  return Response.json(data);
}

Step 3 — Validate the Schema

import { z } from "zod";

const ReceiptSchema = z.object({
  store_name: z.string().min(1),
  date: z.string().regex(/^d{4}-d{2}-d{2}$/),
  items: z.array(z.object({ name: z.string(), qty: z.number(), price: z.number().positive() })).min(1),
  subtotal: z.number().positive(),
  tax: z.number().min(0),
  total: z.number().positive(),
  currency: z.string().length(3),
});

const validated = ReceiptSchema.safeParse(data);
if (!validated.success) {
  return Response.json({ error: "Extraction failed validation", issues: validated.error.issues }, { status: 422 });
}

Self-Check Questions

• What happens with handwritten receipts where OCR accuracy drops significantly?
• How would you handle receipts in languages other than English?
• What's your fallback when GPT-4o returns valid JSON but with obviously wrong numbers?

Stretch Goal

Add a spending summary dashboard: group extracted receipts by store and month, and display total spend per category using item names and an AI classifier.