Vision-Language Models (VLMs) accept images alongside text prompts, enabling applications from document understanding to visual question answering.

Practical Applications

Document AI: Extract structured data from invoices, receipts, forms, and contracts. VLMs understand layout, tables, handwriting, and stamps.

from openai import OpenAI
import base64

client = OpenAI()

with open("invoice.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Extract all line items from this invoice as JSON with fields: description, quantity, unit_price, total"},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_data}"}}
        ]
    }],
)

Chart & Graph Analysis: Upload charts, dashboards, or infographics and ask questions about trends, anomalies, or specific data points.

UI Understanding: Feed screenshots of applications to generate code, identify bugs, or create test descriptions.

Medical Imaging: Analyze X-rays, MRIs, dermatology images (with appropriate disclaimers and human oversight).

Choosing a VLM

Best Practices

• Resolution matters: Higher resolution images give better results for text-heavy content. Use the highest resolution your budget allows.
• Multiple images: Send multiple views/pages in a single request when context spans multiple images.
• Prompting for vision: Be specific about what you want extracted. "Describe this image" gives vague results; "List all text visible in the top-right corner" is precise.
• Grounding: Ask the model to reference specific regions ("In the top-left chart...") to verify it's reading the right area.