Understanding hardware requirements is essential for running AI models locally. This guide covers what you need and how to maximize performance with the hardware you have.

The Key Resource: VRAM

VRAM (Video RAM) on your GPU is the primary bottleneck for local AI. Models must fit in VRAM for fast GPU-accelerated inference.

VRAM Requirements by Model Size:

What is Quantization?

Quantization reduces model precision from 16-bit floats to lower precision (8-bit, 4-bit, even 2-bit), dramatically reducing memory requirements with surprisingly small quality loss.

Think of it like image compression:

• FP16 (full precision) — RAW photo. Maximum quality, maximum size.
• Q8 (8-bit) — High-quality JPEG. Nearly indistinguishable from original.
• Q4_K_M (4-bit) — Good JPEG. Slight quality loss, 75% size reduction. Sweet spot for most users.
• Q2 (2-bit) — Heavily compressed. Noticeable quality loss but runs on minimal hardware.

Common Quantization Formats:

• Q4_K_M — Best balance of quality and size (recommended default)
• Q5_K_M — Slightly better quality, slightly more VRAM
• Q8_0 — Near-full quality, twice the size of Q4
• Q3_K_S — Aggressive compression for limited VRAM
• IQ4_XS — Newer format with better quality-per-bit

CPU vs GPU Inference

GPU Inference (Recommended):

• 10-100x faster than CPU
• Requires VRAM to hold the model
• NVIDIA GPUs (CUDA) have best support
• AMD GPUs work with ROCm (improving)
• Apple Silicon uses Metal (excellent performance)

CPU Inference (Fallback):

• Works on any computer
• Uses regular RAM instead of VRAM
• Much slower (1-5 tokens/sec vs 30-100+ on GPU)
• Viable for small models or if you're patient

Apple Silicon: The Dark Horse

Apple's M-series chips have unified memory (shared between CPU and GPU), making them surprisingly good for local AI:

Apple Silicon achieves excellent tokens/sec thanks to high memory bandwidth.

Recommended Hardware Builds:

Budget (~$300-500): Used RTX 3060 12GB

• Runs 7B-13B models at good speed
• Best value entry point

Mid-Range (~$800-1200): RTX 4070 Ti Super 16GB

• Runs 13B models comfortably, 34B quantized
• Good for daily use and development

High-End (~$1600-2000): RTX 4090 24GB

• Runs 34B models well, 70B with heavy quantization
• Fast enough for real-time applications

Apple Alternative: M2/M3 Pro MacBook (16-36GB)

• Runs 7B-13B models silently and efficiently
• Great for laptops — no discrete GPU needed
• Unified memory makes it simple

RAM Considerations

System RAM matters too:

• Model loading and preprocessing uses system RAM
• CPU inference uses system RAM directly
• 32GB recommended minimum for serious local AI work
• 64GB if you want to run models + other applications comfortably