AI Model Wiki

    Skip to main content
    7 min read
    Lesson 3 of 5

    Running Models Locally

    From the course Open-Source AI Models: Self-Hosting & Deployment

    You don't need a data center to run AI models. Modern tools let you run capable models on a laptop, desktop, or single GPU server.

    Ollama: The Easiest Way to Start

    Ollama packages models into a simple CLI tool:

    1. Install: Download from ollama.com (macOS, Linux, Windows)
    2. Run a model: "ollama run llama3.1" in your terminal
    3. Chat: Type naturally and get responses instantly
    4. API: Ollama runs a local API server on port 11434

    Ollama automatically downloads models and handles quantization. It supports dozens of models and runs on CPU (slower) or GPU (fast).

    Hardware Requirements

    A general guide for running quantized models:

    • 7B models (Q4) — 4-6 GB RAM, any modern CPU. Runs on a MacBook Air.
    • 13B models (Q4) — 8-10 GB RAM. Comfortable on a 16GB laptop.
    • 34B models (Q4) — 20+ GB RAM. Needs a decent desktop or workstation.
    • 70B models (Q4) — 40+ GB RAM. Requires a dedicated GPU server or Mac Studio.

    Quantization Explained

    Full-precision models use 16-bit (FP16) weights. Quantization reduces this to 8-bit, 4-bit, or even 2-bit, dramatically reducing memory requirements with modest quality loss.

    • Q8 — Minimal quality loss. 50% memory reduction.
    • Q4_K_M — Good balance. ~75% memory reduction. Most popular for local use.
    • Q2 — Significant quality loss but runs on very limited hardware.

    Open WebUI: A ChatGPT-Like Interface

    For a polished chat experience with local models:

    1. Install Docker
    2. Run the Open WebUI container pointing to your Ollama instance
    3. Get a full-featured chat interface with conversation history, model switching, and file uploads

    LM Studio: GUI Alternative

    LM Studio provides a visual application for:

    • Browsing and downloading models from Hugging Face
    • Running models with a chat interface
    • Adjusting parameters (temperature, context length)
    • Running a local API server compatible with the OpenAI API format

    Performance Tips

    • GPU acceleration — CUDA (NVIDIA) or Metal (Apple Silicon) dramatically speed up inference
    • Context length — Shorter context windows are faster; only extend when needed
    • Batch processing — Group multiple requests for better GPU utilization
    • Model selection — Use the smallest model that meets your quality requirements

    Key Takeaways

    • •Ollama is the simplest way to run models locally with a single command
    • •7B quantized models run on laptops; 70B models need dedicated GPU servers
    • •Quantization (Q4, Q8) trades minimal quality for dramatic memory savings
    • •Open WebUI and LM Studio provide polished chat interfaces for local models
    • •GPU acceleration (CUDA/Metal) is essential for responsive local inference
    0 of 5 lessons marked complete

    Progress is kept in this browser only.