Three tools dominate the local AI landscape. Each serves different needs and skill levels.

Ollama — The Docker of AI Models

Ollama is the simplest way to run AI locally. It manages model downloads, handles GPU acceleration, and exposes an OpenAI-compatible API.

Installation:

• macOS: brew install ollama
• Linux: curl -fsSL https://ollama.com/install.sh | sh
• Windows: Download from ollama.com

Running Your First Model:

ollama run llama3.2        # Download and chat with Llama 3.2
ollama run mistral         # Try Mistral 7B
ollama run codellama       # Code-focused model
ollama run gemma2          # Google's Gemma 2
ollama run phi3            # Microsoft's Phi-3 (small but capable)

Key Commands:

ollama list                # See downloaded models
ollama pull deepseek-r1    # Download without running
ollama rm mistral          # Delete a model
ollama serve               # Start the API server (port 11434)

Ollama as an API Server: Once running, Ollama exposes an API compatible with the OpenAI format:

curl http://localhost:11434/v1/chat/completions \
  -d '{"model": "llama3.2", "messages": [{"role": "user", "content": "Hello!"}]}'

This means any app that supports the OpenAI API can use your local model — just change the base URL.

LM Studio — The GUI Approach

LM Studio provides a beautiful desktop app for running local models, with no command line needed.

Features:

• Browse and download models from Hugging Face with one click
• Chat interface similar to ChatGPT
• Side-by-side model comparison
• Built-in API server (OpenAI-compatible)
• Real-time performance monitoring (tokens/sec, VRAM usage)
• Supports GGUF, GGML model formats

Best For: Users who prefer a visual interface, comparing models, and quick experimentation.

llama.cpp — Maximum Performance

llama.cpp is the foundational C++ library that most local AI tools are built on. It provides the fastest inference and most configuration options.

Why Use llama.cpp Directly:

• Absolute maximum performance
• Custom quantization options
• Fine-grained memory management
• Support for latest model architectures immediately
• Can run on minimal hardware (even Raspberry Pi!)

When to Use Each:

Other Notable Tools:

• GPT4All — Simple desktop app, works offline, privacy-focused
• Jan — Open-source ChatGPT alternative for local models
• Kobold.cpp — Popular for creative writing and roleplay
• LocalAI — Docker-based, drop-in OpenAI API replacement