Running Llama Locally
From the course Meta Llama for Developers
Running Llama on your own machine gives you full control, privacy, and zero API costs. Here's how to set it up.
Option 1: Ollama (Easiest)
Ollama is the simplest way to run Llama locally:
# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# Pull and run Llama 3.1
ollama run llama3.1
# Or pull a specific size
ollama run llama3.1:70b
Ollama handles model downloading, quantization, and serving automatically.
Option 2: llama.cpp
For maximum performance and control:
# Clone and build
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make
# Run with a GGUF model file
./main -m models/llama-3.1-8b-q4_k_m.gguf -p "Explain quantum computing"
llama.cpp runs quantized models on CPU or GPU with excellent performance.
Option 3: Hugging Face Transformers
For Python developers who want programmatic control:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "meta-llama/Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
inputs = tokenizer("Write a haiku about coding:", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))
Hardware Requirements
| Model | Min VRAM | Recommended |
|---|---|---|
| 8B (Q4) | 6 GB | 8 GB GPU |
| 8B (FP16) | 16 GB | 24 GB GPU |
| 70B (Q4) | 40 GB | 2x 24 GB GPUs |
| 405B (Q4) | 200 GB+ | 8x A100 80GB |
Quantization (Q4, Q5, Q8) reduces memory by 2-4x with minimal quality loss.
Apple Silicon Performance
Apple M-series chips use unified memory shared between CPU and GPU, which means a MacBook Pro with 64 GB RAM can comfortably run a 70B model at Q4 quantization — something that would require two high-end NVIDIA GPUs on a PC. Throughput is typically 15-25 tokens/sec on an M3 Max, sufficient for interactive applications. This makes Macs an excellent local development platform for large models without GPU investment.
Pitfall: Chat Template Format
Each Llama variant uses a specific chat template — a structured way to format the system prompt and conversation turns. If you feed raw text to the model without the correct template, outputs will be poor or nonsensical. Hugging Face's tokenizer.apply_chat_template() handles this automatically. For llama.cpp, use the correct prompt format flag matching the model variant. Mismatched templates are responsible for a significant fraction of "the model gives bad answers" reports on developer forums.
Key Takeaways
- •Ollama is the easiest way to run Llama locally with one command
- •llama.cpp offers maximum performance with quantized GGUF models
- •Hugging Face Transformers provides full Python programmatic control
- •8B quantized models run on consumer GPUs with 6-8 GB VRAM
Progress is kept in this browser only.