AI Model Wiki

    Skip to main content
    7 min read
    Lesson 2 of 6

    Running Llama Locally

    From the course Meta Llama for Developers

    Running Llama on your own machine gives you full control, privacy, and zero API costs. Here's how to set it up.

    Option 1: Ollama (Easiest)

    Ollama is the simplest way to run Llama locally:

    # Install Ollama
    curl -fsSL https://ollama.com/install.sh | sh
    
    # Pull and run Llama 3.1
    ollama run llama3.1
    
    # Or pull a specific size
    ollama run llama3.1:70b
    

    Ollama handles model downloading, quantization, and serving automatically.

    Option 2: llama.cpp

    For maximum performance and control:

    # Clone and build
    git clone https://github.com/ggerganov/llama.cpp
    cd llama.cpp && make
    
    # Run with a GGUF model file
    ./main -m models/llama-3.1-8b-q4_k_m.gguf -p "Explain quantum computing"
    

    llama.cpp runs quantized models on CPU or GPU with excellent performance.

    Option 3: Hugging Face Transformers

    For Python developers who want programmatic control:

    from transformers import AutoModelForCausalLM, AutoTokenizer
    import torch
    
    model_id = "meta-llama/Llama-3.1-8B-Instruct"
    tokenizer = AutoTokenizer.from_pretrained(model_id)
    model = AutoModelForCausalLM.from_pretrained(
        model_id, torch_dtype=torch.bfloat16, device_map="auto"
    )
    
    inputs = tokenizer("Write a haiku about coding:", return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=50)
    print(tokenizer.decode(outputs[0]))
    

    Hardware Requirements

    ModelMin VRAMRecommended
    8B (Q4)6 GB8 GB GPU
    8B (FP16)16 GB24 GB GPU
    70B (Q4)40 GB2x 24 GB GPUs
    405B (Q4)200 GB+8x A100 80GB

    Quantization (Q4, Q5, Q8) reduces memory by 2-4x with minimal quality loss.

    Apple Silicon Performance

    Apple M-series chips use unified memory shared between CPU and GPU, which means a MacBook Pro with 64 GB RAM can comfortably run a 70B model at Q4 quantization — something that would require two high-end NVIDIA GPUs on a PC. Throughput is typically 15-25 tokens/sec on an M3 Max, sufficient for interactive applications. This makes Macs an excellent local development platform for large models without GPU investment.

    Pitfall: Chat Template Format

    Each Llama variant uses a specific chat template — a structured way to format the system prompt and conversation turns. If you feed raw text to the model without the correct template, outputs will be poor or nonsensical. Hugging Face's tokenizer.apply_chat_template() handles this automatically. For llama.cpp, use the correct prompt format flag matching the model variant. Mismatched templates are responsible for a significant fraction of "the model gives bad answers" reports on developer forums.

    Key Takeaways

    • •Ollama is the easiest way to run Llama locally with one command
    • •llama.cpp offers maximum performance with quantized GGUF models
    • •Hugging Face Transformers provides full Python programmatic control
    • •8B quantized models run on consumer GPUs with 6-8 GB VRAM
    0 of 6 lessons marked complete

    Progress is kept in this browser only.