Groq is a semiconductor company that designs Language Processing Units (LPUs) — AI inference chips built around a Tensor Streaming Processor (TSP) architecture optimised specifically for the sequential, memory-bandwidth-constrained workload of autoregressive token generation. Unlike GPUs, which excel at highly parallel batch compute, Groq's chips minimise the latency of generating one token at a time, making them particularly well-suited for real-time AI assistants and low-latency API serving. The company operates GroqCloud, a public API that runs open-weight models like Llama 3, Mixtral, and Gemma at speeds commonly exceeding 500–800 tokens per second per user — roughly 5–10× faster than typical GPU-backed services.

Think of it like comparing a wide freight motorway (GPU: enormous throughput when moving many trucks in parallel) to a single-lane racing circuit engineered for pure lap-time speed (LPU: built to move one token — one car — as fast as physically possible, regardless of how many others are waiting).

Using the Groq API is nearly identical to any OpenAI-compatible endpoint:

from groq import Groq
client = Groq()  # uses GROQ_API_KEY env var
chat = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Explain diffusion models in one paragraph."}]
)
print(chat.choices[0].message.content)

Groq sits at the intersection of Inference (its core purpose), LLM serving infrastructure, Quantization (Groq compiles models to fixed-point formats for its hardware), and Temperature / sampling settings that govern each generation request. It contrasts with general-purpose Foundation Model training hardware, which still dominates NVIDIA GPU clusters.

You will encounter Groq when evaluating inference providers for latency-sensitive applications — real-time voice assistants, coding copilots, or any product where the delay between prompt submission and first token materially affects user experience.