CUDA (Compute Unified Device Architecture) is NVIDIA's proprietary parallel computing platform and programming model that exposes GPU hardware to general-purpose software, enabling the thousands of cores in a GPU to execute many operations simultaneously rather than sequentially. For AI workloads, CUDA underpins the key libraries cuDNN for deep neural network primitives and cuBLAS for linear algebra, which frameworks such as PyTorch and TensorFlow rely on to perform matrix multiplications and convolutions efficiently on NVIDIA hardware. Because virtually all major deep learning frameworks have been built atop CUDA, NVIDIA GPUs became the default compute substrate for AI training and inference, a position that has proven difficult for competing platforms to displace.

Think of it like a traffic management system built specifically for a particular brand of highway. No matter what kind of vehicle you drive, if you want to use that highway at full speed, you need the manufacturer's on-ramp system, and CUDA is that on-ramp for NVIDIA GPUs.

A PyTorch model is moved to GPU with a single call (model.to("cuda")), after which all tensor operations dispatch through CUDA kernels automatically. Writing a custom kernel directly requires CUDA C:

__global__ void add_vectors(float* a, float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) c[i] = a[i] + b[i];
}

CUDA is foundational to understanding why Quantization targets specific CUDA data types, how Inference serving systems exploit tensor parallelism across multiple GPUs, and why Gradient Descent at scale depends on GPU memory bandwidth. Competing platforms such as AMD's ROCm and Intel's oneAPI exist but have significantly smaller ecosystem support.

You will encounter CUDA in error messages during environment setup, in GPU memory management discussions, and in performance profiling whenever a neural network is trained or served on NVIDIA hardware.