AI Model Wiki

    Skip to main content

    AI Glossary

    Essential AI terminology explained simply. 240 terms, A to Z, each linking to its own definition page.

    A

    AI Agent

    An AI agent is a software system that perceives its environment, makes decisions, and takes actions to achieve a specified goal with some degree of autonomy. At its core, an agent receives…

    AI Alignment

    AI alignment is the research discipline concerned with ensuring that an AI system's goals, behaviors, and outputs remain consistent with the intentions and values of the humans who build and use it…

    AI Chip

    An AI chip is a semiconductor device engineered specifically to accelerate the mathematical operations that underpin machine learning—primarily large matrix multiplications, convolutions, and…

    AI Ethics

    AI ethics is the applied branch of ethics that examines the moral responsibilities of those who design, deploy, and use AI systems, as well as the societal consequences that follow from those systems…

    AI Governance

    AI governance refers to the ensemble of laws, regulations, standards, voluntary commitments, and organizational processes that guide how AI systems are developed, deployed, monitored, and held…

    AI Orchestration

    AI orchestration is the discipline of coordinating multiple AI models, data sources, tools, and services into a coherent automated pipeline that accomplishes a goal none of them could achieve…

    AI Provenance

    AI provenance is the documented lineage of an AI system, covering the origin and composition of its training data, the sequence of training and fine-tuning stages it underwent, the versions of code…

    AI Safety

    AI safety is a research and engineering field focused on ensuring that AI systems behave reliably, remain under meaningful human control, and do not cause harm—whether through honest mistakes…

    AI Watermarking

    AI watermarking embeds hidden markers or signatures into content produced by AI systems — images, text, audio, or video — so that the origin of that content can be verified later. Unlike a visible…

    API (Application Programming Interface)

    An Application Programming Interface (API) is a defined contract that lets one piece of software call the functionality of another. In the AI context, a model API exposes endpoints that accept…

    API Rate Limiting

    API rate limiting is the practice of capping how many requests a client can send to a model provider within a defined time window — for example, 60 requests per minute or 1 million tokens per day…

    Activation Function

    An activation function is a mathematical operation applied to a neuron's output that determines whether and how strongly that neuron "fires," introducing non-linearity into the network. Without this…

    Activation Patching

    Activation patching is a mechanistic interpretability technique that isolates the causal contribution of specific internal states in a neural network. A researcher runs a model on two different…

    Adapter Layers

    Adapter layers are small, trainable modules inserted at fixed points inside a frozen pre-trained neural network to enable efficient task-specific fine-tuning. Each adapter typically contains a…

    Agentic AI

    Agentic AI refers to AI systems that can autonomously pursue multi-step goals by planning sequences of actions, invoking external tools, and adapting based on intermediate results—all without…

    Alignment

    Alignment is the field of research and engineering concerned with ensuring that AI systems pursue goals that are consistent with human values, intentions, and safety requirements. The core problem is…

    Attention Mechanism

    The attention mechanism is a component in neural networks that allows the model to weigh the relevance of each part of its input when producing each part of its output. Rather than compressing the…

    Audio Language Model

    An audio language model is a machine learning model that processes audio as a sequence of discrete or continuous tokens, applying the same probabilistic modelling principles used in text language…

    Autoregressive Model

    An autoregressive model generates output one element at a time, where each new element is conditioned on all previously generated elements. In text generation, this means the model produces tokens…

    B

    BLEU Score

    Bilingual Evaluation Understudy (BLEU) is an automatic metric for evaluating the quality of machine-generated text—most commonly machine translation—by measuring how closely it matches one or more…

    Backpropagation

    Backpropagation is the algorithm that computes the gradient of a loss function with respect to every parameter in a neural network by systematically applying the chain rule of calculus from the…

    Batch Size

    Batch size is the number of training examples that are processed together in a single forward and backward pass before the model parameters are updated. Rather than updating weights after every…

    Benchmark

    A benchmark is a standardised evaluation task — or a collection of such tasks — used to measure and compare the capabilities of AI models under controlled, reproducible conditions. A good benchmark…

    Bias

    In AI systems, bias refers to systematic errors or skews in model outputs that arise from imbalances or prejudices present in training data, from the design of the objective function, or from the way…

    Byte-Pair Encoding (BPE)

    Byte-Pair Encoding (BPE) is a data-compression-derived Tokenization algorithm that builds a subword vocabulary by iteratively merging the most frequent adjacent byte or character pairs in a training…

    C

    CLIP

    CLIP (Contrastive Language-Image Pre-training) is a neural network model developed by OpenAI that learns a shared Embedding space for images and text by training on approximately 400 million…

    CLIP (Contrastive Language-Image Pre-training)

    CLIP (Contrastive Language-Image Pre-training) is an OpenAI model that learns joint representations of images and natural-language descriptions by training on around 400 million internet…

    CLIP Model

    The CLIP Model (Contrastive Language-Image Pre-training) is a neural architecture developed by OpenAI that learns to associate images with natural-language descriptions by training on roughly 400…

    CUDA

    CUDA (Compute Unified Device Architecture) is NVIDIA's proprietary parallel computing platform and programming model that exposes GPU hardware to general-purpose software, enabling the thousands of…

    Chain of Thought

    Chain of Thought (CoT) refers to a reasoning pattern in which an AI model generates a sequence of intermediate steps before producing a final answer, rather than jumping directly from prompt to…

    Chain of Thought Prompting

    Chain of Thought Prompting is a specific Prompt Engineering strategy that elicits step-by-step reasoning from a language model by including worked examples of intermediate reasoning in the prompt, or…

    Chain-of-Thought (CoT)

    A prompting technique that encourages AI models to break down complex problems into intermediate reasoning steps before giving a final answer. CoT dramatically improves performance on math, logic…

    Chinchilla Scaling

    Chinchilla Scaling refers to the compute-optimal training laws established by DeepMind's 2022 paper "Training Compute-Optimal Large Language Models," which trained a 70B-parameter model called…

    Circuit Analysis

    In the context of AI interpretability, Circuit Analysis is a methodology for reverse-engineering the internal computations of a neural network by identifying minimal subgraphs—"circuits"—of neurons…

    Compound AI Systems

    Compound AI Systems are architectures that combine multiple AI models, retrieval components, external tools, and deterministic program logic into a single coordinated pipeline to accomplish tasks…

    Compute Optimal Training

    Compute Optimal Training refers to the practice of allocating a fixed computational budget—measured in floating-point operations (FLOPs)—across model size and training data volume so as to minimize…

    Computer Vision

    Computer vision is a field of artificial intelligence concerned with enabling machines to extract meaningful information from images, videos, and other visual inputs. Systems in this field perform…

    Computer-Use Agents

    Computer-use agents are AI systems that can perceive graphical user interfaces — typically via screenshots — and execute actions such as mouse clicks, keyboard input, and scrolling to accomplish…

    Constitutional AI

    Constitutional AI is an alignment approach developed by Anthropic in which a language model is trained to evaluate and revise its own outputs according to a written set of principles called the…

    Content Credentials

    Content Credentials are a standardized framework for embedding verifiable, tamper-evident provenance metadata directly into digital files such as images, videos, audio recordings, and documents…

    Context Length Extension

    Context length extension refers to a collection of techniques that allow a transformer-based model to process sequences significantly longer than the sequence lengths it was trained on, without…

    Context Stuffing

    Context stuffing is the practice of deliberately filling a large portion of a model's available context window with retrieved or pre-selected information such as documents, database records, code…

    Context Window

    The context window is the total number of tokens that a model can consider at one time, spanning both the input prompt and the generated output combined. Any text outside this limit is invisible to…

    Continuous Batching

    Continuous batching is a serving strategy for large language models in which the inference engine dynamically adds new incoming requests to an in-progress batch as individual sequences finish…

    Contrastive Learning

    Contrastive learning is a self-supervised training technique in which a model learns representations by being trained to bring similar data points close together in embedding space while pushing…

    ControlNet

    ControlNet is an architectural extension for diffusion-based image generation models that introduces a trainable copy of a portion of the base model's encoder, conditioned on a spatial control signal…

    Cross Modal Transfer

    Cross-modal transfer is the process by which a model applies knowledge acquired from training on one data modality, such as images, text, or audio, to tasks in a different modality, leveraging shared…

    Curriculum Learning

    Curriculum learning is a training strategy in which examples are presented to a model in a deliberate sequence, typically ordered from simpler or more prototypical instances to harder or more…

    D

    DPO

    DPO (Direct Preference Optimization) is a supervised fine-tuning algorithm that teaches a language model human preferences by training directly on pairs of outputs — one preferred, one rejected — for…

    DPO (Direct Preference Optimization)

    DPO (Direct Preference Optimization) is an alignment fine-tuning algorithm that adjusts a language model's behavior using human preference comparisons without requiring a separately trained reward…

    Data Augmentation

    Data augmentation is a set of techniques that artificially expand a training dataset by creating modified copies of existing examples without collecting new raw data. For images, this includes…

    Data Mixing

    Data mixing refers to deliberate strategies for combining samples from different sources, domains, or synthetic transformations during training to shape the distribution a model learns from. In…

    Data Parallelism

    Data parallelism is a distributed training strategy in which the full model is replicated on each device (GPU or TPU), but each replica processes a different mini-batch of training data in parallel…

    Decoder-Only

    A decoder-only architecture is a Transformer variant that retains only the masked self-attention stack, discarding the separate encoder. Each token can attend to all tokens that precede it but not to…

    Deep Learning

    Deep learning is a branch of machine learning that trains Neural Networks with many layers — typically dozens to hundreds — stacked between the input and output. Each layer learns increasingly…

    Deep Research

    Deep Research is an AI-assisted workflow, and in some products a named feature, in which a language model autonomously plans a multi-step information-gathering process, executes web searches or…

    DeepSpeed ZeRO

    DeepSpeed ZeRO (Zero Redundancy Optimizer) is a memory optimization system developed by Microsoft that eliminates the redundant copies of model state that standard data parallelism creates. In…

    Deepfake

    A deepfake is a piece of synthetic media — video, audio, or a still image — in which a person's likeness, voice, or both have been fabricated or replaced using deep learning techniques. Early…

    Deepfake Detection

    Deepfake detection is the task of classifying whether a piece of media — video, audio, or image — has been synthetically generated or manipulated to impersonate a real person. Detection systems are…

    Diffusion Model

    A diffusion model is a generative model that learns to produce data by being trained to reverse a gradual noise-corruption process. During training, clean examples are progressively destroyed over…

    Direct Preference Optimization

    Direct Preference Optimization (DPO) is a fine-tuning algorithm that aligns a language model with human preferences by directly updating the model's parameters using paired comparison data, without…

    E

    Edge AI

    Edge AI refers to the practice of running artificial intelligence inference and, in some cases, training directly on end-user devices — smartphones, IoT sensors, wearables, autonomous vehicles, or…

    Embedding

    An embedding is a dense numerical vector — an ordered list of floating-point numbers — that encodes the semantic meaning of a piece of data such as a word, sentence, image, or audio clip. The key…

    Emergent Abilities

    Emergent abilities are capabilities that appear in large neural network models but are effectively absent in smaller models trained on the same data with the same objective. They are described as…

    Encoder-Decoder

    An encoder-decoder is a neural network architecture composed of two distinct modules: an encoder that reads the input and compresses it into an intermediate representation, and a decoder that reads…

    Explainability (XAI)

    Explainability, often abbreviated XAI (Explainable Artificial Intelligence), refers to methods and practices that make an AI model's decisions or predictions understandable to humans. A model's…

    F

    FLOPS

    FLOPS — Floating Point Operations Per Second — is the standard unit for measuring raw computational throughput in AI hardware and training runs. One petaFLOP equals one quadrillion floating-point…

    Federated Learning

    Federated learning is a machine learning training paradigm in which a model is updated collaboratively across many decentralised participants — phones, hospital servers, enterprise nodes — without…

    Few Shot Prompting

    Few-shot prompting is an in-context learning technique in which a small number of worked examples — typically two to ten input-output pairs — are embedded directly in the prompt before the actual…

    Few-Shot Learning

    Few-Shot Learning (as a prompting and generalisation technique for large language models) describes the ability to perform a new task after seeing only a small number of demonstrations, either in the…

    Fine-Tuning

    Fine-Tuning is the second stage of the standard two-phase deep learning pipeline, in which a model that has already been pre-trained on a large general corpus is trained further on a smaller…

    Flash Attention

    Flash Attention is a memory-efficient algorithm for computing the scaled dot-product Attention operation that is the core of Transformer models. Standard attention requires materialising the full N x…

    Foundation Model

    A foundation model is a large neural network trained on massive, broadly diverse data — text, images, code, or a combination — in a self-supervised fashion, producing a single general-purpose system…

    Function Calling

    Function calling (also called tool use) is a capability that lets a language model signal, in a structured way, that it wants to invoke an external function — passing typed arguments — rather than…

    G

    GAN (Generative Adversarial Network)

    A Generative Adversarial Network is a framework in which two neural networks — a generator and a discriminator — are trained simultaneously in opposition. The generator's job is to produce synthetic…

    GPU (Graphics Processing Unit)

    A Graphics Processing Unit is a processor built around thousands of small, parallel arithmetic cores rather than a handful of powerful sequential ones. Originally designed to rasterize pixels…

    GRPO

    GRPO (Group Relative Policy Optimisation) is a reinforcement learning algorithm designed specifically for training language models from reward signals, introduced by DeepSeek in 2024 as a more…

    Generative AI

    Generative AI refers to machine learning systems whose primary output is new content — text, images, audio, video, code, or structured data — synthesized by learning statistical patterns from large…

    Gradient Descent

    Gradient descent is the iterative optimization algorithm used to train nearly every modern neural network. At each step, it computes the gradient of the loss function with respect to all model…

    Graph Neural Networks

    Graph Neural Networks (GNNs) are a class of neural networks designed to operate directly on graph-structured data, where entities are represented as nodes and relationships between them are…

    GraphRAG

    GraphRAG (Graph Retrieval-Augmented Generation) is an extension of standard RAG that organises retrieved knowledge as a graph of entities and relationships rather than a flat list of document chunks…

    Groq

    Groq is a semiconductor company that designs Language Processing Units (LPUs) — AI inference chips built around a Tensor Streaming Processor (TSP) architecture optimised specifically for the…

    Grounding

    Grounding is the process of anchoring a language model's outputs to verifiable, external sources of information rather than relying solely on patterns memorised during pretraining. A grounded system…

    Grouped Query Attention

    Grouped Query Attention (GQA) is a memory-efficient variant of the multi-head attention mechanism used in Transformer models. Standard multi-head attention assigns one separate key-value (KV) head…

    Guardrails

    Guardrails are constraints and safety mechanisms layered into an AI system to prevent it from generating outputs that are harmful, factually reckless, discriminatory, or otherwise outside acceptable…

    H

    Hallucination

    Hallucination refers to the tendency of a language model to generate text that is grammatically fluent and contextually plausible but factually incorrect, unverifiable, or entirely fabricated. The…

    Hallucination Rate

    Hallucination rate is a quantitative metric expressing how frequently a given model produces factually incorrect, fabricated, or unverifiable content across a defined evaluation set. It is typically…

    Hugging Face

    Hugging Face is an AI company and open-source platform that functions as the central hub for sharing, discovering, and deploying machine learning models, datasets, and applications. Founded in 2016…

    HumanEval

    HumanEval is a code-generation benchmark introduced by OpenAI in 2021, consisting of 164 hand-written Python programming problems. Each problem includes a function signature, a docstring describing…

    HyDE

    HyDE, or Hypothetical Document Embeddings, is a retrieval technique in which a language model first generates a hypothetical answer document for a query, and that synthetic document — rather than the…

    I

    In Context Learning

    In-context learning (ICL) is the ability of a pretrained language model to perform a new task at inference time by conditioning on examples or instructions placed directly in the prompt, without any…

    Inference

    Inference is the phase of a machine learning model's lifecycle in which the trained model receives new input and produces an output — a prediction, a generated text sequence, a classification label…

    Inpainting

    Inpainting is an image editing technique in which a model fills in a designated region of an image — defined by a binary mask — so that the result is visually coherent with the surrounding, unmasked…

    Instruction Tuning

    Instruction tuning is a supervised Fine-tuning stage in which a pretrained language model is trained on a curated dataset of (instruction, response) pairs, teaching it to interpret and follow…

    J

    JSON Mode

    JSON Mode is a feature offered by many large language model APIs that constrains the model's output to syntactically valid JSON, ensuring every response can be parsed by a machine without error…

    Jailbreak

    A jailbreak is a carefully constructed prompt or interaction sequence designed to cause an AI model to bypass its safety training and produce outputs it was specifically conditioned to refuse — such…

    Jailbreaking

    Jailbreaking refers to the broader category of techniques, methods, and adversarial strategies used to circumvent the safety restrictions embedded in an AI model's training. Where the term…

    K

    KV Cache

    A KV Cache (Key-Value Cache) is a memory optimisation used during autoregressive text generation in transformer models. During the forward pass, every token's attention layer computes a key vector…

    Knowledge Distillation

    Knowledge distillation is a model compression technique in which a smaller network (the student) is trained to reproduce the output distribution of a larger, more capable network (the teacher)…

    L

    LLM Routing

    LLM routing is the practice of directing each incoming query to whichever language model in a fleet is best suited to handle it, rather than sending every request to a single fixed endpoint. A…

    Large Language Model (LLM)

    A Large Language Model (LLM) is a neural network trained on large corpora of text using self-supervised objectives — most commonly next-token prediction — to learn statistical relationships between…

    Latency

    Latency, in the context of AI inference, is the elapsed time between sending a request to a model and receiving a usable response. For generative models two metrics matter most: time-to-first-token…

    Latent Space

    Latent space is the multi-dimensional numerical space in which a machine learning model represents compressed, abstract versions of its input data. After an encoder reduces high-dimensional inputs…

    Learning Rate

    The learning rate is a scalar Hyperparameter that controls the step size used to update a model's weights during each iteration of Gradient Descent. After the Loss Function is computed and…

    LoRA

    LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique for large neural networks that avoids updating the full weight matrices of a pre-trained model. Instead, each target weight…

    LoRA (Low-Rank Adaptation)

    LoRA, short for Low-Rank Adaptation, is a technique that makes fine-tuning large pre-trained models practical on modest hardware by decomposing weight updates into pairs of small matrices rather than…

    Long Context

    Long context refers to the ability of a language model to process and reason over input sequences that span tens of thousands to millions of tokens within a single Context Window. Early Transformer…

    Loss Function

    A loss function (also called a cost function or objective function) is a mathematical expression that quantifies the discrepancy between a model's predictions and the desired ground-truth outputs…

    M

    MCP

    MCP, or Model Context Protocol, is an open standard introduced by Anthropic in 2024 that defines a uniform way for AI models and agents to connect with external tools, data sources, and services…

    MCP Servers

    Model Context Protocol (MCP) Servers are lightweight service processes that expose tools, resources, and prompt templates to AI agents over a standardised JSON-RPC interface. Introduced by Anthropic…

    MMLU

    MMLU, or Massive Multitask Language Understanding, is a benchmark designed to evaluate an AI model's breadth of world knowledge and reasoning by testing it across 57 academic and professional…

    Machine Learning

    Machine learning is the subfield of artificial intelligence concerned with building systems that improve their performance on a task through exposure to data, rather than through explicit rule…

    Mamba

    Mamba is a sequence modeling architecture built on selective state space models (SSMs) rather than the self-attention mechanism used in Transformers. Unlike attention-based models, whose memory and…

    Mamba Architecture

    Mamba Architecture refers to the full neural network design introduced in the 2023 paper "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." It replaces the quadratic self-attention…

    Mechanistic Interpretability

    Mechanistic interpretability is a subfield of AI safety and research that aims to reverse-engineer trained neural networks at the level of individual weights, neurons, and circuits to explain…

    Mixture of Agents

    Mixture of Agents (MoA) is a system design in which multiple independent AI model instances — each acting as an autonomous agent — collaborate on a task, with their outputs aggregated by a…

    Mixture of Experts

    Mixture of Experts (MoE) is a neural network design principle in which a model is composed of multiple specialised sub-networks called experts, plus a learned routing component that decides which…

    Mixture of Experts (MoE)

    Mixture of Experts (MoE) is a neural network architecture in which a model contains multiple specialised sub-networks (experts) and a trainable router that selects a small subset of those experts to…

    Model Card

    A model card is a short document published alongside an AI model that provides structured information about what the model does, how it was built, and where it should or should not be used. Standard…

    Model Collapse

    Model collapse is a degradation process that occurs when an AI model is trained predominantly on data generated by other AI models rather than on original human-produced content. Each generation of…

    Model Distillation

    Model distillation, also called knowledge distillation, is a training technique in which a smaller "student" model is trained to replicate the behaviour of a larger "teacher" model by learning from…

    Model Pruning

    Model pruning is a compression technique that removes weights, neurons, or entire structural components from a trained neural network while attempting to preserve as much of its predictive accuracy…

    Model Sharding

    Model sharding is a parallelism strategy that partitions a large neural network's layers or parameter tensors across multiple accelerators (GPUs or TPUs) so that no single device needs to hold the…

    Multi-Head Attention

    Multi-head attention is an extension of the Self-Attention mechanism in which the input is linearly projected into several separate query, key, and value spaces - called heads - and attention is…

    Multi-agent System

    A multi-agent system is an architecture in which two or more autonomous AI agents operate within a shared environment, each perceiving inputs and taking actions that may affect the others. Agents can…

    Multimodal

    A multimodal AI model is one that can accept and produce more than one type of data - commonly some combination of text, images, audio, video, and structured data - within a single unified…

    N

    NER

    Named Entity Recognition (NER) is an NLP task that locates and classifies named entities within unstructured text into predefined categories such as person names, organisations, geographic locations…

    Natural Language Processing (NLP)

    Natural Language Processing (NLP) is the subfield of artificial intelligence concerned with enabling computers to read, understand, and generate human language in its written and spoken forms. It…

    Neural Network

    A neural network is a parameterised mathematical function loosely inspired by the structure of biological brains, built from layers of interconnected nodes called neurons. Each neuron computes a…

    Neural Scaling Laws

    Neural scaling laws are empirically derived relationships - typically following power laws - that describe how a model's performance improves as one or more of three quantities increases: the number…

    O

    OCR

    Optical Character Recognition (OCR) is the process of detecting and extracting text from raster images - scanned documents, photographs, screenshots, or PDF page renders - and converting it into…

    ONNX Runtime

    ONNX Runtime is an open-source, cross-platform inference engine developed by Microsoft that executes machine learning models stored in the Open Neural Network Exchange (ONNX) format. It applies a…

    Omni Model

    An omni model (sometimes stylised as an "omni" or "any-to-any" model) is an AI system designed to receive and produce multiple modalities - text, audio, images, and video - within a single end-to-end…

    On-device AI

    On-device AI refers to running machine-learning model Inference directly on end-user hardware - smartphones, laptops, tablets, wearables, or embedded systems - rather than sending data to a remote…

    Open Source AI

    Open Source AI refers to AI systems — including models, training code, and sometimes training data — whose components are publicly released under licenses that permit inspection, modification, and…

    Open-weight Model

    An open-weight model is an AI system whose trained parameter values — the numerical weights learned during training — are publicly released so that anyone can download, inspect, run, and modify the…

    OpenAI Operators

    In OpenAI's usage policy framework, an Operator is a company or individual that accesses the OpenAI API to build products and services, as distinguished from the end user who interacts with those…

    Operator Pattern

    The Operator Pattern is an architectural design in which a language model acts as an autonomous intermediary agent that receives a high-level goal from a human user, decomposes it into sub-tasks…

    Outpainting

    Extending an image beyond its original borders by generating new content that seamlessly continues the existing composition. Outpainting uses diffusion models to predict what lies outside the frame…

    Overfitting

    Overfitting occurs when a model learns the training data so thoroughly that it captures noise and idiosyncratic patterns rather than the underlying structure it is meant to generalize from. The…

    P

    PagedAttention

    PagedAttention is a memory management algorithm for the key-value (KV) cache used during transformer inference, developed by the vLLM team at UC Berkeley. During autoregressive text generation, each…

    Parameter

    A parameter is a numerical value stored inside a neural network that is adjusted during training to minimize the model's loss on its training data. Parameters are the actual learned content of the…

    Parameter Efficient Fine Tuning

    Parameter-Efficient Fine-Tuning (PEFT) is a family of techniques for adapting a pretrained model to a new task or domain by updating only a small fraction of its parameters rather than all of them…

    Perplexity

    Perplexity is an information-theoretic metric that quantifies how confidently a language model predicts a held-out text sequence. Formally, it is the exponentiated average negative log-likelihood per…

    Pipeline Parallelism

    Pipeline parallelism is a strategy for training or running neural networks that are too large to fit on a single accelerator by splitting the model's layers into sequential stages, each assigned to a…

    Polysemanticity

    Polysemanticity in neural networks refers to the phenomenon where a single neuron or internal feature activates in response to multiple semantically unrelated inputs. In a fully interpretable…

    Positional Encoding

    Positional encoding is a mechanism that injects information about token order into the input representations of a transformer model, which would otherwise treat the sequence as an unordered set…

    Pre-training

    Pre-training is the first and most computationally intensive phase of building a large AI model, in which the model is trained on a massive, general-purpose corpus - billions of web pages, books, and…

    Prefix Caching

    Prefix caching is an inference optimisation in which the key-value (KV) attention states computed for a fixed prompt prefix - such as a long system prompt or a shared document - are calculated once…

    Prefix Tuning

    Prefix tuning is a parameter-efficient Fine-tuning method in which a small set of continuous, task-specific vectors - the "prefix" - is prepended to the key and value matrices inside each Transformer…

    Probing Classifiers

    Probing classifiers are lightweight supervised models - typically a single linear layer or shallow MLP - trained on top of frozen internal representations extracted from a larger pre-trained model to…

    Process Reward Model

    A Process Reward Model (PRM) is a neural network trained to assign a scalar reward to each intermediate step in a reasoning chain rather than only to the final answer. This contrasts with an outcome…

    Prompt Chaining

    Prompt chaining is an orchestration pattern in which a complex task is decomposed into a sequence of smaller sub-tasks, each handled by a separate LLM call whose output is passed as input to the next…

    Prompt Engineering

    Prompt engineering is the discipline of deliberately crafting the text inputs given to an AI model - including instructions, examples, context, persona specifications, and output constraints - in…

    Prompt Injection

    Prompt injection is a class of adversarial attack against AI-powered systems in which malicious content embedded in external input - a webpage, a document, a user message - overrides or manipulates…

    Proximal Policy Optimization

    Proximal Policy Optimisation (PPO) is a reinforcement learning algorithm that trains an agent's decision-making policy by iteratively collecting experience and updating the policy with gradient…

    Pruning

    Pruning is a model compression technique that identifies and removes parameters - or entire structural units - from a trained Neural Network that contribute little to its outputs, reducing model size…

    Q

    QLoRA

    QLoRA (Quantized Low-Rank Adaptation) is a memory-efficient Fine-tuning technique that combines Quantization of the frozen base model with LoRA adapter training in higher numerical precision…

    Quantization

    Quantization is a model compression technique that represents a neural network's weights and activations using lower-precision numerical formats - typically reducing from 32-bit floating point (FP32)…

    R

    RAG

    Retrieval-Augmented Generation (RAG) is an architecture that supplements a language model's parametric knowledge - the facts baked into its weights during Pretraining - with non-parametric knowledge…

    RLHF

    Reinforcement Learning from Human Feedback (RLHF) is a training technique that uses human preference judgements as a reward signal to align a language model's outputs with human values and…

    RLHF (Reinforcement Learning from Human Feedback)

    Reinforcement Learning from Human Feedback (RLHF) is a training methodology that uses human preference judgments to steer a language model toward outputs that are more helpful, accurate, and aligned…

    ReAct Framework

    ReAct (Reasoning + Acting) is a prompting and training framework for language models that interleaves verbal reasoning traces with discrete actions, allowing a model to plan, execute tool calls…

    Red Teaming

    Red teaming in AI is the structured practice of probing a model or system with adversarial inputs to uncover vulnerabilities, failure modes, biases, and unsafe outputs before deployment. Borrowed…

    Reinforcement Learning

    Reinforcement learning (RL) is a machine-learning paradigm in which an agent learns a policy-a mapping from states to actions-by interacting with an environment and optimising for cumulative reward…

    Reinforcement Learning (RL)

    Reinforcement learning (RL) is a training paradigm where an agent learns by interacting with an environment, receiving scalar reward signals for actions, and iteratively updating its policy to…

    Reranking

    Reranking is a two-stage retrieval strategy in which an initial set of candidate documents or passages-retrieved quickly by a lightweight model-is reordered by a more powerful cross-encoder model…

    Responsible AI

    Responsible AI is an organisational and technical framework that governs how AI systems are designed, deployed, and monitored to ensure they are ethical, fair, transparent, accountable, and safe. It…

    Retrieval Augmented Generation

    Retrieval-Augmented Generation (RAG) is an AI architecture that enhances a generative language model by first querying an external knowledge source-such as a document corpus or database-and injecting…

    Retrieval-Augmented Generation (RAG)

    Retrieval-Augmented Generation (RAG) is a technique that extends a generative language model by coupling it with a retrieval system: at inference time, relevant documents or passages are fetched from…

    Reward Hacking

    Reward hacking-also called specification gaming-occurs when a reinforcement learning agent finds a way to maximise its reward signal through unintended means that satisfy the letter of the reward…

    Reward Model

    A reward model is a neural network trained to predict how much a human would prefer one AI output over another, producing a scalar score that substitutes for direct human judgment during RLHF…

    Ring Attention

    Ring Attention is a distributed Self-Attention algorithm that partitions an extremely long sequence across multiple devices arranged in a logical ring, allowing each device to compute attention over…

    RoPE (Rotary Position Embedding)

    Rotary Position Embedding (RoPE) is a positional encoding scheme for transformer models that represents each token's position by rotating its query and key vectors in a high-dimensional space…

    Rotary Position Embedding

    Rotary Position Embedding (RoPE) encodes each token's position in a sequence by applying a rotation transformation to the query and key vectors inside every attention layer of a transformer. The key…

    S

    STT

    STT, short for speech-to-text, is the technology that converts spoken audio into a written transcript, enabling machines to understand and process human voice input. Modern STT systems are end-to-end…

    Safety Benchmarks

    Safety benchmarks are standardised evaluation suites that measure how well an AI model behaves under potentially harmful, deceptive, or out-of-distribution conditions. They cover a range of failure…

    Scalable Oversight

    Scalable oversight is a research agenda in AI safety concerned with how humans can reliably supervise AI systems whose capabilities, in some domains, may exceed those of the supervisors. The core…

    Scaling Laws

    Scaling laws describe the empirical relationship between a language model's performance and the three main resources poured into training it: compute (measured in FLOPs), dataset size (tokens), and…

    Search-Time Scaling

    Search-time scaling refers to the empirical finding that increasing the amount of search performed at inference time — rather than increasing model parameters or training data — reliably improves…

    Self Consistency Decoding

    Self-consistency decoding is an inference-time technique that improves the reliability of a language model's answers by sampling multiple independent reasoning paths for the same prompt and then…

    Self-Attention

    Self-attention is the mechanism at the heart of transformer architectures that allows every token in a sequence to directly query, and be influenced by, every other token in the same sequence. For…

    Semantic Chunking

    Semantic chunking is a preprocessing strategy for retrieval systems that splits a long document into segments based on meaning rather than on fixed character counts or paragraph breaks. Rather than…

    Semi-supervised Learning

    Semi-supervised learning is a training paradigm that combines a small labelled dataset with a much larger pool of unlabelled data, sitting between purely supervised learning (all examples labelled)…

    SentencePiece

    SentencePiece is an unsupervised text tokenisation library that segments raw Unicode text into subword units without requiring a language-specific pre-tokeniser such as whitespace splitting or a…

    Sentiment Analysis

    Sentiment analysis is a branch of natural language processing that classifies the emotional tone or opinion expressed in a piece of text, typically labeling it as positive, negative, or neutral…

    Sliding Window Attention

    Sliding window attention is a variant of the standard Attention mechanism that restricts each token to attending only to a fixed-size neighborhood of nearby tokens rather than every other token in…

    Soft Prompting

    Soft prompting is a parameter-efficient adaptation technique in which a small set of trainable continuous vectors—often called "soft tokens" or a "prompt prefix"—is prepended to the input Embedding…

    Softmax

    Softmax is a mathematical function that takes a vector of arbitrary real numbers and transforms it into a valid probability distribution: every output value falls between 0 and 1, and all values sum…

    Sparse Mixture of Experts

    Sparse mixture of experts (sparse MoE) is a neural network architecture that replaces a single dense feed-forward block with a large collection of parallel sub-networks, called experts, plus a…

    Speculative Decoding

    Speculative decoding is an Inference acceleration technique that pairs a small, fast "draft" model with a large, accurate "target" model to generate text faster than the target model can alone. The…

    Speech-to-Text (STT)

    Speech-to-text (STT), also called automatic speech recognition (ASR), is the technology that converts a stream of audio containing human speech into a written transcript. Modern systems are…

    Stable Diffusion

    A widely-used open-source image generation model created by Stability AI. Stable Diffusion operates in latent space for efficiency and supports text-to-image, image-to-image, inpainting, and many…

    State Space Model

    A state space model (SSM) is a mathematical framework for sequence modeling, rooted in classical control theory, that represents a system through a compact hidden state vector that evolves over time…

    State Space Model (SSM)

    A state space model (SSM) is a mathematical framework for sequence modeling, rooted in classical control theory, that represents a system through a compact hidden state vector that evolves over time…

    Superposition

    In the context of neural network interpretability, superposition refers to the phenomenon where a model encodes more independent concepts than it has neurons, by representing each concept as a…

    Supervised Learning

    Supervised learning is a machine learning paradigm in which a model is trained on a dataset of labeled input-output pairs, learning to map inputs to the correct outputs by minimising a Loss Function…

    Synthetic Data

    Synthetic data is artificially generated information — text, images, tabular records, sensor readings — produced by algorithms, simulations, or generative models rather than collected from the real…

    System Prompt

    A system prompt is a privileged instruction block passed to a language model before any user turn begins, used to establish the model's persona, task scope, output format, and behavioural constraints…

    T

    TPU (Tensor Processing Unit)

    A Tensor Processing Unit (TPU) is a custom application-specific integrated circuit (ASIC) designed by Google to accelerate the matrix multiplications and tensor operations that dominate neural…

    TTS

    Text-to-Speech (TTS) is an AI-driven synthesis pipeline that converts written text into natural-sounding spoken audio. Modern TTS systems use neural architectures — typically combining a spectrogram…

    Temperature

    Temperature is a scalar Hyperparameter that controls how sharply or flatly a language model's probability distribution is peaked before a token is sampled. Mathematically it divides the raw logits by…

    Tensor Parallelism

    Tensor parallelism is a distributed training and inference strategy that partitions the weight matrices — tensors — of a neural network's layers across multiple GPUs, so that each device holds and…

    TensorRT-LLM

    TensorRT-LLM is an open-source inference library from NVIDIA that compiles and optimises LLMs specifically for NVIDIA GPUs, exposing a Python API that wraps CUDA kernels, fused attention operators…

    Test-Time Compute

    Test-time compute refers to the computational resources expended during inference rather than during training. It is the practice of allocating additional processing budget at query time so a model…

    Text Classification

    Text classification is the task of assigning one or more predefined labels to a piece of text based on its content, intent, or sentiment. A classifier learns a mapping from an Embedding or token…

    Text-to-Image

    Text-to-image is a generative AI capability in which a model synthesises a pixel-level image conditioned on a natural-language description, with no hand-crafted rules or templates. Modern systems…

    Text-to-Speech (TTS)

    Text-to-speech (TTS) is the AI task of synthesising natural-sounding spoken audio from written text, converting a sequence of characters or Tokens into a waveform that captures human prosody, rhythm…

    Throughput

    Throughput, in the context of AI inference, is the number of tokens (or complete requests) a deployed model can process per unit of time — typically measured as tokens per second or requests per…

    Token

    The basic unit of text that language models process. A token can be a word, part of a word, or a punctuation mark. For example, "unhappiness" might be split into "un", "happiness". Models have limits…

    Tokenization

    Tokenization is the preprocessing step that converts raw text into a sequence of integer IDs — tokens — that a neural network can ingest. Rather than operating on characters or dictionary words, most…

    Tokenizer

    A tokenizer converts raw text into a sequence of tokens—discrete units such as words, subwords, or individual characters—that a neural network can process numerically. Modern tokenizers used in large…

    Tool Use

    Tool use is the capability that lets an AI model reach outside its own weights to interact with external systems—search engines, calculators, code interpreters, REST APIs, or databases—and…

    Tool Use (Function Calling)

    Function calling (also called tool use) is a protocol through which an AI model signals that it wants to invoke a specific external function, providing structured arguments in a machine-readable…

    Tool Use in LLMs

    Tool use in LLMs refers to the ability of a large language model to detect when a user's request requires external information or actions, generate a structured invocation of a predefined tool (such…

    Tool-Use Reasoning

    Tool-use reasoning is the capability of a language model to determine, mid-generation, that it should call an external function, format a structured tool invocation, incorporate the returned result…

    Top-k Sampling

    Top-k sampling is a decoding strategy for language models in which, at each generation step, the probability distribution over the entire vocabulary is truncated to only the k highest-scoring tokens…

    Top-p

    Top-p, also called nucleus sampling, is a text-generation strategy where, instead of fixing the number of candidate tokens, you fix the cumulative probability mass you're willing to sample from…

    Top-p (Nucleus Sampling)

    Top-p (nucleus sampling) is a decoding parameter that restricts token selection to the smallest set of candidates whose cumulative Softmax probability sums to at least p. At each generation step…

    Training

    Training is the iterative process by which an AI model learns to perform a task by repeatedly adjusting its parameters to minimise a loss function measured against labelled or self-supervised data…

    Transfer Learning

    Transfer learning is a machine learning methodology in which a model trained on one large task is reused as the starting point for a different, often narrower task, rather than training from random…

    Transformer

    The Transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need" that replaced recurrent and convolutional designs for sequence modelling. Its core innovation…

    Transformer Architecture

    Transformer architecture refers to the full structural blueprint—encoder blocks, decoder blocks, or both—built on Self-Attention and feed-forward sub-layers that was introduced in 2017 and now…

    Tree of Thought

    Tree of Thought (ToT) is a prompting framework that structures model reasoning as a branching search tree rather than a single linear chain of steps. At each decision point the model generates…

    U

    Underfitting

    Underfitting occurs when a model is too simple to capture the true structure of the data it is trained on, resulting in high error on both the training set and any held-out evaluation data. The model…

    Unsupervised Learning

    Unsupervised learning is a category of machine learning in which a model discovers structure in data without relying on labelled examples or explicit feedback signals. Instead of learning from…

    V

    VAE (Variational Autoencoder)

    A Variational Autoencoder (VAE) is a generative model that learns a probabilistic mapping between raw data — images, audio, text — and a compact, structured latent space. Unlike a plain autoencoder…

    Vector Database

    A vector database is a data store purpose-built for indexing, storing, and querying high-dimensional floating-point vectors — the kind produced when an embedding model encodes a document, image, or…

    Vision Language Model

    A Vision Language Model (VLM) is an AI system that jointly processes visual inputs — photographs, screenshots, diagrams, document scans — and natural language, enabling tasks that require reasoning…

    Vision-Language Model (VLM)

    Vision-Language Model (VLM) refers to the same class of AI systems described under Vision Language Model: multimodal architectures that unify image understanding with natural language generation and…

    Voice Cloning

    Voice cloning is the process of training or conditioning a speech synthesis model on a small sample of a target speaker's audio — typically a few seconds to a few minutes — so that the model can…

    vLLM

    vLLM is an open-source inference engine for large language models that achieves high throughput and low latency through two key innovations: PagedAttention and continuous batching. PagedAttention…

    W

    Weight Quantization

    Weight quantization is the process of representing a neural network's learned parameters at a lower numerical precision than the full 32-bit or 16-bit floating-point format used during training. In…

    Whisper

    Whisper is an automatic speech recognition (ASR) model developed by OpenAI and released as open-source in 2022. It was trained on 680,000 hours of multilingual audio scraped from the internet, which…

    Z

    Zero Shot Prompting

    Zero-shot prompting is a technique where you ask a language model to perform a task by providing only an instruction — no worked examples, no demonstrations, and no task-specific fine-tuning. The…

    Zero-Shot Learning

    Zero-shot learning, as the term is used in the LLM context, refers to a large language model's ability to follow an instruction and perform a task correctly without being shown any worked examples…

    Frequently Asked Questions