Adapter layers are small, trainable modules inserted at fixed points inside a frozen pre-trained neural network to enable efficient task-specific fine-tuning. Each adapter typically contains a down-projection that compresses the hidden dimension, a non-linearity, and an up-projection that restores the original dimension—together adding only a fraction of a percent of the original parameter count. Because the backbone weights remain unchanged, multiple task-specific adapters can be swapped in and out of the same base model without retraining it from scratch each time.

Think of it like adding a removable lens filter to a camera. The camera body and its optics stay identical; you just clip on a different filter for portrait, landscape, or macro work, and the output shifts accordingly without any permanent modification.

A minimal adapter block in PyTorch looks like this:

class Adapter(nn.Module):
    def __init__(self, d_model, bottleneck=64):
        super().__init__()
        self.down = nn.Linear(d_model, bottleneck)
        self.up   = nn.Linear(bottleneck, d_model)
    def forward(self, x):
        return x + self.up(F.relu(self.down(x)))  # residual connection

In practice, adapters are inserted after the attention sub-layer and after the feed-forward sub-layer of each Transformer block, letting the model learn task-specific adjustments while all other gradients are blocked.

Adapter layers are part of the same family of parameter-efficient methods as LoRA, and they interact with concepts like Fine-tuning, Foundation Model reuse, Knowledge Distillation, and Hyperparameter tuning (adapter bottleneck size is the key hyperparameter). They differ from full fine-tuning in that the original weights are never updated.

You will encounter adapter layers in papers on parameter-efficient fine-tuning (PEFT), in the Hugging Face PEFT library documentation, and in multi-task serving setups where a single hosted model needs to serve dozens of specialized variants.