Data mixing refers to deliberate strategies for combining samples from different sources, domains, or synthetic transformations during training to shape the distribution a model learns from. In pretraining large language models, it typically means blending web text, books, code, scientific papers, and other corpora at carefully tuned proportions. At the sample level, methods like Mixup interpolate between two input examples and their labels simultaneously, creating smooth decision boundaries.

Think of it like a chef calibrating a spice blend: the quality of each ingredient matters, but so do the ratios. Too much of one source and the model overfits that domain; too little and the model underrepresents it. The final blend determines what flavors the model can reproduce at inference.

A Mixup implementation in PyTorch looks like this:

import numpy as np

lam = np.random.beta(0.4, 0.4)
mixed_x = lam * x_a + (1 - lam) * x_b
mixed_y = lam * y_a + (1 - lam) * y_b
loss = criterion(model(mixed_x), mixed_y)

For LLM pretraining, data mixing is usually expressed as token-budget weights — for example, 45% web, 25% books, 15% code, 15% other — applied before sampling each batch.

Data mixing is tightly coupled to Pretraining and Overfitting, since the mixture controls both the capability range and the generalization of the resulting model. It is also relevant to Fine-tuning, where mixing task-specific data with general data prevents catastrophic forgetting. Decisions about mixing ratios are treated as Hyperparameters and are often ablated systematically.

You will encounter data mixing when reading foundation model technical reports, where mixture recipes are one of the most impactful but least standardized design choices disclosed.