Probing classifiers are lightweight supervised models - typically a single linear layer or shallow MLP - trained on top of frozen internal representations extracted from a larger pre-trained model to test whether specific information is encoded at that layer. The core idea is that if a simple classifier can predict, say, part-of-speech tags from a hidden-layer Embedding, the underlying model must have learned that syntactic structure internally, even without being explicitly taught it. Probing studies have become a standard interpretability tool for understanding what Neural Network layers represent across BERT, GPT-style LLMs, and vision transformers.

Think of it like tapping into a telephone cable at different junction boxes and listening to the signal at each point - you are not modifying the cable, just eavesdropping to figure out what kind of information passes through each section.

A minimal probing experiment in PyTorch:

import torch
from transformers import BertModel, BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
bert = BertModel.from_pretrained("bert-base-uncased").eval()

inputs = tokenizer("The cat sat on the mat", return_tensors="pt")
with torch.no_grad():
    outputs = bert(**inputs, output_hidden_states=True)

# extract layer-6 hidden states as features for a POS-tag classifier
layer6_reps = outputs.hidden_states[6]  # shape: (1, seq_len, 768)
# then train a logistic regression on layer6_reps -> POS labels

Probing connects to Attention analysis (both interrogate model internals), Fine-tuning (probing reveals what adapts), and Overfitting concerns - if the probe itself overfits, results are unreliable. Researchers also use probing alongside Perplexity measures to distinguish fluency from linguistic knowledge.

You will encounter probing classifiers in NLP interpretability papers, model audits, and fairness investigations - they are a go-to tool whenever a team needs evidence about what a model has implicitly learned or where demographic biases might be encoded in its representations.