Inference-time search is a technique in which a language model generates multiple candidate reasoning paths or output sequences during a single inference call, then uses a scoring mechanism to select the best one before returning a response. Instead of greedily decoding one token at a time, the model explores a tree or beam of possibilities and evaluates them.

Think of it like a chess engine that does not play the first move that comes to mind. It projects forward, evaluates many lines, and picks the one with the highest expected outcome.

In practice, a system might use beam search to retain the top-k partial outputs at each decoding step, or it might generate N complete responses and pass them to a verifier. For math problems, a Process Reward Model can score each step of a candidate chain-of-thought, pruning paths that seem wrong early and expanding promising ones. This is distinct from training-time search, which occurs during reinforcement learning from human feedback.

Inference-time search is closely tied to Test-Time Compute since both involve spending more resources at query time. It relates to Chain of Thought because the paths being searched are typically explicit reasoning traces. Process Reward Models act as the scoring function that guides the search. The technique can also apply to RAG pipelines, where multiple retrieval queries are tried and results are ranked. Foundation Models with large Context Windows benefit more from inference-time search because they can hold longer candidate traces.

You will encounter this term in research papers on reasoning models and in engineering discussions about latency versus accuracy trade-offs at serving time.