SentencePiece is an unsupervised text tokenisation library that segments raw Unicode text into subword units without requiring a language-specific pre-tokeniser such as whitespace splitting or a part-of-speech tagger. It trains a segmentation model — using either Byte-Pair Encoding or a unigram language model objective — directly on the character stream of a corpus, learning vocabulary entries that balance frequency (common words stay whole) against coverage (rare words are broken into meaningful pieces). Because SentencePiece treats the input as a stream of Unicode codepoints with a special whitespace symbol, it handles scripts that lack explicit word boundaries — Chinese, Japanese, Thai — just as naturally as space-separated European languages.

Think of it like a universal luggage-packing algorithm that learns the optimal box sizes by studying millions of suitcases, without needing to know in advance whether the items are shirts or books — it just finds the recurring shapes and optimises around them.

import sentencepiece as spm

spm.SentencePieceTrainer.train(
    input="corpus.txt",
    model_prefix="mymodel",
    vocab_size=32000,
    model_type="bpe",           # or "unigram"
    character_coverage=0.9995   # higher for CJK corpora
)
sp = spm.SentencePieceProcessor(model_file="mymodel.model")
print(sp.encode("Hello, world!", out_type=str))
# ['▁Hello', ',', '▁world', '!']

SentencePiece is the tokeniser behind LLMs including LLaMA, T5, and many multilingual models; it connects directly to Embedding tables and affects Perplexity scores. Tokenization choices made here propagate into Context Window efficiency — a vocabulary that over-fragments a language wastes tokens and increases inference cost.

You will encounter SentencePiece whenever you load a HuggingFace tokeniser for a non-GPT model family, or when you need to train a custom tokeniser for a low-resource or mixed-script language.