Navigating the Model Hub

Finding Models

from huggingface_hub import HfApi

api = HfApi()
models = api.list_models(
    filter="text-generation",
    sort="downloads",
    limit=10,
)
for model in models:
    print(f"{model.id}: {model.downloads} downloads")

Model Cards

Every model has a card with:

• Description and intended use
• Training data and methodology
• Performance benchmarks
• Limitations and biases
• License information

Tokenizers Deep Dive

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

# Tokenize
tokens = tokenizer("Hello, how are you?", return_tensors="pt")
print(tokens.input_ids)  # tensor([[128000, 9906, 11, ...]])

# Decode
text = tokenizer.decode(tokens.input_ids[0])
print(text)  # "Hello, how are you?"

Tokenizer Concepts

• Vocabulary: The set of all tokens the model knows
• Subword tokenization: Words split into smaller units (BPE, WordPiece)
• Special tokens: [CLS], [SEP], <|endoftext|> for model control
• Attention masks: Tell the model which tokens to attend to
• Token limits: Context window = max tokens in a single input

Caching & Offline Use

# Models are cached locally after first download
# Default: ~/.cache/huggingface/

# Force offline mode
import os
os.environ["TRANSFORMERS_OFFLINE"] = "1"

Common Pitfall: Ignoring the Context Window Limit

Every tokenizer has a maximum sequence length (e.g., 512 tokens for BERT, 4096 for Llama 2, 131072 for Llama 3.1). Inputs longer than this limit are silently truncated by default. For document processing, chunk inputs to stay within the limit:

inputs = tokenizer(
    long_text,
    max_length=512,
    truncation=True,
    return_overflowing_tokens=True,  # Returns all chunks, not just the first
    stride=50,  # Overlap between chunks for context preservation
    return_tensors="pt",
)

Best Practice: Use the datasets Library for Efficient Data Loading

For tokenizing large datasets, Hugging Face's datasets library applies tokenization in parallel using Apache Arrow and caches the result — far faster than processing raw text files in Python:

from datasets import load_dataset

dataset = load_dataset("csv", data_files="training_data.csv")

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True, max_length=512)

tokenized = dataset.map(tokenize, batched=True, num_proc=4)

Named Product Reference: Sentence Transformers Library

The sentence-transformers library (built on top of Hugging Face Transformers) is the easiest way to generate semantic embeddings. It wraps models like all-MiniLM-L6-v2 and BAAI/bge-large-en-v1.5 with a clean API specifically designed for embedding text pairs, semantic similarity, and retrieval tasks. It handles pooling, normalization, and batching automatically — use it instead of the raw AutoModel API for any embedding use case.