Deploying Models to Production

Inference Endpoints

Hugging Face's managed deployment service:

from huggingface_hub import InferenceClient

client = InferenceClient(
    model="meta-llama/Llama-3.1-8B-Instruct",
    token="hf_your_token",
)

response = client.text_generation(
    "Explain quantum computing in simple terms",
    max_new_tokens=200,
)

Deployment Options

Text Generation Inference (TGI)

Self-host high-performance inference:

docker run --gpus all -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct

Optimization Techniques

• Quantization: Reduce model size (FP16, INT8, INT4) with minimal quality loss
• Batching: Process multiple requests together for throughput
• KV-Cache: Reuse computed attention values for faster generation
• Speculative decoding: Use a small model to predict, large model to verify

Model Sharing

Push your fine-tuned model to the Hub:

model.push_to_hub("my-org/my-fine-tuned-model")
tokenizer.push_to_hub("my-org/my-fine-tuned-model")

Common Pitfall: Not Quantizing Before Deployment

Deploying a full float32 model uses 4 bytes per parameter — a 7B model occupies 28GB of VRAM. Always quantize for deployment:

from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=quantization_config,
    device_map="auto",
)

4-bit quantization reduces a 7B model to ~4GB VRAM with less than 2% quality degradation on most tasks.

Best Practice: Export ONNX for Production Inference

For high-throughput production serving, export models to ONNX format. ONNX models run on ONNX Runtime, which applies hardware-specific optimizations (fused operators, kernel fusion) that PyTorch does not perform by default:

python -m optimum.exporters.onnx --model bert-base-uncased ./onnx_output

Hugging Face's Optimum library handles ONNX export and quantization for Transformers models.

Named Product Reference: AWS Inferentia2

For very-high-throughput inference, AWS Inferentia2 chips (available via EC2 Inf2 instances) offer 4x better price-performance than GPU instances for many Transformer workloads. Hugging Face's Optimum library includes optimum-neuron, which compiles Transformers models for Inferentia2 with minimal code changes. This is the production choice for large-scale text classification, embedding, or NER at millions of requests per day.