Optimization decisions have direct dollar consequences at production scale. This lesson maps technical choices to cloud cost structures.

Understanding Your Cost Drivers

GPU Compute Costs

At current spot pricing on major clouds (mid-2024):

• A10G (24 GB): ~$0.75/hr spot, $1.40/hr on-demand
• A100 (80 GB): ~$2.20/hr spot, $3.40/hr on-demand
• H100 (80 GB): ~$4.50/hr spot, $7.00/hr on-demand

A model that runs on one A10G instead of one A100 saves ~66% on compute. Quantization often enables exactly this jump.

Tokens Per Dollar Analysis

def cost_per_1k_tokens(gpu_cost_per_hour, tokens_per_second):
    tokens_per_hour = tokens_per_second * 3600
    cost_per_token = gpu_cost_per_hour / tokens_per_hour
    return cost_per_token * 1000

# FP16 Llama-2-7B on A100
fp16_cost = cost_per_1k_tokens(2.20, 30)
print(f"FP16 on A100: ${fp16_cost:.4f} per 1K tokens")

# INT4 on A10G
int4_cost = cost_per_1k_tokens(0.75, 67)
print(f"INT4 on A10G: ${int4_cost:.4f} per 1K tokens")

# vLLM AWQ on A10G (50 concurrent users at 140 tok/s total)
vllm_cost = cost_per_1k_tokens(0.75, 140)
print(f"vLLM AWQ on A10G: ${vllm_cost:.4f} per 1K tokens")

Typical result: FP16 on A100 costs ~$0.020/1K tokens; vLLM AWQ on A10G costs ~$0.0015/1K tokens — a 13x cost reduction for roughly equivalent quality.

Batching Strategy by Traffic Pattern

Low traffic (< 5 req/min): Static batching

• Batch size 1-4
• Use single GPU, scale to zero when idle
• Serverless GPU (Modal, RunPod serverless) ideal

Medium traffic (5-50 req/min): Continuous batching

• vLLM or TGI with default settings
• 1-2 A10G instances, auto-scaling
• Target GPU utilization 60-80%

High traffic (50+ req/min): Tensor parallelism + load balancing

# vLLM multi-GPU configuration
llm = LLM(
    model="meta-llama/Llama-2-70b-hf",
    tensor_parallel_size=4,       # spread across 4 GPUs
    pipeline_parallel_size=1,
    gpu_memory_utilization=0.90,
    max_num_seqs=256,             # max concurrent sequences
)

Prompt Caching

Repeated system prompts waste compute. Most inference frameworks support prefix caching:

# vLLM enables prefix caching automatically when prompt prefix is identical
# Ensure system prompt is always at the FRONT of the message
messages = [
    {"role": "system", "content": SHARED_SYSTEM_PROMPT},  # cached
    {"role": "user",   "content": user_query},             # unique
]

With prefix caching, a 1,000-token system prompt costs near-zero after the first request in a batch. At scale, this alone can cut TTFT (time to first token) by 40-60%.

Flash Attention 2

from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # requires pip install flash-attn
)
# Provides 2-4x memory savings + 10-20% latency improvement on long sequences