Optimize resource utilization and operate AI workloads efficiently on Kubernetes.

Cluster Autoscaling

Node Autoscaler for GPU Nodes

# Cluster Autoscaler configuration
annotations:
  cluster-autoscaler.kubernetes.io/safe-to-evict: "false"

# Node pool configuration for GPU autoscaling
nodePool:
  name: gpu-pool
  config:
    machineType: n1-standard-96
    accelerators:
    - acceleratorCount: 8
      acceleratorType: nvidia-tesla-a100
  autoscaling:
    enabled: true
    minNodeCount: 0  # Scale to zero when idle
    maxNodeCount: 32

Job Scheduling Optimization

Priority Classes

# Production inference - highest priority
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: inference-critical
value: 1000000
---
# Training - can be preempted
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: training-batch
value: 100
preemptionPolicy: Never

Resource Quotas

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: ml-team
spec:
  hard:
    requests.nvidia.com/gpu: "16"
    limits.nvidia.com/gpu: "32"
    requests.memory: "512Gi"
    limits.memory: "1Ti"

Monitoring and Alerts

# GPU utilization alert
groups:
- name: gpu-alerts
  rules:
  - alert: LowGPUUtilization
    expr: nvidia_gpu_duty_cycle < 50
    for: 30m
    annotations:
      summary: "GPU utilization below 50% for 30 minutes"
      description: "Consider scaling down or consolidating workloads"

Cost Optimization Strategies

• Use spot/preemptible nodes for training jobs (70% savings)
• Scale GPU nodes to zero when no training workloads
• Right-size GPU requests using historical data
• Implement job queuing to maximize GPU utilization
• Use time-sharing for development workloads