20 min
Lesson 3 of 5
Cluster Operations and Monitoring
From the course GPU Cluster Management
Operate and monitor production GPU clusters for maximum uptime and utilization.
Monitoring Stack
Prometheus + Grafana
# DCGM Exporter for GPU metrics
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: dcgm-exporter
spec:
template:
spec:
containers:
- name: dcgm-exporter
image: nvcr.io/nvidia/k8s/dcgm-exporter:3.1.3-3.1.4-ubuntu20.04
env:
- name: DCGM_EXPORTER_LISTEN
value: ":9400"
Key Metrics
- GPU utilization per node
- Memory usage and bandwidth
- Temperature and power consumption
- Network throughput (IB/RoCE)
- Job queue depth and wait times
Maintenance Best Practices
- Schedule maintenance windows for firmware updates
- Implement gradual rollouts for driver updates
- Monitor thermal management and cooling
- Track GPU error rates and RMA trends
- Automate health checks and alerting
Key Takeaways
- •DCGM Exporter + Prometheus provides cluster-wide GPU monitoring
- •Track utilization, memory, power, and network metrics in Grafana dashboards
- •Scheduled maintenance and gradual rollouts minimize cluster downtime
- •Automated health checks detect and remediate GPU failures proactively
0 of 5 lessons marked complete
Progress is kept in this browser only.