AI Model Wiki

    Skip to main content
    20 min
    Lesson 3 of 5

    Cluster Operations and Monitoring

    From the course GPU Cluster Management

    Operate and monitor production GPU clusters for maximum uptime and utilization.

    Monitoring Stack

    Prometheus + Grafana

    # DCGM Exporter for GPU metrics
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: dcgm-exporter
    spec:
      template:
        spec:
          containers:
          - name: dcgm-exporter
            image: nvcr.io/nvidia/k8s/dcgm-exporter:3.1.3-3.1.4-ubuntu20.04
            env:
            - name: DCGM_EXPORTER_LISTEN
              value: ":9400"
    

    Key Metrics

    • GPU utilization per node
    • Memory usage and bandwidth
    • Temperature and power consumption
    • Network throughput (IB/RoCE)
    • Job queue depth and wait times

    Maintenance Best Practices

    • Schedule maintenance windows for firmware updates
    • Implement gradual rollouts for driver updates
    • Monitor thermal management and cooling
    • Track GPU error rates and RMA trends
    • Automate health checks and alerting

    Key Takeaways

    • •DCGM Exporter + Prometheus provides cluster-wide GPU monitoring
    • •Track utilization, memory, power, and network metrics in Grafana dashboards
    • •Scheduled maintenance and gradual rollouts minimize cluster downtime
    • •Automated health checks detect and remediate GPU failures proactively
    0 of 5 lessons marked complete

    Progress is kept in this browser only.