Monitor Compute Engine instances and Slurm clusters

Clustered GPUs

This document explains how to use Cloud Monitoring dashboards to monitor A4X Max, A4X, A4, A3 Ultra, and A3 Mega instances that you created by using reservation-bound capacity. Using these dashboards helps you identify and troubleshoot performance bottlenecks in your standalone Compute Engine instances or Slurm clusters, minimizing downtime in your workloads.

By creating custom dashboards or using prebuilt Monitoring dashboards, you can monitor the following:

Compute instance health
GPU performance
Network transmission efficiency
Network efficiency among blocks and sub-blocks
Machine learning (ML) workload efficiency
Straggler detection
Unresponsive workload detection

To monitor clusters Cluster Director, see Monitor cluster performance with prebuilt dashboards.

Before you begin

Before monitoring your workload, if you haven't already done so, complete the following steps:

Deploy a workload that you can monitor. To learn which workloads are supported, see the limitations in this document. To learn how to deploy a workload, see Deployment options overview.
Learn about the Google Cloud services for monitoring workloads:
- The metrics in this document use Monitoring dashboards. Learn about Monitoring dashboards, Monitoring retention periods, and Monitoring pricing.
- Straggler detection also provides log entries in Cloud Logging. Learn about Logging interfaces, Logging retention periods, and Logging pricing.

When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

Limitations

The metrics in this document are only supported for workloads that run on compute instances that meet all the following criteria:
- The compute instances must be created as either standalone Compute Engine instances or as part of a Slurm cluster.
- The compute instances must have been created by using reservation-bound capacity.
- The compute instances must use the A4X Max, A4X, A4, A3 Ultra, or A3 Mega machine series.
  - However, straggler detection also supports virtual machine (VM) instances that use the A3 Mega machine series.
Caution: Although metrics might appear for other machine series than A4X Max, A4X, A4, A3 Ultra, or A3 Mega, those metrics might be incorrect.
Known issues for infrastructure metrics:
- Some compute instances might not display GPU Power Consumption metrics.
- Some compute instances might not display GPU Temperature or GPU Thermal Margin metrics, or they might display NaN instead of metrics.
- Metrics might not appear in the Google Cloud console for up to seven minutes after they're collected.
- You might see incorrect metrics for machine series other than A4X Max, A4X, A4, A3 Ultra, or A3 Mega.

The metrics in this document are only supported for workloads that run on compute instances that meet all of the following criteria:

The compute instances must be created as either standalone Compute Engine instances or as part of a Slurm cluster.
The compute instances must have been created by using reserved capacity.
The compute instances must use the A4X Max, A4X, A4, A3 Ultra, or A3 Mega machine series.

To monitor ML workload metrics, you must set up monitoring for your workload.

Straggler detection limitations

Straggler detection metrics have the following additional limitations:

For supported machine series other than A3 Mega, straggler detection only supports compute instances that enable the Collective Communication Analyzer (CoMMA) library to export NCCL telemetry to Google Cloud services. For more information, see CoMMA overview.
Straggler detection typically takes up to 10 minutes to report a straggler.
Unlike the other metrics in this document, you can't filter straggler detection metrics for your projects by cluster, block, sub-block, or compute instance. However, you can filter queries for straggler detection logs by the ID of one or more compute instances that are suspected stragglers.

Unresponsive workload detection limitations

Unresponsive workload detection metrics only support compute instances that use the Collective Communication Analyzer (CoMMA) library to export NCCL telemetry to Google Cloud services. For more information, see CoMMA overview.

Required roles

To get the permissions that you need to monitor metrics for AI Hypercomputer workloads, ask your administrator to grant you the following IAM roles:

To view metrics in Cloud Monitoring: Monitoring Editor (roles/monitoring.editor) on the project
To view straggler detection logs in Logging: Logs Viewer (roles/logging.viewer) on the project

For more information about granting roles, see Manage access to projects, folders, and organizations.

These predefined roles contain the permissions required to monitor metrics for AI Hypercomputer workloads. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to monitor metrics for AI Hypercomputer workloads:

To view dashboards: monitoring.dashboards.get on the project
To create dashboards: monitoring.dashboards.create on the project
To view log entries: logging.logEntries.list on the project

You might also be able to get these permissions with