Prometheus & OpenTelemetry Metrics

Amazon SageMaker detailed observability runs an AWS-managed OpenTelemetry (OTel) Collector on every instance backing your endpoint. The collector gathers per-GPU accelerator metrics, host-level system metrics, and Prometheus metrics scraped from the model container, and exports them to CloudWatch, where you query them with PromQL. This gives you finer-grained visibility than the standard CloudWatch metrics covered in Observability for Amazon SageMaker — including per-GPU utilization on multi-GPU instances and metrics emitted directly by the Deepgram engine.

With detailed observability enabled, three metric sources publish to CloudWatch’s OTel-compatible metric store:

Every series carries SageMaker resource labels, including aws.sagemaker.endpoint.name, the variant name, and the instance ID, so you can filter and group across a scaled-out fleet.

These metrics work with AWS Marketplace deployments: the collector runs on the host, outside the model container, so it is unaffected by the network isolation that Marketplace model packages require.

Detailed observability is on by default for newly created endpoints, publishing every 60 seconds — no configuration is needed to start using it on a new deployment.

To set it explicitly — for example to change the publish frequency, or to add it to an endpoint created before the feature launched — use MetricsConfig on the endpoint configuration, the same field used for enhanced metrics. The publish frequency accepts 10, 30, 60 (default), 120, 180, 240, or 300 seconds.

aws sagemaker create-endpoint-config \
  --endpoint-config-name YOUR_CONFIG_NAME \
  --production-variants file://production-variants.json \
  --metrics-config '{"EnableDetailedObservability": true, "MetricPublishFrequencyInSeconds": 60}' \
  --region YOUR_AWS_REGION

import boto3

sagemaker = boto3.client("sagemaker")

sagemaker.create_endpoint_config(
    EndpointConfigName="YOUR_CONFIG_NAME",
    ProductionVariants=[...],  # same variants as your existing config
    MetricsConfig={
        "EnableDetailedObservability": True,
        "MetricPublishFrequencyInSeconds": 60,
    },
)

To change the setting on an endpoint that is already serving traffic — including opting out with "EnableDetailedObservability": false — create a new endpoint configuration with the same production variants plus MetricsConfig, then run update-endpoint. The update is a blue/green deployment and the endpoint stays InService; see Update an Amazon SageMaker Endpoint.

The metrics land in CloudWatch’s OTel metric store, which is queried with PromQL rather than the classic get-metric-statistics API.

In the CloudWatch console, open Metrics and use the PromQL query editor to explore the new series. Start by listing what a busy endpoint reports:

DCGM_FI_DEV_GPU_UTIL

Then filter to one endpoint. The SageMaker resource labels use OTel dotted names, so quote them in PromQL:

DCGM_FI_DEV_GPU_UTIL{"aws.sagemaker.endpoint.name"="YOUR_ENDPOINT_NAME"}

CloudWatch exposes a standard Prometheus query API for these metrics at https://monitoring.YOUR_AWS_REGION.amazonaws.com/api/v1/query, authenticated with SigV4 (service name monitoring). Any tool that can sign requests works; for example with awscurl:

awscurl --service monitoring --region YOUR_AWS_REGION \
  "https://monitoring.YOUR_AWS_REGION.amazonaws.com/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL"

The discovery endpoints work too — list every metric name the account is receiving:

awscurl --service monitoring --region YOUR_AWS_REGION \
  "https://monitoring.YOUR_AWS_REGION.amazonaws.com/api/v1/label/__name__/values"

Because the API is Prometheus-compatible, you can also point Grafana or other Prometheus-native observability tools at it. See the AWS documentation for supported integrations.

sagemaker_endpoint_health reports the container’s own view of whether it can serve inference. Exactly one state reports 1:

sagemaker_endpoint_health{state="critical"}

A 1 means the container cannot recover without being replaced. Query the family without a state filter to see the current state across a fleet and read whichever series is 1. See Read health as a metric for what each state means.

For earlier warning, sagemaker_time_to_critical_seconds counts down the seconds remaining before a container would enter critical. It reports -1 when nothing is counting, so guard the comparison or the alarm fires on every healthy container:

sagemaker_time_to_critical_seconds >= 0 and sagemaker_time_to_critical_seconds < 120

See Warning before a container is written off.

• Observability for Amazon SageMaker — standard CloudWatch metrics, logs, and alarms
• Metrics Guide — reference for the Deepgram API and Engine Prometheus metrics
• Deepgram Enhanced Metrics — usage and billing metrics emitted by the Deepgram container
• Detailed observability for SageMaker AI endpoints (AWS)
• Getting started with detailed observability (AWS)