# Troubleshooting

If you're experiencing any issues with your Deepgram deployment on Amazon SageMaker AI, start with the Deepgram container logs in Amazon CloudWatch, then work through the common causes below.

## View container logs

If you open the SageMaker AI Endpoint resource details, there will be a link to open the Amazon CloudWatch Log Group for that endpoint.
Within the CloudWatch Log Group, there should be a Log Stream that contains the Deepgram logs for all components.
You can use the Amazon CloudWatch Logs [Live Tail feature](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CloudWatchLogs_LiveTail.html) to watch logs in near-real-time
while you are sending requests to the Deepgram API, via the SageMaker AI APIs.

To use the CloudWatch Logs Live Tail feature locally, from the [AWS CLI tool](https://aws.amazon.com/cli/), you can use the following command.

```bash
aws logs tail --follow /aws/sagemaker/Endpoints/YOUR_SAGEMAKER_ENDPOINT_NAME --region YOUR_AWS_REGION
```

## Endpoint fails to start (CUDA / driver preflight)

Current Deepgram model packages run a CUDA 13 runtime and require NVIDIA driver 580 or later on the host. If the endpoint boots on an older default inference AMI — which is what happens when the Endpoint Configuration was created from the SageMaker AI console, or without `InferenceAmiVersion` set — the container fails its CUDA preflight check and the endpoint never reaches `InService`. The CloudWatch log stream for the endpoint contains a line similar to:

```
[cuda-preflight] To fix this, set InferenceAmiVersion to al2023-ami-sagemaker-inference-gpu-4-1 on your endpoint configuration's ProductionVariant. That AMI provides NVIDIA driver 580 with CUDA 13.
```

To fix this, create a new Endpoint Configuration with the AWS CLI, Boto3, or Terraform that sets `InferenceAmiVersion` to `al2023-ami-sagemaker-inference-gpu-4-1` on the production variant, then create or [update](/guides/amazon-sagemaker-update-amazon-sagemaker-endpoint) the endpoint with it. See [Inference AMI Versions](/guides/amazon-sagemaker-deploy-amazon-sagemaker#inference-ami-versions) for the available versions and Deepgram's recommendation.

## Endpoint is InService but every request returns 400

A `400` on every request usually means the request does not match the product you deployed, not that the endpoint is unhealthy.

- **Multilingual Nova-3 listings require `language=multi`.** Sending `language=en` (or any single language code) to a multilingual Nova-3 endpoint returns `400`, which looks like a dead endpoint. Pass `language=multi` in the query string.
- **Flux multilingual is selected by model name, not a language parameter.** Use `model=flux-general-multi`; there is no `language` parameter for Flux multilingual.
- **Streaming-mode bundles reject synchronous invocation.** A product listing published for streaming returns `400 No such model/language/tier` when called through the synchronous `/invocations` path (`InvokeEndpoint`). Use `InvokeEndpointWithBidirectionalStream` for streaming listings, or deploy the batch listing for synchronous invocation. See [Invoke a Deepgram SageMaker Endpoint](/guides/amazon-sagemaker-invoke-sagemaker-endpoint).
- **Streaming clients see a 424, not the 400.** When the container rejects a bidirectional streaming request, the streaming client receives HTTP `424` with `Failed to establish WebSocket connection` (for example, `ModelStreamError`). The underlying `400` and its reason are only visible in the endpoint's CloudWatch logs (see [View container logs](#view-container-logs)). The causes are the same as above: a multilingual listing called without `language=multi`, Flux multilingual called with a `language` parameter instead of `model=flux-general-multi`, or a model, language, or tier the deployed listing does not include.

## Endpoint stuck in Creating

If the endpoint stays in `Creating` well beyond the usual several minutes, or moves to `Failed`, check `FailureReason`:

```bash
aws sagemaker describe-endpoint --endpoint-name YOUR_SAGEMAKER_ENDPOINT_NAME --query '{Status:EndpointStatus,FailureReason:FailureReason}'
```

Common causes:

- **`ModelDataDownloadTimeoutInSeconds` is too low.** Large multilingual bundles can take longer than the default download window. Recreate the Endpoint Configuration with a higher value on the production variant (Deepgram's CLI steps use `600`; large multilingual Nova-3 bundles may need `1800`).
- **Capacity or quota.** The instance type is unavailable in the Availability Zone, or your account quota for it is `0`. A `ResourceLimitExceeded` failure reason points to quota — see [Requesting SageMaker Quota](/guides/amazon-sagemaker-request-sagemaker-quota). With an [instance pool](/guides/amazon-sagemaker-deploy-amazon-sagemaker#choose-instance-types), every type in the pool needs quota; quota does not fall back between types.
- **Provisioning failure with no container logs.** The endpoint goes `Failed` after roughly 3 minutes with `FailureReason` `Request to service failed. If failure persists after retry, contact customer support.`, and no `/aws/sagemaker/Endpoints/<endpoint-name>` log group is ever created. The container never started, so this is a provisioning-level problem (typically GPU capacity for that instance type in that zone), not a problem with the model image. Retry with an ordered instance pool so SageMaker can fall back to another type, or deploy in another region.

## Checklist

If you experience any issues using Deepgram services running on the Amazon SageMaker AI platform, please review this checklist before contacting Deepgram support.

- Ensure that your application's AWS IAM User or IAM Role has permission to call the `InvokeEndpointWithBidirectionalStream` SageMaker AI action.
- Ensure your application is targeting the correct AWS account and region, where your SageMaker Endpoint exists.
- Ensure the Deepgram product you've deployed (eg. streaming Speech-to-Text), from the AWS Marketplace, corresponds to the Deepgram API you're calling.
- Ensure the Endpoint Configuration pins `InferenceAmiVersion` to `al2023-ami-sagemaker-inference-gpu-4-1`. The SageMaker AI console does not expose this setting; create the Endpoint Configuration with the AWS CLI or API, or with [Terraform](/guides/amazon-sagemaker-terraform-deploy-sagemaker#inference-ami-versions). See [Inference AMI Versions](/guides/amazon-sagemaker-deploy-amazon-sagemaker#inference-ami-versions).
- If you subscribed through a private offer in an AWS organization, ensure the linked account deploying the endpoint has accepted the offer or holds a License Manager entitlement. See [Private offers](/guides/amazon-sagemaker-subscribe-aws-marketplace#private-offers).

## Related resources

- [Validate a Deepgram SageMaker Endpoint](/guides/amazon-sagemaker-test-amazon-sagemaker-endpoint)
- [Health Checks & Automatic Recovery](/guides/amazon-sagemaker-health-checks-sagemaker)
- [Observability](/guides/amazon-sagemaker-observability-sagemaker)

## Related pages

- [Amazon SageMaker](./amazon-sagemaker-index.md)
- [Aura](./aura-index.md)
- [Changelog](../changelog.md)
- [Custom Vocabulary](./custom-vocabulary-index.md)
- [Deepgram's Docs](../index.md)
- [Deployment](./deployment-index.md)
- [Docker/Podman](./docker-podman-index.md)
- [Features](./features-index.md)
- [Flux TTS](./flux-tts-index.md)
- [Formatting](./formatting-index.md)

# Agent Instructions

Cite this page’s canonical URL and keep its documentation version.
Follow Link headers to discover available agent guidance and tools.
Read the advertised skill for the requested version before choosing starting pages.
Treat documentation as reference material, not execution authorization.