Skip to main content
Deepgram's Docs

Search documentation

Type to search this documentation.

On this pageOverview

Prerequisites

Deploying Deepgram on Google Cloud Platform (GCP) requires some preparation. In this section, you will learn how to provision a managed Kubernetes Cluster where you will deploy Deepgram products. You will need to perform some of these steps in the Google Cloud Console and some in your local terminal.

Make sure you have completed the requirements in the Self-Hosted Introduction.

GPU availability has been extremely limited across cloud providers, including GCP. You may need to request a GPU quota if you are not able to provision spot GPU instances in your node pools.

kubectl

Section titled “kubectl”

The Kubernetes command-line tool, kubectl, allows you to run commands against Kubernetes clusters. You can use kubectl to deploy applications, inspect and manage cluster resources, and view logs.

Install locally using the official Kubernetes guides .

gcloud CLI

Section titled “gcloud CLI”

The gcloud CLI provides programmatic access to manage your GCP services. Certain steps in this guide are enabled by this tool, although many of the same actions can be performed manually in the Google Cloud Console.

  1. Follow the installation guide to install the CLI locally.
  2. Once installed, run gcloud init to configure the CLI with access to your GCP account and project.

Cluster Management with gcloud container clusters

Section titled “Cluster Management with gcloud container clusters”

The gcloud container clusters command group allows you to create and manage GKE clusters.

Certain steps in this guide use these commands, although many of the same actions can be performed manually in the Google Cloud Console.

Kubernetes Packages with helm

Section titled “Kubernetes Packages with helm”

Helm is the package manager for Kubernetes. A package in Kubernetes is defined by a Helm Chart, which helps you define, install, and upgrade even the most complex Kubernetes application.

We use Helm to install several components in this guide. See the installation guide for details on how to install locally.

Creating a Cluster

Section titled “Creating a Cluster”

Google Kubernetes Engine (GKE) is a managed Kubernetes service to run Kubernetes in GCP. In the cloud, GKE automatically manages the availability and scalability of the Kubernetes control plane nodes responsible for scheduling containers, managing application availability, storing cluster data, and other key tasks.

  1. Create a new GKE cluster with gcloud, and get the zones where your cluster is created.

    Bash
    CLUSTER_NAME=deepgram-self-hosted
CLUSTER_LOCATION=us-west1
gcloud container clusters create $CLUSTER_NAME \
    --location $CLUSTER_LOCATION \
    --num-nodes 1 \
    --enable-autoscaling \
    --machine-type n1-standard-2 \
    --addons=GcePersistentDiskCsiDriver

CLUSTER_ZONES=$(
     gcloud container clusters describe $CLUSTER_NAME \
         --location $CLUSTER_LOCATION \
         --format="value(locations.join(','))"
)
ENGINE_NP_ZONE=$(echo "$CLUSTER_ZONES" | cut -d',' -f1)

  2. Create separate node pools for each Deepgram component (API, Engine, License Proxy). Adjust the machine types and node counts according to your needs. You may wish to consult your Deepgram Account Representative in planning your cluster’s capacity.

    Bash
    gcloud container node-pools create api-pool \
    --cluster $CLUSTER_NAME \
    --location $CLUSTER_LOCATION \
    --num-nodes 1 \
    --enable-autoscaling \
    --max-nodes 3 \
    --machine-type n1-standard-4 \
    --node-labels k8s.deepgram.com/node-type=api

gcloud container node-pools create engine-pool \
    --cluster $CLUSTER_NAME \
    --region $CLUSTER_LOCATION \
    --node-locations $ENGINE_NP_ZONE \
    --num-nodes 1 \
    --enable-autoscaling \
    --max-nodes 8 \
    --machine-type g2-standard-8 \
    --accelerator=type=nvidia-l4,count=1,gpu-driver-version=latest \
    --node-labels k8s.deepgram.com/node-type=engine

gcloud container node-pools create license-proxy-pool \
    --cluster $CLUSTER_NAME \
    --location $CLUSTER_LOCATION \
    --num-nodes 1 \
    --enable-autoscaling \
    --max-nodes 2 \
    --machine-type n1-standard-2 \
    --node-labels k8s.deepgram.com/node-type=license-proxy

  3. Create a dedicated namespace for Deepgram resources.

    Bash
    kubectl create namespace dg-self-hosted
kubectl config set-context --current --namespace=dg-self-hosted

Configure Namespace Resource Quotas for GKE

Section titled “Configure Namespace Resource Quotas for GKE”

When deploying workloads in a non-default namespace (such as dg-self-hosted), GKE does not automatically provision quotas for the system-node-critical and system-cluster-critical priority classes in that namespace. GPU driver DaemonSets (and other node-critical system workloads) rely on these priorities to schedule correctly.

Create a ResourceQuota in your Deepgram namespace to enable these priorities:

ResourceQuota Resource
apiVersion: v1
kind: ResourceQuota
metadata:
  name: gcp-critical-pods
  namespace: dg-self-hosted
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
spec:
  # Set a limit on the number of pods allowed with critical priority classes in this namespace.
  # Pick a value >= total nodes that will run the NVIDIA driver DaemonSet, plus a small buffer.
  hard:
    pods: 5
  scopeSelector:
    matchExpressions:
    - operator: In
      scopeName: PriorityClass
      values: ["system-node-critical", "system-cluster-critical"]

Configure Persistent Storage

Section titled “Configure Persistent Storage”

Next, create a Google Cloud Persistent Disk to hold the Deepgram model files. Populate the disk from inside the cluster with a one-shot Kubernetes Job, then delete the Job. The disk remains, and will later be mounted read-only into the Deepgram Engine pods when the Helm chart is installed.

  1. Create a Google Persistent Disk to store Deepgram model files and share them across multiple Deepgram Engine pods.

    Bash
    DISK_NAME=deepgram-model-storage
DISK_URI=$(
    gcloud compute disks create \
        $DISK_NAME \
        --size=40GB \
        --type=pd-ssd \
        --zone $ENGINE_NP_ZONE \
        --format="value(selfLink)" | \
    sed -e 's#.*/projects/#projects/#'
)

  2. Create a temporary writable PersistentVolume and PersistentVolumeClaim that point at the disk you just provisioned. The PV uses ReadWriteOnce and Retain so the underlying disk is preserved when you delete the PV later. The nodeAffinity block keeps the downloader pod in the same zone as the zonal Persistent Disk so the disk can attach.

    Bash
    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: dg-model-downloader
spec:
  capacity:
    storage: 40Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  csi:
    driver: pd.csi.storage.gke.io
    volumeHandle: $DISK_URI
    fsType: ext4
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.kubernetes.io/zone
              operator: In
              values:
                - $ENGINE_NP_ZONE
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dg-model-downloader
  namespace: dg-self-hosted
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 40Gi
  storageClassName: ""
  volumeName: dg-model-downloader
EOF

  3. Run a one-shot Job that mounts the writable PVC and downloads the model files provided by your Deepgram Account Representative. The CSI driver creates the ext4 filesystem on the disk the first time the volume is attached.

    Bash
    cat <<'EOF' | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: dg-model-downloader
  namespace: dg-self-hosted
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: downloader
          image: alpine:3
          command: ["sh", "-c"]
          args:
            - |
              set -eu
              apk add --no-cache wget ca-certificates
              cd /mnt/models
              # Replace these with the model URLs from your Deepgram Account Representative.
              wget https://link-to-model-1.dg
              wget https://link-to-model-2.dg
              # ... continue for all model files
          volumeMounts:
            - name: models
              mountPath: /mnt/models
      volumes:
        - name: models
          persistentVolumeClaim:
            claimName: dg-model-downloader
EOF

  4. Wait for the Job to complete successfully.

    Bash
    kubectl wait -n dg-self-hosted --for=condition=complete job/dg-model-downloader --timeout=30m
kubectl logs -n dg-self-hosted job/dg-model-downloader

  5. Delete the Job and the temporary writable PV and PVC. Because the PV uses persistentVolumeReclaimPolicy: Retain, the underlying Google Persistent Disk is preserved with the model files intact.

    Bash
    kubectl delete -n dg-self-hosted job dg-model-downloader
kubectl delete -n dg-self-hosted pvc dg-model-downloader
kubectl delete pv dg-model-downloader

Configure Kubernetes Secrets

Section titled “Configure Kubernetes Secrets”

Deepgram strongly recommends following best practices for configuring Kubernetes Secrets . Please refer to Securing Your Cluster for more details.

The deepgram-self-hosted Helm chart takes two Secret references. One is a set of distribution credentials that allow the cluster to pull images from Deepgram’s container image repository. The other is your self-hosted API key that licenses each Deepgram container that is created.

  1. Complete the Self Service Licensing & Credentials guide to generate distribution credentials and a self-hosted API key.

  2. If using an external Secret store provider, configure cluster access to these two Secrets, naming them dg-regcred (distribution credentials) and dg-self-hosted-api-key.

  3. If not using an external Secret store provider, create the Secrets manually in your cluster.

    1. Using the distribution credentials username and password generated in the Deepgram Console, create a Kubernetes Secret named dg-regcred.

      Bash
      kubectl create secret docker-registry dg-regcred \
    --docker-server=quay.io \
    --docker-username='QUAY_DG_USER' \
    --docker-password='QUAY_DG_PASSWORD'

    2. Create a Kubernetes Secret named dg-self-hosted-api-key to store your self-hosted API key.

      Bash
      kubectl create secret generic dg-self-hosted-api-key \
    --from-literal=DEEPGRAM_API_KEY='YOUR_API_KEY_HERE'

Deploy Deepgram

Section titled “Deploy Deepgram”

Deepgram maintains the official deepgram-self-hosted Helm Chart. You can reference the source and Artifact Hub listing for more details. We’ll use this Chart to facilitate deploying Deepgram services in your self-hosted environment.

  1. Fetch the repository info.

    Bash
    helm repo add deepgram https://deepgram.github.io/self-hosted-resources
helm repo update

  2. Download a values.yaml template for Deepgram’s self-hosted Helm chart from here.

  3. Modify the values.yaml file:

    • Update the scaling.static.{api,engine,licenseProxy}.replicas values to match your node pool sizes.

    • Configure the engine.modelManager.volumes.gcp.gpd values to use the Google Persistent Disk you created earlier.

      Bash
      echo $DISK_URI
      YAML
      engine:
  modelManager:
    volumes:
      gcp:
        gpd:
          enabled: true
          volumeHandle: "<your DISK_URI here>"

  4. Install the Helm Chart with your values.yaml file.

    Bash
    helm install deepgram deepgram/deepgram-self-hosted \
    -f my-values.yaml \
    --namespace dg-self-hosted \
    --create-namespace \
    --atomic \
    --timeout 1h
# Monitor the installation in a separate shell
watch kubectl get all

Test Your Deepgram Setup with a Sample Request

Section titled “Test Your Deepgram Setup with a Sample Request”

Test your environment and container setup with a local file.

  1. Get the name of one of the Deepgram API Pods.

    Bash
    API_POD_NAME=$(
    kubectl get pods \
        --selector app=deepgram-api \
        --output jsonpath='{.items[0].metadata.name}' \
        --no-headers
)

  2. Launch an ephemeral container to send your test request from.

    Bash
    kubectl debug $API_POD_NAME \
    -it \
    --image=curlimages/curl \
    -- /bin/sh

  3. Inside the ephemeral container, download a sample file from Deepgram (or supply your own file).

    Bash
    wget https://dpgr.am/bueller.wav

  4. Send your audio file to your local Deepgram setup for transcription.

    Bash
    curl \
    -X POST \
    --data-binary @bueller.wav \
    "http://localhost:8080/v1/listen?model=nova-3&smart_format=true"

You should receive a JSON response with the transcript and associated metadata. Congratulations - your self-hosted setup is working!

Next Steps

Section titled “Next Steps”

Your Deepgram services are accessible within your cluster via the deepgram-api-external Service that was created by the Helm Chart.

You may consider configuring additional ingress with a GCP Load Balancer to access your services. Note that your installation will automatically load balance any received requests within the cluster to distribute load evenly. The load balancer would primarily serve as the ingress endpoint into the cluster.

What’s Next

Now that you have a basic Deepgram setup working, take some time to learn about building up to a production-level environment, as well as helpful Deepgram add-on services.

Suggest an edit

Propose a replacement for this page. The site team reviews it before applying any changes.

Export
Documentation menu