Skip to main content
Deepgram's Docs

Search documentation

Type to search this documentation.

On this pageOverview

Build a Voice Agent with Pipecat and Deepgram

If you already use Pipecat for voice AI pipelines, you can add Deepgram’s speech-to-text and text-to-speech models to your Pipecat agent. Pipecat’s pipeline architecture connects separate STT, LLM, and TTS services, so this guide pairs Deepgram’s audio models with OpenAI for language understanding — though any Pipecat-compatible LLM works.

For a standalone voice agent without Pipecat or an external LLM, see the Deepgram Voice Agent API, which bundles STT, LLM routing, and TTS in a single WebSocket connection.

For a CLI-scaffolded approach using pipecat init, see the Pipecat and Deepgram integration guide.

Before You Begin

Section titled “Before You Begin”

This guide assumes you are familiar with Python and have a basic understanding of how voice agents work.

Get OpenAI Credentials

Section titled “Get OpenAI Credentials”

This tutorial uses OpenAI for its LLM. You’ll need to sign up for an OpenAI account and obtain an API key.

Requirements

Section titled “Requirements”
  • Python 3.11+

Set Up Your Project

Section titled “Set Up Your Project”

This implementation is a starting reference for building your own voice agent with Pipecat and Deepgram. It is not designed for production deployments.

Create a new directory, set up a virtual environment, and install Pipecat with the Deepgram, OpenAI, and runner extras:

Bash
mkdir deepgram-pipecat-agent
cd deepgram-pipecat-agent
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install "pipecat-ai[deepgram,openai,silero,webrtc]" python-dotenv

The runner extra includes a built-in WebRTC transport with a browser-based test client — no external account needed.

Set Environment Variables

Section titled “Set Environment Variables”

Create a .env file in your project root with the credentials you collected earlier. The agent reads these at startup to authenticate with each service:

DEEPGRAM_API_KEY=your_deepgram_api_key
OPENAI_API_KEY=your_openai_api_key

Build the Agent

Section titled “Build the Agent”

The agent creates a pipeline that connects audio input to Deepgram for transcription, OpenAI for a response, and Deepgram again for speech synthesis. Pipecat handles the audio transport, turn-taking, and interruption detection.

The key components are:

  • Pipeline — connects frame processors in sequence: transport input, STT, context aggregation, LLM, TTS, and transport output.
  • LLMContextAggregatorPair — manages conversation context and uses Silero VAD to detect when the user starts and stops speaking.
  • LLMRunFrame — triggers the LLM to generate a response. Used here to make the agent greet the user on connect.

Create bot.py:

Python
# bot.py

import os

from dotenv import load_dotenv

from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.frames.frames import LLMRunFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
    LLMContextAggregatorPair,
    LLMUserAggregatorParams,
)
from pipecat.runner.types import RunnerArguments
from pipecat.runner.utils import create_transport
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.deepgram.tts import DeepgramTTSService
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.base_transport import BaseTransport, TransportParams

load_dotenv(override=True)

transport_params = {
    "webrtc": lambda: TransportParams(
        audio_in_enabled=True,
        audio_out_enabled=True,
    ),
}

async def run_bot(transport: BaseTransport, runner_args: RunnerArguments):
    stt = DeepgramSTTService(
        api_key=os.getenv("DEEPGRAM_API_KEY"),
        settings=DeepgramSTTService.Settings(
            model="nova-3-general",
            language="en",
            punctuate=True,
            smart_format=True,
        ),
    )

    llm = OpenAILLMService(
        api_key=os.getenv("OPENAI_API_KEY"),
        settings=OpenAILLMService.Settings(
            model="gpt-4o",
            system_instruction=(
                "You are a friendly, helpful voice assistant. "
                "Keep your responses concise — aim for 1-3 sentences "
                "unless the user asks for detail. "
                "Your responses will be spoken aloud, so avoid emojis, "
                "bullet points, or other formatting that can't be spoken."
            ),
        ),
    )

    tts = DeepgramTTSService(
        api_key=os.getenv("DEEPGRAM_API_KEY"),
        settings=DeepgramTTSService.Settings(
            voice="aura-2-thalia-en",
        ),
    )

    context = LLMContext()

    user_aggregator, assistant_aggregator = LLMContextAggregatorPair(
        context,
        user_params=LLMUserAggregatorParams(
            vad_analyzer=SileroVADAnalyzer(),
        ),
    )

    pipeline = Pipeline([
        transport.input(),
        stt,
        user_aggregator,
        llm,
        tts,
        transport.output(),
        assistant_aggregator,
    ])

    task = PipelineTask(
        pipeline,
        params=PipelineParams(
            enable_metrics=True,
            enable_usage_metrics=True,
        ),
    )

    @transport.event_handler("on_client_connected")
    async def on_client_connected(transport, client):
        context.add_message(
            {"role": "developer", "content": "Greet the user and ask how you can help."}
        )
        await task.queue_frames([LLMRunFrame()])

    @transport.event_handler("on_client_disconnected")
    async def on_client_disconnected(transport, client):
        await task.cancel()

    runner = PipelineRunner(handle_sigint=runner_args.handle_sigint)
    await runner.run(task)

async def bot(runner_args: RunnerArguments):
    transport = await create_transport(runner_args, transport_params)
    await run_bot(transport, runner_args)

if __name__ == "__main__":
    from pipecat.runner.run import main

    main()

How the pipeline works

Section titled “How the pipeline works”

Audio flows through the pipeline from left to right:

  1. transport.input() captures microphone audio from the browser.
  2. stt (Deepgram Nova-3) transcribes audio into text in real-time.
  3. user_aggregator collects transcription frames and uses Silero VAD to detect when the user finishes speaking, then adds the complete utterance to the conversation context.
  4. llm (OpenAI GPT-4o) generates a text response based on the conversation context.
  5. tts (Deepgram Aura) converts the response text to speech.
  6. transport.output() sends the audio back to the browser.
  7. assistant_aggregator records the assistant’s response in the conversation context for future turns.

When a user speaks over the agent, Silero VAD detects the interruption and cancels the current TTS output so the agent stops and listens.

Run the Agent

Section titled “Run the Agent”

Start the agent with the WebRTC transport. The -t webrtc flag launches a built-in browser client for testing:

Bash
python bot.py -t webrtc

Test the Agent

Section titled “Test the Agent”

Once the agent is running, open http://localhost:7860/client in your browser.

  1. Allow microphone access when prompted
  2. Click to connect — the agent greets you automatically
  3. Start talking — the agent responds in real time

Try interrupting the agent mid-sentence. Silero VAD detects your speech and cancels the current response so the agent listens to you instead.

Further Reading

Section titled “Further Reading”
Suggest an edit

Propose a replacement for this page. The site team reviews it before applying any changes.

Export
Documentation menu