Voice Agent Message Flow

This guide walks you through implementing the correct message flow when building a Voice Agent client. Follow these steps to establish a connection, configure settings, and handle the conversation loop.

1. Open a WebSocket connection to the Voice Agent endpoint.

2. Wait for the server to send a Welcome message confirming the connection:

{ "type": "Welcome", "request_id": "uuid" }

3. Send a Settings message with your audio and agent configuration:

{
  "type": "Settings",
  "audio": {
    "input": {
      "encoding": "linear16",
      "sample_rate": 16000
    },
    "output": {
      "encoding": "linear16",
      "sample_rate": 24000,
      "container": "none"
    }
  },
  "agent": {
    "listen": { "provider": { "type": "deepgram", "model": "nova-3" } },
    "think": {
      "provider": { "type": "open_ai", "model": "gpt-4o-mini" }
    },
    "speak": { "provider": { "type": "deepgram", "version": "v2", "model": "flux-kit-en" } }
  }
}

4. Wait for the server to send a SettingsApplied message:

{ "type": "SettingsApplied" }

5. After receiving SettingsApplied, begin streaming binary audio data (PCM) continuously to the server.

6. Optionally, send text input using InjectUserMessage:

{ "type": "InjectUserMessage", "content": "Hello" }

7. Process the following events as the conversation progresses:

sequenceDiagram
    participant CLIENT
    participant SERVER (Deepgram)

    CLIENT->>SERVER (Deepgram): Connect
    SERVER (Deepgram)-->>CLIENT: Welcome
    CLIENT->>SERVER (Deepgram): Settings
    SERVER (Deepgram)-->>CLIENT: SettingsApplied

    loop Conversation Loop
        CLIENT->>SERVER (Deepgram): Binary Audio
        Note over CLIENT,SERVER (Deepgram): (or InjectUserMessage)
        SERVER (Deepgram)-->>CLIENT: UserStartedSpeaking
        SERVER (Deepgram)-->>CLIENT: ConversationText
        SERVER (Deepgram)-->>CLIENT: AgentThinking
        SERVER (Deepgram)-->>CLIENT: ConversationText
        SERVER (Deepgram)-->>CLIENT: [binary audio]
        SERVER (Deepgram)-->>CLIENT: AgentAudioDone
    end

    CLIENT->>SERVER (Deepgram): Close

Confirm your implementation works correctly by checking:

• You receive a Welcome message immediately after connecting.
• You receive a SettingsApplied message after sending your Settings.
• The agent responds with ConversationText and binary audio when you speak or inject text.
• Audio playback stops when you receive UserStartedSpeaking (barge-in detection).

• Configure the Voice Agent for detailed settings options.
• Outputs: Server Events for detailed event documentation.