Entity Detection
detect_entities boolean. Default: false
Pre-recorded Streaming:Nova English (all available regions)
Entity Detection
Section titled “Entity Detection”When Entity Detection is enabled, the Punctuation feature will be enabled by default.
Model Support
Section titled “Model Support”Entity Detection is available for both pre-recorded and streaming speech-to-text.
Enable Feature
Section titled “Enable Feature”To enable Entity Detection, when you call Deepgram's API, add a detect_entities parameter set to true in the query string:
detect_entities=true
Pre-recorded Audio
Section titled “Pre-recorded Audio”To transcribe audio from a file on your computer, run the following curl command in a terminal or your favorite API client.
cURL
curl \
--request POST \
--header 'Authorization: Token YOUR_DEEPGRAM_API_KEY' \
--header 'Content-Type: audio/wav' \
--data-binary @youraudio.wav \
--url 'https://api.deepgram.com/v1/listen?detect_entities=true'Streaming Audio
Section titled “Streaming Audio”To enable Entity Detection for streaming audio, establish a WebSocket connection with the detect_entities=true parameter. Remember that streaming Entity Detection is supported on Nova, Nova-2, Nova-3, and Enhanced models.
// Example filename: index.js
const { DeepgramClient } = require("@deepgram/sdk");
const fetch = require("cross-fetch");
const dotenv = require("dotenv");
dotenv.config();
// URL for the realtime streaming audio you would like to transcribe
const url = "http://stream.live.vc.bbcmedia.co.uk/bbc_world_service";
const live = async () => {
// STEP 1: Create a Deepgram client using the API key
const deepgram = new DeepgramClient({ apiKey: process.env.DEEPGRAM_API_KEY });
// STEP 2: Create a live transcription connection with entity detection enabled
const connection = await deepgram.listen.v1.connect({
model: "nova-3",
language: "en-US",
smart_format: "true",
detect_entities: "true",
});
// STEP 3: Listen for events from the live transcription connection
connection.on("open", () => {
console.log("Connection opened.");
connection.on("close", () => {
console.log("Connection closed.");
});
connection.on("message", (data) => {
if (data.type === "Results") {
const transcript = data.channel.alternatives[0].transcript;
// Only process final results which contain entities
if (data.is_final) {
const entities = data.channel.alternatives[0].entities;
if (entities && entities.length > 0) {
console.log("\nTranscript:", transcript);
console.log("Entities detected:");
entities.forEach(entity => {
console.log(` - ${entity.label}: ${entity.value} (confidence: ${entity.confidence})`);
// raw_value is present when formatting features (like smart_format) are enabled
if (entity.raw_value) {
console.log(` Raw value: ${entity.raw_value}`);
}
});
}
}
}
});
connection.on("error", (err) => {
console.error("Error:", err);
});
// STEP 4: Fetch the audio stream and send it to the live transcription connection
fetch(url)
.then((r) => r.body)
.then((res) => {
res.on("readable", () => {
connection.sendMedia(res.read());
});
});
});
connection.connect();
await connection.waitForOpen();
};
live();# Example filename: main.py
import httpx
import threading
from deepgram import DeepgramClient
from deepgram.core.events import EventType
from deepgram.listen.v1.types import ListenV1Results
# URL for the realtime streaming audio you would like to transcribe
URL = "http://stream.live.vc.bbcmedia.co.uk/bbc_world_service"
def main():
try:
# Create a Deepgram client
deepgram = DeepgramClient()
# Create a websocket connection with entity detection enabled
with deepgram.listen.v1.connect(
model="nova-3",
language="en-US",
smart_format=True,
detect_entities=True
) as connection:
def on_message(message) -> None:
# Only process final results which contain entities
if hasattr(message, 'is_final') and message.is_final:
if hasattr(message, 'channel') and hasattr(message.channel, 'alternatives'):
transcript = message.channel.alternatives[0].transcript
entities = message.channel.alternatives[0].entities
if entities and len(entities) > 0:
print(f"\nTranscript: {transcript}")
print("Entities detected:")
for entity in entities:
print(f" - {entity.label}: {entity.value} (confidence: {entity.confidence})")
# raw_value is present when formatting features (like smart_format) are enabled
if hasattr(entity, 'raw_value') and entity.raw_value:
print(f" Raw value: {entity.raw_value}")
connection.on(EventType.OPEN, lambda _: print("Connection opened"))
connection.on(EventType.MESSAGE, on_message)
connection.on(EventType.CLOSE, lambda _: print("Connection closed"))
connection.on(EventType.ERROR, lambda error: print(f"Error: {error}"))
lock_exit = threading.Lock()
exit = False
# Define a thread for start_listening with error handling
def listening_thread():
try:
connection.start_listening()
except Exception as e:
print(f"Error in listening thread: {e}")
# Start listening in a separate thread
listen_thread = threading.Thread(target=listening_thread)
listen_thread.start()
# Define a worker thread for HTTP streaming with error handling
def http_stream_thread():
try:
with httpx.stream("GET", URL) as r:
for data in r.iter_bytes():
lock_exit.acquire()
if exit:
break
lock_exit.release()
connection.send_media(data)
except Exception as e:
print(f"Error in HTTP streaming thread: {e}")
# Start the HTTP streaming thread
stream_thread = threading.Thread(target=http_stream_thread)
stream_thread.start()
# Wait for user input to stop
input("Press Enter to stop...\n")
lock_exit.acquire()
exit = True
lock_exit.release()
# Wait for both threads to close
stream_thread.join(timeout=5.0)
listen_thread.join(timeout=5.0)
print("Finished")
except Exception as e:
print(f"Could not open socket: {e}")
return
if __name__ == "__main__":
main()Analyze Response
Section titled “Analyze Response”The response structure differs between pre-recorded and streaming transcription.
Pre-recorded Response
Section titled “Pre-recorded Response”When the file is finished processing (often after only a few seconds), you'll receive a JSON response that has the following basic structure:
JSON
{
"metadata": {
"transaction_key": "string",
"request_id": "string",
"sha256": "string",
"created": "string",
"duration": 0,
"channels": 0
},
"results": {
"channels": [
{
"alternatives":[],
}
]
}
}Let's look more closely at the alternatives object:
JSON
"alternatives":[
{
"transcript":"Welcome to the Ai show. I'm Scott Stephenson, cofounder of Deepgram...",
"confidence":0.9816771,
"words": [...],
"entities":[
{
"label":"NAME",
"value":" Scott Stephenson",
"raw_value": "scott stephenson",
"confidence":0.9999924,
"start_word":6,
"end_word":8
},
{
"label":"ORGANIZATION",
"value":" Deepgram",
"raw_value": "deepgram",
"confidence":0.9999757,
"start_word":10,
"end_word":11
},
{
"label": "CARDINAL",
"value": "one",
"raw_value": "one",
"confidence": 1,
"start_word": 186,
"end_word": 187
},
...
]
}
]Streaming Response
Section titled “Streaming Response”For streaming transcription, entities are included in final results only (when is_final: true). Interim results do not contain the entities array.
Here's an example of a streaming response with Entity Detection enabled:
JSON - Final Result with Entities
{
"type": "Results",
"channel_index": [0, 1],
"duration": 4.64,
"start": 0.0,
"is_final": true,
"speech_final": true,
"channel": {
"alternatives": [
{
"transcript": "Hi, I'm calling to update my account. My name is Jane Doe and my phone number is (555) 123-4567. You can reach me at jane.doe@email.com.",
"confidence": 0.99,
"words": [...],
"entities": [
{
"label": "NAME",
"value": "Jane Doe",
"raw_value": "jane doe",
"confidence": 0.9999,
"start_word": 9,
"end_word": 11
},
{
"label": "PHONE_NUMBER",
"value": "(555) 123-4567",
"raw_value": "five five five one two three four five six seven",
"confidence": 0.9998,
"start_word": 15,
"end_word": 16
},
{
"label": "EMAIL_ADDRESS",
"value": "jane.doe@email.com",
"raw_value": "jane dot doe at email dot com",
"confidence": 0.9999,
"start_word": 21,
"end_word": 22
}
]
}
]
}
}Streaming Finalization Behavior
Section titled “Streaming Finalization Behavior”When using Entity Detection with streaming audio, Deepgram will attempt to detect and format entities as they are spoken. For entities that seem like they may be incomplete, our system will:
- Wait until the speaker continues to non-entity speech, OR
- Finalize the transcript after 3 seconds of silence, OR
- Receive a Finalize control message
- Return only completed entities based on the available audio at that point
This approach ensures transcripts are returned promptly while maintaining entity detection precision.
Using No Delay
Section titled “Using No Delay”Setting no_delay=true forces immediate finalization of streaming transcripts without waiting for entity completion.
To use no_delay with Entity Detection:
const connection = await deepgram.listen.v1.connect({
model: "nova-3",
language: "en-US",
detect_entities: "true",
no_delay: "true" // Forces immediate finalization, may miss entities
});with deepgram.listen.v1.connect(
model="nova-3",
language="en-US",
detect_entities=True,
no_delay=True # Forces immediate finalization, may miss entities
) as connection:Understanding Entity Fields
Section titled “Understanding Entity Fields”Each entity object in the entities array contains the following fields:
label: Type of entity identified (e.g., NAME, PHONE_NUMBER, EMAIL, ADDRESS).value: The formatted text of the entity. When Smart Formatting is enabled, this field reflects the formatted output.raw_value: (When formatting is enabled) The original, non-formatted text as spoken. This field is included in both pre-recorded and streaming responses when formatting features (such as Smart Formatting) are enabled.confidence: Floating point value between 0 and 1 that indicates overall transcript reliability. Larger values indicate higher confidence.start_word: Index of the first word, inclusive, of the entity in the transcript.end_word: Index of the last word, exclusive, of the entity in the transcript.
Key Differences Between Pre-recorded and Streaming:
| Field | Pre-recorded | Streaming |
|---|---|---|
value |
Always included | Always included |
raw_value |
Included when formatting is enabled | Included when formatting is enabled |
| Availability | Always | Only in is_final: true messages |
Identifiable Entities
Section titled “Identifiable Entities”View all options here: Supported Entity Types
Use Cases
Section titled “Use Cases”Some examples of uses for Entity Detection include:
- Customers who want to improve Conversational AI and Voice Assistant by triggering particular workflows and responses based on identified name, address, location, and other key entities.
- Customers who want to enhance customer service and user experience by extracting meaningful and relevant information about key entities such as a person, organization, email, and phone number.
- Customers who want to derive meaningful and actionable insights from the audio data based on identified entities in conversations.