Grok TTS + Vision: Build a Healthcare Appointment Agent

1
2
3
4
5
6
7
8
9
10
# Initialize a Python Project
uv init

# Activate your environment
uv venv
source .venv/bin/activate

# Install Vision Agents
uv add vision-agents
uv add "vision-agents[getstream, gemini, deepgram]"

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Use this codebase to create a custom Python text-to-speech (TTS) plugin for Grok TTS (Voice) to connect with Vision Agents so that it can be used with any AI provider. Aside from adding a basic example, include an example for a medical front-desk receptionist using Sal's voice.

Steps
Follow the Vision Agents Python plugin creation docs to do the implementation and generate all the required plugin directories and files: https://visionagents.ai/integrations/create-your-own-plugin 

Grok Voice TTS docs: https://x.ai/api/voice#text-to-speech

Grok Voice: https://x.ai/api/voice

Text to speech: https://docs.x.ai/developers/model-capabilities/audio/text-to-speech

Example Vision Agents TTS plugins for reference: 

https://github.com/GetStream/Vision-Agents/tree/main/plugins/pocket

https://github.com/GetStream/Vision-Agents/tree/main/plugins/fish

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
plugins/grok_tts/
├── pyproject.toml                          # Package config (hatchling build, aiohttp dep)
├── README.md                               # Full plugin documentation
├── py.typed                                # PEP 561 type marker
├── vision_agents/
│   └── plugins/
│       └── grok_tts/
│           ├── __init__.py                 # Exports TTS, Voice, VOICE_DESCRIPTIONS
│           └── tts.py                      # Core TTS implementation
├── tests/
│   └── test_tts.py                         # Unit tests
└── example/
    ├── pyproject.toml                      # Example dependencies
    ├── README.md                           # Example docs with run instructions
    ├── basic_example.py                    # Basic assistant (Eve voice)
    ├── medical_receptionist_example.py     # Medical receptionist (Sal voice)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
"""
Grok TTS — Basic Example

A minimal Vision Agents setup that demonstrates Grok text-to-speech
with Deepgram STT, Gemini LLM, and Stream's real-time edge transport.

Requirements (environment variables):
    XAI_API_KEY          — xAI / Grok API key
    DEEPGRAM_API_KEY     — Deepgram STT key
    GOOGLE_API_KEY       — Google Gemini key
    STREAM_API_KEY       — Stream API key
    STREAM_API_SECRET    — Stream API secret
"""

import asyncio
import logging

from dotenv import load_dotenv
from vision_agents.core import Agent, Runner, User
from vision_agents.core.agents import AgentLauncher
from vision_agents.plugins import deepgram, gemini, getstream, smart_turn
from vision_agents.plugins import grok_tts

logger = logging.getLogger(__name__)

load_dotenv()

async def create_agent(**kwargs) -> Agent:
    """Create an agent with Grok TTS using the default 'eve' voice."""
    agent = Agent(
        edge=getstream.Edge(),
        agent_user=User(name="Grok Voice AI", id="agent"),
        instructions=(
            "You are a friendly and helpful voice assistant powered by Grok. "
            "Keep your responses concise and conversational."
        ),
        tts=grok_tts.TTS(voice="eve"),
        stt=deepgram.STT(eager_turn_detection=True),
        llm=gemini.LLM(),
        turn_detection=smart_turn.TurnDetection(
            silence_duration_ms=2000,
            speech_probability_threshold=0.5,
        ),
    )
    return agent

async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    """Join a call and greet the user."""
    call = await agent.create_call(call_type, call_id)

    logger.info("Starting Grok TTS Agent (basic example)...")

    async with agent.join(call):
        logger.info("Agent joined call")

        await asyncio.sleep(3)
        await agent.llm.simple_response(
            text="Hello! I'm your voice assistant running on Grok TTS. How can I help?"
        )

        await agent.finish()

if __name__ == "__main__":
    Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
"""
Grok TTS — Medical Receptionist Example

A voice/vision agent that acts as a professional medical office receptionist.
Uses the 'sal' voice (smooth, balanced) for a calm and reassuring tone. Before assisting patients to schedule appointments, the agent sees through the patient's camera feed to assess and check if there is the need to visit a doctor or give an online medical treatment/advice.

Requirements (environment variables):
    XAI_API_KEY          — xAI / Grok API key
    DEEPGRAM_API_KEY     — Deepgram STT key
    GOOGLE_API_KEY       — Google Gemini key
    STREAM_API_KEY       — Stream API key
    STREAM_API_SECRET    — Stream API secret
"""

import asyncio
import logging

from dotenv import load_dotenv
from vision_agents.core import Agent, Runner, User
from vision_agents.core.agents import AgentLauncher
from vision_agents.plugins import deepgram, gemini, getstream, smart_turn
from vision_agents.plugins import grok_tts

logger = logging.getLogger(__name__)

load_dotenv()

MEDICAL_RECEPTIONIST_INSTRUCTIONS = """\
You are Sal, the front-desk receptionist at "Greenfield Family Practice."

Your personality:
- Professional, patient, and empathetic
- Calm and reassuring, especially with anxious callers
- Clear and precise when relaying medical office information

Your responsibilities:
- Answer incoming calls and greet patients by name when possible
- Check if the patient needs to visit a doctor or give an online medical treatment/advice by seeing through the patient's camera feed. For minor sicknesses, you can give an online medical treatment/advice. For major sicknesses, you should refer the patient to a doctor by scheduling an appointment.
- Also, check out the patients’ uploaded documents/images and screensharing content to make a decision on whether to visit a doctor in-person or not. 
- Schedule, reschedule, or cancel appointments
- Provide office hours, location, and directions
- Explain what to bring to a first visit (insurance card, ID, medication list)
- Triage urgency: direct emergencies to 911, urgent concerns to the nurse line
- Handle prescription refill requests by taking details and forwarding to the provider

Important guidelines:
- NEVER provide medical advice, diagnoses, or treatment recommendations
- Always confirm the patient's date of birth for identity verification
- If a caller describes symptoms that sound urgent, calmly recommend they
  call 911 or go to the nearest emergency room
- Keep responses empathetic but efficient — patients value their time

Office details you may reference:
- Hours: Mon–Fri 8 AM – 5 PM, Sat 9 AM – 12 PM, closed Sunday
- Address: 240 Greenfield Avenue, Suite 100
- Providers: Dr. Sarah Chen (Family Medicine), Dr. James Okafor (Internal Medicine)
- New patient appointments: 45 minutes; follow-ups: 20 minutes
"""

async def create_agent(**kwargs) -> Agent:
    """Create a medical receptionist agent with Grok TTS (sal voice)."""
    agent = Agent(
        edge=getstream.Edge(),
        agent_user=User(name="Maya - Greenfield Family Practice", id="agent"),
        instructions=MEDICAL_RECEPTIONIST_INSTRUCTIONS,
        tts=grok_tts.TTS(voice="sal"),
        stt=deepgram.STT(eager_turn_detection=True),
        llm=gemini.LLM(),
        turn_detection=smart_turn.TurnDetection(
            silence_duration_ms=2500,
            speech_probability_threshold=0.5,
        ),
    )
    return agent

async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    """Join the call and greet the patient caller."""
    call = await agent.create_call(call_type, call_id)

    logger.info("Starting Medical Receptionist Agent...")

    async with agent.join(call):
        logger.info("Agent joined call")

        await asyncio.sleep(3)
        await agent.llm.simple_response(
            text=(
                "Thank you for calling Greenfield Family Practice. "
                "This is Maya. How can I assist you today — "
                "would you like to schedule an appointment or do you have a question about your visit?"
            )
        )

        await agent.finish()

if __name__ == "__main__":
    Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# 1. Clone the repo
git clone https://github.com/GetStream/stream-tutorial-projects.git

# 2. Install dependencies:
cd AI/VisionAgents/VisionAgentsPythonPlugins/GrokTTS/plugins/grok_tts/example
uv sync

# 3. Create a `.env` file with your API keys:
# Required for Grok TTS
XAI_API_KEY=your_xai_api_key

# Required for speech-to-text
DEEPGRAM_API_KEY=your_deepgram_api_key

# Required for LLM
GOOGLE_API_KEY=your_google_api_key

# Required for real-time transport
STREAM_API_KEY=your_stream_api_key
STREAM_API_SECRET=your_stream_api_secret
EXAMPLE_BASE_URL=https://demo.visionagents.ai

# 4. Run the Medical receptionist
uv run medical_receptionist_example.py run

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from vision_agents.plugins import grok_tts

# Default voice (eve) — energetic, upbeat
tts = grok_tts.TTS()

# Specify a voice
tts = grok_tts.TTS(voice="ara")   # warm, friendly
tts = grok_tts.TTS(voice="leo")   # authoritative, strong
tts = grok_tts.TTS(voice="rex")   # confident, clear
tts = grok_tts.TTS(voice="sal")   # smooth, balanced

# Custom output format
tts = grok_tts.TTS(
    voice="rex",
    codec="mp3",
    sample_rate=44100,
    bit_rate=192000,
)

# Explicit API key (otherwise reads XAI_API_KEY env var)
tts = grok_tts.TTS(api_key="xai-your-key-here")