5 Lessons from Building a Safe Medical Voice AI Agent

Picture this scenario:

A patient calls a pharmacy line and says: "Hi, I need to refill my mom's Augmentin, the 875."

A voice agent takes care of this request. It listens, pulls out the structured details, asks one clarifying question, accepts the request, and sends the medication out.

That setup is both technically impressive and incredibly dangerous. Automation is one of the big promises we're getting from the agentic era of AI. However, blindly implementing a use case without questioning its implications can pose great dangers.

That's why we built a demo to see what it actually takes to run a voice agent in a domain where being wrong has consequences.

Let's break it down.

What's at Stake?

A few examples where the exact path we laid out can go wrong:

Auto-accepting prescription requests can lead to incorrect deliveries that might have serious health implications for patients.
A fully-automated process can miss important details and context with medication.
Small transcription errors can lead to major problems if the wrong medication is prescribed and delivered for a specific treatment.

And, these errors are increasingly more likely with modern technology. The sheer wiring up of a voice agent is now a straightforward task. You...

capture audio,
run speech-to-text,
send the transcript to an LLM,
speak the response back.

This is where most tutorials stop.

As with many other things, the devil lies in the details. The steps laid out are not wrong, but without further thinking and guardrails, this is not enough. That's why we're laying out five decisions that allowed us to build a trustworthy agent.

(Note: We're applying these lessons to the medical space, but they're also applicable to any voice agent where mistakes have serious implications.)

The Stack

Stream's Vision Agents SDK to handle the real-time call and agent orchestration
AssemblyAI for speech-to-text
LLM (Gemini 3.5 Flash, in this build) for the reasoning
TTS model from Inworld to give our agent a voice

Lesson 1: Design the Flow Around the Stakes

The version that looks great in a demo is where the agent closes the loop itself: caller asks, agent confirms, "Great, your refill is approved and on the way."

It's satisfying to watch. It's also the wrong design for anything with real consequences.

We made the agent an intake assistant, not an approver. Its job starts when the call connects and ends at "prepared for pharmacist review." A human is always in the loop before anything happens. The agent's own instructions spell out what it's not allowed to do:

text

1
2
3
4
You are not a doctor, pharmacist, or prescriber.
You must not provide medical advice.
You must not approve, prescribe, dispense, recommend, or modify medication.
You must not say that an order is complete or approved.

But instructions alone are a soft boundary, and anyone with a bit of experience can argue an LLM out of them. So the boundary needs to also be enforced structurally. The agent cannot declare the intake finished on its own by design. In our case that means that it has to call a prepare_pharmacist_handoff tool, and that tool refuses to return a "ready" state until every required field is present:

1
2
3
4
REQUIRED_FIELDS = (
    "patient_name", "request_type", "medication", "strength",
    "instructions", "quantity", "prescriber", "delivery_preference",
)

The frontend has a "Send to pharmacist" button which is only active when the exact same conditions are met, so the UI and the agent agree on what "done" means.

The general takeaway here is that for any high-consequence workflow, the first design question is not: what can the agent do? Instead, it's: where does the agent stop?

Define the handoff boundary before you write the prompt, and encode it somewhere deterministic that imposes a hard limit that the model can't bypass.

Lesson 2: Pick the Right Tool for the Job

Speech-to-text providers look interchangeable until you point them at your actual vocabulary. On a generic benchmark, they cluster together. Then your agent hears "atorvastatin" as "a tour of a statin". That leads to an issue where everything that follows (extraction, validation, the handoff packet) is built on a wrong transcript. In this domain, garbage in - garbage out means a major safety issue.

The number that mattered to us was the word error rate on the words that carry the risk: drug names, strengths, and units. That's exactly where general-purpose models tend to slip, because those terms are rare in ordinary training data and easy to confuse with common words.

We used AssemblyAI because it held up well on dense pharmaceutical terminology, which is where it separated from more general STT for this use case. Because the Vision Agents SDK abstracts the call transport, the STT provider is a pluggable component.

This allowed us to evaluate and change different options with a one-liner:

1
2
3
4
5
stt = assemblyai.STT(
    api_key=settings.assemblyai_api_key,
    speech_model="universal-3-5-pro",
    keyterms_prompt=ASSEMBLYAI_KEYTERMS,  # more on this in Lesson 3
)

The broader lesson here is that the metric for choosing the best model should not be an arbitrary leaderboard on the web.

Instead, we should first declare "for what" we're using the model and evaluate its quality only on that specific use case. We're not building a product that needs to work in any setting, so we shouldn't choose a model that tries to be applicable for all scenarios.

Ready to integrate? Our team is standing by to help you. Contact us today and launch tomorrow!

Lesson 3: Give the Agent a World Model

With the right selection of an STT provider and a capable LLM, they still don't know our domain out of the box. The recognizer doesn't know "amox clav" is a thing a caller might say; the model doesn't know which strengths of levothyroxine are plausible. We have to supply that knowledge, and it turned out to matter at two different layers.

At the ear (recognition). AssemblyAI accepts a keyterms_prompt, which is a list of terms to bias recognition toward. We fill it with the vocabulary this agent actually encounters: medication names, brand/generic pairs, the prescriber's name, even the agent's own name. Here is a simplified list of key terms we feed to the model:

1
2
3
4
ASSEMBLYAI_KEYTERMS = [
    "atorvastatin", "Lipitor", "amoxicillin clavulanate", "amoxicillin-clavulanate", "Augmentin",
    "levothyroxine", "Synthroid", "pharmacist review",
]

This is where Lesson 2 and Lesson 3 meet: picking a strong recognizer is step one, but you get a further accuracy jump by telling it which uncommon words to expect. We can already see here that building a product means connecting multiple approaches.

At the brain (reasoning). The agent has a medication reference that it can use to look things up in a database. These can be canonical names, aliases, common strengths, and a per-drug review flag. Here is an example for a medication:

json

1
2
3
4
5
6
7
{
  "name": "Amoxicillin-clavulanate",
  "aliases": ["amoxicillin clavulanate", "Augmentin", "amox clav"],
  "commonStrengths": ["500 mg / 125 mg", "875 mg / 125 mg"],
  "type": "antibiotic",
  "requiresPharmacistReview": true
}

So when a caller says "amox clav," the agent normalizes it to the canonical "Amoxicillin-clavulanate" rather than guessing, and it knows "the 875" maps to a real strength.

We can summarize that an LLM is a reasoning engine, not a knowledge base. Don't assume it memorized your catalog, your SKUs, or your entity names. It's our job to give it a source of truth to consult. This will then enrich both the layer that hears and the layer that thinks.

Lesson 4: Don't Make the LLM Do Everything

It's tempting to hand the LLM the whole job, including extraction, validation, flow control, and tone. A cleaner split is to let the model do the fuzzy work and let plain code do the mechanical work.

The deterministic side handles anything that's verifiable and boring. Normalizing units and frequencies, for example. This can be a simple lookup and does not need to be a judgment call:

1
2
3
4
5
6
7
8
9
STRENGTH_ALIASES = {
    "20 milligrams": "20 mg",
    "875 125": "875 mg / 125 mg",
}
FREQUENCY_ALIASES = {
    "every night": "Once nightly",
    "twice a day": "Twice daily",
    "every 12 hours": "Every 12 hours",
}

Completeness is enforced in code, too. Instead of trusting the model to remember every field it still needs, a function checks the intake state against the required list and reports what's missing.

This is the exact condition that gates the handoff we described in Lesson 1:

1
2
def missing_required_fields(state) -> list[str]:
    return [f for f in REQUIRED_FIELDS if not getattr(state, f, None)]

Meanwhile, the LLM does what it's genuinely good at: interpreting what the caller meant, deciding which clarifying question is worth asking, and keeping replies short and warm. Its instructions push it toward that role through formulations like this:

Ask at most one clarification question at a time
Keep spoken replies short, usually one sentence

The principle is a division of labor by reliability.

If a rule can be checked with a for loop, check it with a for loop; you don't want correctness riding on a probabilistic system when a deterministic one is exact. Save the model for the calls that actually require judgment.

Lesson 5: Guardrails Are Not Simply a Line in the Prompt

"Be safe and don't give medical advice" in a system prompt is both necessary and, at the same time, nowhere near enough.

Prompts can be worked around. On the other hand, if we add deterministic logic checks to each phase of the flow, we can assure that our safeguards can't be breached.

Our demo has three cooperating layers.

A deterministic safety net. Before the LLM's judgment enters into it, transcript text is scanned for high-risk language. Emergency terms trigger a hard stop; dosage-change requests and medical-advice questions get forced into pharmacist review regardless of what the model would have done.

1
2
3
4
5
6
7
EMERGENCY_TERMS = ["chest pain", "can't breathe", "overdose",
                   "allergic reaction", "suicidal", ...]

def run_safety_checks_on_text(text):
    if _contains_any(text, EMERGENCY_TERMS):
        return SafetyResult(status="blocked", blocked=True,
                            response=EMERGENCY_RESPONSE)

If a caller says "chest pain," the agent breaks out of the intake flow and points them to urgent care. That behavior does not depend on the LLM deciding to be careful.

Instruction guardrails. The behavioral rules from Lesson 1 (never approve, never diagnose) always frame the outcome as "prepared for pharmacist review." These shape the model's default behavior.
Structural gates. The prepare_pharmacist_handoff tool makes the unsafe end-state ("the agent said it's done") unreachable until the required data exists. There's also a standing "pharmacist review required" check applied to every request, plus extra checks when a caregiver is calling on someone's behalf or a second medication comes up.

The idea is defense in depth. Assume any single layer can fail, and arrange things so the next layer catches it. For any agent that touches health, money, legal, or identity, the guardrails need to hold even when the model behaves in a way you didn't anticipate.

Conclusion

We see one theme across all five lessons. A trustworthy voice agent is a systems problem, not a prompt problem (alone).

The LLM is one component among several. Most of the value is in how you surround it. That means the right transport, an STT model that holds up on your vocabulary, real domain knowledge fed to both the recognizer and the reasoner, deterministic checks for anything verifiable, and guardrails that reliably catch when the model is misbehaving.

This tooling lets us spend our effort in the right place. Stream's Vision Agents SDK made the real-time call, orchestration, and provider swapping cheap, so the work went into the domain decisions above rather than plumbing. AssemblyAI made the speech layer accurate enough on medical terms to build the rest on top of.

The end result is a demo that deliberately is a limited one.

No real fulfillment, no medical advice, a human pharmacist at the end of every request.

But the pattern generalizes well beyond pharmacy, with real-time voice, a high-stakes domain, and a human kept firmly in the loop. If you're building in that shape, the five lessons here are a reasonable place to start.