Build a Voice Agent That Calls to Confirm Fraud Alerts

This tutorial builds a voice agent that places an outbound call to a cardholder, reads back a suspicious transaction, and either clears it or freezes the card, all in Python with Vision Agents.

You'll use speech-to-text from Deepgram, an LLM for the conversation, Cartesia for the voice, and Twilio to place the call. The build is for a fictional issuer, Meridian Bank.

A real bank confirms a transaction and never asks the customer to prove themselves with sensitive data. A scammer asks for exactly that. So the rules that keep this bot legitimate are the same rules that keep it from sounding like a scam, and we enforce them in the agent's instructions.

Let's build it.

What We're Building

Meridian Bank's fraud system flags a charge and places an outbound call to the cardholder.

The agent:

1. Identifies itself as Meridian Bank's fraud line and states why it is calling.
2. Reads the flagged transaction back: merchant, amount, location.
3. Asks the customer to confirm or deny it.
4. If confirmed, it clears the hold so the purchase completes.
5. If denied, it freezes the card, opens a dispute, and hands off to a human specialist.

It runs over Stream's edge network for low latency, with Twilio bridging the phone call.

The trust boundary from the opening shows up concretely in steps one through three: 1) the agent identifies itself, 2) reads a transaction back, and 3) takes a yes or no.

It never turns the call into a request for the customer's card number, PIN, password, codes, or money. We'll enforce that in the instructions.

Why Vision Agents Fits This

A fraud callbot is a plumbing problem more than a model problem.

You need to place a phone call, understand speech coming through a phone codec (mulaw at 8kHz, with background noise and interruptions), respond fast enough that it feels like a conversation and not an interactive voice menu, and take real actions like freezing a card.

The interesting part is the fraud logic. Everything between the phone line and the model is undifferentiated work.

Vision Agents collapses that work into one Agent object with swappable parts. Speech-to-text, the LLM, and text-to-speech are constructor arguments you can change in a single line. Turn detection is built into Deepgram, so the agent knows when the caller stopped talking. Actions are ordinary async Python functions you register with a decorator. The phone call bridges in through the Twilio plugin without changing the agent itself.

The customer can also join from anywhere, not just a phone.

The agent runs on a Stream call, and Stream ships frontend SDKs for React, iOS, Android, Flutter, React Native, JavaScript, and Unity. So the same fraud agent can take a Twilio phone call, answer from a "verify this charge" button inside your mobile banking app, or run in a browser.

You build the agent once and let the customer reach it however they already talk to you.

Setup

Vision Agents uses uv and targets Python 3.13 for the examples. Install the framework with the plugins this build needs:

1
uv add "vision-agents[getstream, deepgram, cartesia, openai, twilio]"

Get a free Stream API key from the Stream dashboard. The free tier includes 333,000 participant minutes per month, which is more than enough to build and test this.

Set your keys in a .env file:

1
2
3
4
5
6
7
8
STREAM_API_KEY=...
STREAM_API_SECRET=...
DEEPGRAM_API_KEY=...
CARTESIA_API_KEY=...
OPENAI_API_KEY=...
TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
NGROK_URL=...

The NGROK_URL is the public hostname Twilio will use to reach your machine during development. More on that in the phone section.

The Core Agent

Start with the agent itself, no phone yet. This is the part you can run in a browser to hear it work before wiring up telephony.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from vision_agents.core import Agent, User
from vision_agents.plugins import getstream, deepgram, cartesia, openai


async def create_agent() -> Agent:
    llm = openai.LLM(model="gpt-5.5")

    agent = Agent(
        edge=getstream.Edge(),
        agent_user=User(id="agent", name="Meridian Fraud Line"),
        instructions=FRAUD_INSTRUCTIONS,
        stt=deepgram.STT(model="nova-3", eager_turn_detection=True),
        llm=llm,
        tts=cartesia.TTS(model_id="sonic-3.5"),
    )
    return agent

A few things to note, all of which matter in practice:

llm = openai.LLM(model="gpt-5.5") is the conversation. This is a one-line swap: any current OpenAI model works, and you can drop in Gemini, Claude, or an OpenAI-compatible endpoint by changing this line alone. gpt-5.5 is OpenAI's current flagship at the time of writing.

stt=deepgram.STT(...) sets up speech-to-text. eager_turn_detection=True lowers latency by letting the model decide sooner that the caller has finished talking. It costs a few more tokens but the call feels more natural, which matters when someone is already on edge about a possible fraud charge. Pass model= explicitly. The plugin's default has changed between releases, so pinning the model keeps your behavior stable.

tts=cartesia.TTS(...) sets up the voice. Pin model_id for the same reason.

edge=getstream.Edge() is the transport. It runs on Stream's global edge network, which Stream reports at sub-second join times and audio latency under 30 milliseconds on the transport hop, with a P50 voice round-trip around 240 milliseconds across the full speech-to-text, LLM, and text-to-speech pipeline. Twilio will bridge the phone call into this same call later.

To run it in a browser during development, wire it to the CLI launcher:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
from vision_agents.core import Agent, AgentLauncher, Runner, User
from vision_agents.plugins import getstream, deepgram, cartesia, openai


async def create_agent() -> Agent:
    # ... as above ...
    return agent


async def join_call(agent: Agent, call_type: str, call_id: str, **kwargs) -> None:
    call = await agent.create_call(call_type, call_id)
    async with agent.join(call):
        await agent.simple_response(
            "Introduce yourself as Meridian Bank's fraud line and explain you're "
            "calling about a recent transaction that needs to be confirmed."
        )
        await agent.finish()


if __name__ == "__main__":
    Runner(AgentLauncher(create_agent=create_agent, join_call=join_call)).cli()

Run it:

1
uv run main.py run

This opens a browser UI where you can talk to the agent directly. You will hear it introduce itself and start the confirmation flow. Get this feeling right before you put it on a phone, because debugging tone and pacing is much easier in the browser than over a live call.

The Instructions Are the Product

For most voice agents, the system prompt is a personality. For a fraud bot, it is the compliance boundary, and it is the single most important part of this build.

The instructions have to do two things at once: run the confirmation flow, and hold the line on what the agent will never ask for.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
FRAUD_INSTRUCTIONS = """
You are the automated fraud line for Meridian Bank. You are placing an outbound
call to a cardholder about a transaction our system flagged as unusual. Keep
replies short and conversational. Do not use special characters or formatting.

HOW TO OPEN THE CALL
- Say you are calling from Meridian Bank's fraud department.
- Say you are calling to confirm a recent transaction, nothing more.
- Do not create panic. This is a routine check, not an accusation.

HOW TO VERIFY THE TRANSACTION
- Read back only the flagged transaction: merchant, amount, and location.
- Ask the customer to confirm whether they made this purchase, yes or no.
- If they made it, tell them you'll clear the hold and the purchase will go through.
- If they did not, tell them you'll freeze the card, open a dispute, and connect
  them to a specialist.

WHAT YOU MUST NEVER DO
- Never ask for the full card number, CVV, PIN, or expiration date.
- Never ask for the online banking password or Social Security number.
- Never ask for a one-time passcode or verification code. No real fraud line
  ever asks for these. Asking would make you indistinguishable from a scammer.
- Never ask the customer to move, send, or reverse money.
- Never send or read out a link for them to click.
- We already have the customer's account on file. You do not need to collect
  identifying secrets to do your job.

IF THE CUSTOMER IS UNSURE OR THE CALL GETS COMPLICATED
- Offer to let them hang up and call the number on the back of their card. A
  real bank encourages this. Then escalate to a human specialist.
"""

The line that does the heavy lifting is the reasoning attached to the one-time passcode rule. A model told simply "do not ask for codes" can be argued out of it by a plausible-sounding edge case. A model told why (asking would make it indistinguishable from a scammer) holds the boundary under pressure, because the instruction now carries its own justification.

This mirrors what real issuers publish.

A legitimate fraud alert confirms a transaction with a yes or no. It does not extract secrets. Banks state plainly that they will never ask you to verify a full account number, CVV, PIN, or a verification code, and consumer protection guidance is absolute that no real fraud department will ever ask for a one-time code. Encoding that directly is what makes the agent credible to the person on the other end.

Giving the Agent Actions

The agent needs to do things, not just talk.

In Vision Agents, you register plain async Python functions as tools with the @llm.register_function() decorator, and the LLM calls them when the conversation calls for it.

One rule to know before you write them: registered functions must be async. If you pass a synchronous function, registration raises a ValueError. The framework checks with inspect.iscoroutinefunction at registration time, so this fails fast rather than at runtime.

Here are the four actions the fraud flow needs. Wire them onto the LLM before you attach it to the agent:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
async def create_agent() -> Agent:
    llm = openai.LLM(model="gpt-5.5")

    @llm.register_function(
        description="Confirm a flagged transaction was legitimate and clear the hold."
    )
    async def confirm_transaction(transaction_id: str) -> dict:
        # Call your core banking system to release the hold.
        return {"status": "confirmed", "transaction_id": transaction_id}

    @llm.register_function(
        description="Freeze the customer's card after they deny a transaction."
    )
    async def freeze_card(card_last4: str) -> dict:
        return {"status": "frozen", "card_last4": card_last4}

    @llm.register_function(
        description="Open a fraud dispute for a denied transaction."
    )
    async def open_dispute(transaction_id: str) -> dict:
        return {"status": "dispute_opened", "transaction_id": transaction_id}

    @llm.register_function(
        description="Hand the call off to a human fraud specialist."
    )
    async def escalate_to_human(reason: str) -> dict:
        return {"status": "escalating", "reason": reason}

    agent = Agent(
        edge=getstream.Edge(),
        agent_user=User(id="agent", name="Meridian Fraud Line"),
        instructions=FRAUD_INSTRUCTIONS,
        stt=deepgram.STT(model="nova-3", eager_turn_detection=True),
        llm=llm,
        tts=cartesia.TTS(model_id="sonic-3.5"),
    )
    return agent

The function name and its parameters become the tool schema the LLM sees, and the description tells it when to call each one. Notice these functions take a transaction ID and a card's last four digits, never the full number. The tool signatures enforce the same boundary as the instructions: there is no parameter anywhere that could hold a full PAN, CVV, or code, so even a confused model cannot collect one.

In a real deployment, the function bodies call your core banking and case-management systems. The LLM chains them on its own: a denial leads it to call freeze_card, then open_dispute, then escalate_to_human in sequence. By default the framework allows up to three tool-calling rounds per turn; raise it with openai.LLM(model="gpt-5.5", max_tool_rounds=5) if a flow needs more steps.

Putting It on the Phone

Now the outbound call. Twilio places the call and streams the audio to your agent over a WebSocket, and Vision Agents bridges that stream into the same Stream call the agent already runs on. The agent code above does not change. You are adding a thin telephony layer around it.

The flow for an outbound call is:

1. Register the call and mint a one-time token for the media stream.
2. Tell Twilio to dial the customer and point its audio at your websocket URL.
3. When Twilio connects, validate the token, attach the phone audio to the Stream call, and let the agent run the conversation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
import asyncio
import os
import uuid

import click
import uvicorn
from dotenv import load_dotenv
from fastapi import FastAPI, WebSocket
from twilio.rest import Client

from vision_agents.plugins import twilio

load_dotenv()

NGROK_URL = os.environ["NGROK_URL"]

app = FastAPI()
call_registry = twilio.TwilioCallRegistry()


async def initiate_outbound_call(from_number: str, to_number: str) -> str:
    twilio_client = Client(
        os.environ["TWILIO_ACCOUNT_SID"], os.environ["TWILIO_AUTH_TOKEN"]
    )
    call_id = str(uuid.uuid4())

    async def prepare_call():
        agent = await create_agent()
        phone_user = User(name=f"Cardholder {call_id[:8]}", id=f"phone-{call_id}")
        await agent.edge.create_users([agent.agent_user, phone_user])
        stream_call = await agent.create_call("default", call_id)
        return agent, phone_user, stream_call

    twilio_call = call_registry.create(call_id, prepare=prepare_call)
    url = f"wss://{NGROK_URL}/twilio/media/{call_id}/{twilio_call.token}"

    twilio_client.calls.create(
        twiml=twilio.create_media_stream_twiml(url),
        to=to_number,
        from_=from_number,
    )
    return call_id

The websocket handler takes over once Twilio connects. It validates the token, accepts the media stream, waits for the prepared agent, and bridges the phone into the call with attach_phone_to_call:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
@app.websocket("/twilio/media/{call_sid}/{token}")
async def media_stream(websocket: WebSocket, call_sid: str, token: str):
    twilio_call = call_registry.validate(call_sid, token)

    twilio_stream = twilio.TwilioMediaStream(websocket)
    await twilio_stream.accept()
    twilio_call.twilio_stream = twilio_stream

    try:
        agent, phone_user, stream_call = await twilio_call.await_prepare()
        twilio_call.stream_call = stream_call

        await twilio.attach_phone_to_call(stream_call, twilio_stream, phone_user.id)

        async with agent.join(stream_call, participant_wait_timeout=0):
            await agent.simple_response(
                text="Introduce yourself as Meridian Bank's fraud line and ask the "
                "customer to confirm the flagged transaction."
            )
            await twilio_stream.run()
    finally:
        call_registry.remove(call_sid)

Finally, a small entry point starts the server and places the call. The token in the media URL is what stops a random WebSocket connection from hijacking a call, so it is validated before any audio flows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
async def run_with_server(from_number: str, to_number: str):
    config = uvicorn.Config(app, host="localhost", port=8000, log_level="info")
    server = uvicorn.Server(config)
    server_task = asyncio.create_task(server.serve())

    while not server.started:
        await asyncio.sleep(0.1)

    await initiate_outbound_call(from_number, to_number)
    await server_task


@click.command()
@click.option("--from", "from_number", required=True, help="Your Twilio number.")
@click.option("--to", "to_number", required=True, help="The number to call.")
def main(from_number: str, to_number: str):
    asyncio.run(run_with_server(from_number, to_number))


if __name__ == "__main__":
    main()

Place a call:

1
uv run outbound.py --from "+15551234567" --to "+15559876543"

Twilio sends audio as mulaw at 8kHz. The plugin handles the conversion to and from the format your pipeline expects, so you do not deal with codecs directly. Latency is higher in local development than in production. For a real deployment, run close to your users; on Stream's edge, that means deploying to the region nearest them.

What About Compliance?

A few things are worth saying plainly, and none of this is legal advice.

On the technical side, keep card data out of your transcripts and logs. The pipeline in this tutorial never captures a full card number because it never asks for one, which is the cleanest way to stay out of trouble: the safest sensitive data is the data you never collect. The tool signatures back this up by only accepting a transaction ID and a card's last four digits.

The behavioral rules in the instructions are not "polish you add at the end". They are what make the call trustworthy to the person answering it. A bot that confirms a transaction and offers to let the customer call the number on the back of their card is behaving exactly like a real bank. A bot that asks for a verification code is behaving exactly like a scammer.

The gap between those two is the entire product.

Where to Go Next

You have an outbound fraud-alert agent that places a real phone call, confirms a transaction, and takes action on the result, with the trust boundary enforced in both the instructions and the tool signatures.

From here:

• Swap providers to fit your market. The speech-to-text, LLM, and text-to-speech pieces are all one-line changes. Sarvam AI covers Indian languages, for example.
• Add answering-machine handling for the outbound path so the agent leaves a callback message instead of talking to voicemail.
• Move to a realtime speech-to-speech model if you want the lowest possible latency and can accept coarser transcripts.
• Skip the scaffolding entirely. If you use Claude Code, Cursor, or Codex, Stream's Agent Skills let you drop the voice agent into your project and describe what you want in plain English:

1
npx skills add GetStream/agent-skills -s stream

The phone build in this tutorial is grounded in the framework's own phone example, which pairs telephony with a knowledge base if you want the agent to answer follow-up questions.

Start Building

Clone the repo, wire in your own fraud logic, and place a test call to your own phone: github.com/GetStream/Vision-Agents.

If you want your fraud agent deployed in the same region as your existing Stream chat and video, so a support call, a video session, and a fraud verification all run on one stack, join the hosted-agents waitlist. You can start building on the open-source framework right now either way.