Warm Transfer vs Cold Transfer: Designing AI-to-Human Escalation in Real-Time Voice Systems

TL;DR

• Warm transfer carries the conversation's context to the human agent; cold transfer drops it, and the caller has to start over.
• Building warm transfer means solving three problems: deciding when to escalate, generating a structured summary in parallel with the live call, and moving the parties without breaking the audio session.
• The room-participant model (where the AI joins as a peer in a WebRTC room rather than a step in a call flow) makes all three tractable.
• Policy and mechanism should stay separate.

If you've recently found that companies are answering your customer support calls more quickly, it is because you're talking to AI.

Talking to a robot isn't like it used to be. It's not a synthetic voice and "press 1 for account information." It's a voice called "Piper" that sounds like a friendly support agent who listens and understands your problem.

But it is still AI, and AI can't (yet) do everything.

These agents resolve the bulk of routine calls on their own, the order-status checks, password resets, and billing lookups, without a human ever picking up. The hard calls are a different story. And how they take over can be a seamless handoff or the moment the whole call falls apart.

That gap is the difference between a warm transfer and a cold one. Warm transfer carries the conversation's context to the human agent - the summary, the intent, the caller's mood - so they can pick up mid-story rather than from scratch. Cold transfer carries only the audio. Building the warm version means solving three problems: deciding when to escalate, generating structured context in parallel with the live call, and moving the parties without breaking the audio session.

This post covers how to do all three.

What We Mean by Warm and Cold

Say you're calling about a duplicate charge. The AI confirms you need a refund, but has a hard limit of $100 for automated refunds, so it needs to transfer you to a human. "Let me transfer you to a specialist." Click. Fifteen seconds of hold music. A new voice: "Hi, this is Marcus. Can I get your account number?" And you start all over. The charge date, the amount, everything you already said.

This is a cold transfer, where Marcus, though perfectly friendly (a cold transfer doesn't mean the human is frosty), has no context for the call. Effectively, you've just gone from one system to another without any information transfer.

With a warm transfer, the information is transferred along with the call. You go from the AI telling you they are connecting you to a specialist, via the hold music, to Marcus saying: "Hi, I see the $129 charge from April 12 and your cancellation on the 5th. Let me get that refund going." You didn't repeat a thing. Marcus walked in already knowing the story.

The difference is what gets carried across the handoff:

• Cold: Just the audio. The AI drops off, you land on a new person, and the whole conversation leaves with the AI.
• Warm: The audio plus a summary of everything so far, on the human's screen before they say a word.

For a developer, that summary is where the work is. Building and delivering it means three jobs in three places:

1. The trigger. Decide if an escalation is needed, and what kind. Lives in the AI voice agent.
2. The context. Turn the running conversation into something the next human can absorb in ten seconds. A separate LLM call, ideally to a model that isn't busy producing voice.
3. The handoff. Move the parties: keep the audio alive, get the human in, put the context on their screen, and retire the AI.

Cold skips the context step, keeping the handoff simple. Warm does all three. But if warm is so clearly better, why is cold the default almost everywhere?

• It bolts onto systems that fight it. Most AI agents are added to a contact center built to hand off a call and forget it, not to keep a summary flowing alongside the audio.
• It costs more. A cold handoff lets the AI provider drop off the moment the human picks up; a warm one keeps the meter running while the briefing happens.
• It's more to build. Voicemail detection, sensible timeouts, hold audio, a fallback when nobody picks up, recordings that stitch cleanly across the handoff: each one a small project.

Those three used to be real, but they aren't anymore. Major voice platforms now ship warm transfer as a built-in. The only question left is what architecture you're building on, and whether it makes the handoff easy or hard.

The Telephony Way

The older approach treats the AI as just another task within a contact-center workflow. Twilio's ConversationRelay is the example most people know. The caller's audio comes in over a SIP trunk. ConversationRelay handles speech recognition and synthesis itself; your app receives transcribed prompts over a WebSocket and sends back text tokens, which Twilio plays as the AI's voice. As far as the platform is concerned, the AI is a step in a call flow, not a participant in a conversation.

You won't hand-write the signaling here, but it's worth seeing, because this is where the cold/warm split physically lives. The classic SIP-layer primitive for transfer is REFER, and it shows up in two shapes:

• Blind transfer. The AI sends a REFER pointing at the human, then hangs up. The caller's phone places a fresh call to the human, who picks up cold. The textbook cold transfer.
• Attended transfer. The AI calls the human first on a separate line, briefs them, then sends a REFER that swaps that consultation into the live call. The caller never hears the side conversation. Warm transfer, SIP-style.

On a platform like Twilio, you express the warm version in TwiML when the AI decides to escalate. answerOnBridge keeps the caller on hold audio until the human actually answers, and machineDetection on the <Sip> noun makes sure you're bridging to a person and not their voicemail:

1
2
3
<Dial answerOnBridge="true" timeout="30">
  <Sip machineDetection="DetectMessageEnd">sip:supervisor@example.sip.us1.twilio.com?X-Account-Id=ACME-99231</Sip>
</Dial>

Then the hard part: getting context to the human. The telephony answer is to stuff it into SIP headers.

• User-to-User carries an account ID or intent code across providers without leaving the SIP control plane. But the practical payload is small.
• Custom headers buy a little more room when both ends are yours, but carriers strip them inconsistently, which has cost more than one engineer a Friday afternoon.

That small payload budget is a problem. The information you need for a warm transfer just doesn't fit.

What this architecture does get you is deep integration with an existing contact center. Skills-based routing, queue overflow, compliance recording, and supervisor dashboards. But these weren't built for AI to be a peer in the conversation.

The Room-Participant Way

Instead of treating the AI as a step in a call flow, the room model treats it as a participant in a room. The AI connects to a WebRTC room as a server-side peer. It subscribes to the caller's audio, publishes its own, and sits in the participant list right next to any humans who join. Telephony, when there is any, comes in through a SIP gateway at the edge of the room.

Once the AI is a participant instead of a task, the warm transfer stops being a signaling problem and becomes an ordinary API operation:

• The AI decides to escalate.
• A separate service turns the running transcript into a structured summary.
• The human's screen renders that summary before they accept.
• The human joins the same room the caller has been in the whole time.
• The AI says goodbye and leaves.

The caller never notices a seam. No REFER, no small payload, and no second call leg.

That's the difference that matters. In the telephony world, context has to squeeze through the signaling. In the room world, the audio and the context travel side by side over a single connection, with as much room for the context as you want.

For a general-purpose voice agent in 2026, this is the right default. It maps cleanly onto multi-agent designs, it makes passing context trivial, and it matches how WebRTC-native voice stacks are actually built.

Designing the Warm Transfer

Four services do the work:

• Stream Video for the audio session: WebRTC calls, server-side participant management, the AI agent joins as a peer through Stream's edge network.
• OpenAI Realtime for the AI voice agent, connected to the call through Stream's connectOpenAi helper, so the bidirectional audio flows over the same WebRTC connection.
• Stream Chat for context delivery: one channel per escalation, with custom data fields that the supervisor's dashboard reads to render the context panel.
• GPT-4o (standard chat completions with Structured Outputs against the HandoffContext schema) for summary generation, kept separate from the Realtime session so the AI doesn't need to pause to think.

None of these four is load-bearing; the architecture is. AI as a same-call participant, structured context generated in parallel and delivered to the supervisor before the audio handoff, policy separated from mechanism, so the rules can change without touching the transfer code.

The Data Model

Every part of the system shares two types. The first is a state machine for the transfer itself:

1
2
3
4
5
6
7
8
9
export enum TransferState {
  AI_HANDLING,
  ESCALATION_TRIGGERED,
  PREPARING_TRANSFER,
  AWAITING_AGENT,
  TRANSFERRING,
  HUMAN_HANDLING,
  FAILED,
}

Every transition between states emits an event. The dashboard renders from those events; the metrics layer logs the durations between them; and the orchestrator uses them to guard against double-firing escalations.

The second is the payload that travels from the AI to the human:

1
2
3
4
5
6
7
8
9
10
export interface HandoffContext {
  summary: string;                     // one-paragraph summary
  intent: string;                      // 2-5 word intent classification
  sentiment: SentimentLevel;           // positive | neutral | frustrated | angry
  keyEntities: Record<string, string>; // account IDs, names, dates, order numbers
  escalationReason: EscalationReason;
  transcript: ConversationTurn[];
  conversationDuration: number;
  turnCount: number;
}

The shape of HandoffContext matters more than any single field. The LLM pulls out intent and sentiment so the dashboard can render them as badges at the top. Key entities are extracted by name so the supervisor can verify them before they say a word. The raw transcript remains available for anyone who wants to scroll through it. Every other module in the system is downstream of this object.

Rules as Data

The decision of whether to use a cold or warm transfer doesn't live in code. It lives in configuration:

1
2
3
4
5
6
7
8
9
10
11
export const defaultEscalationConfig: EscalationConfig = {
  rules: [
    { reason: EscalationReason.USER_REQUESTED,    transferType: 'warm', confidenceThreshold: 0.8 },
    { reason: EscalationReason.NEGATIVE_SENTIMENT, transferType: 'warm', confidenceThreshold: 0.7 },
    { reason: EscalationReason.AI_CAPABILITY_LIMIT, transferType: 'warm', confidenceThreshold: 0.6 },
    { reason: EscalationReason.TOPIC_BASED_RULE,   transferType: 'cold', confidenceThreshold: 0.9 },
    { reason: EscalationReason.MAX_TURNS_EXCEEDED, transferType: 'cold', confidenceThreshold: 1.0 },
  ],
  defaultTransferType: 'warm',
  agentTimeoutSeconds: 120,
};

The default is warm. Topic-based routing drops to cold because the AI hasn't engaged with the caller's specifics yet, so there's nothing worth carrying anyway. Frustrated callers get warm at a lower confidence threshold (0.7) than explicit human requests (0.8), because the cost of a false negative on sentiment - a frustrated caller dropped into a cold queue - is the worst possible failure mode.

In production, these rules belong in a database so escalation tuning doesn't require a deployment. To determine tuning, you should think about these heuristics for when the AI should escalate:

1. Explicit user request. Always wins. Don't gate this behind retry loops.
2. Intent confusion. The intent classifier returns low confidence, or the LLM's response confidence falls below the threshold. Production bands tend to land around 0.5 for hard escalation, 0.8 for confident proceed.
3. Sentiment trend. Real-time prosody and language analysis tracking a moving sentiment score. Trigger on sustained negative shift, not a single angry sample. False positives are common.
4. Compliance hard stops. Regulated financial advice, HIPAA-sensitive disclosures, refund disputes over a dollar threshold, and mentions of self-harm.
5. Loop detection. Same intent cycling without resolution, or repeat contact within a window. The most under-instrumented trigger.

The failure mode to watch for is escalating too late, not too early.

Policy and Mechanism Are Kept Apart

The orchestrator matches the incoming trigger against the rules, decides cold or warm, and hands off to a handler:

1
2
3
4
5
6
7
8
const rule = this.matchRule(trigger);
const transferType = rule?.transferType ?? this.config.defaultTransferType;

if (transferType === 'cold') {
  await this.coldHandler.execute(event);
} else {
  await this.warmHandler.execute(event, transcript);
}

That's the whole architectural point: policy lives in the rules, mechanism lives in the handlers, and the orchestrator just makes the call. Want to add a new transfer type, say a "supervised" mode where the human listens silently before taking over? Write a new handler, extend the transferType union, and leave the orchestrator alone.

The Warm Transfer Flow

This is the core of how the transfer works:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
async execute(event: EscalationEvent, transcript: ConversationTurn[]) {
  // Phase 1: Build structured context (parallel with AI talking to caller)
  const context = await this.contextBuilder.buildContext(transcript, event.trigger);
  event.context = context;
  this.emitStateChange(event, TransferState.PREPARING_TRANSFER);

  // Phase 2: Create a Stream Chat channel seeded with the context
  const channelId = `escalation-${event.id}`;
  await this.contextBuilder.postContextToChannel(
    channelId, context, event.assignedAgentId!,
  );
  event.streamChannelId = channelId;
  this.emitStateChange(event, TransferState.AWAITING_AGENT);

  // Phase 3: Notify the supervisor's dashboard
  this.emit('agent-notification', {
    escalationId: event.id,
    streamChannelId: channelId,
    context: { summary: context.summary, intent: context.intent, sentiment: context.sentiment },
  });

  // Phase 4: Add the supervisor to the SAME call and hand them a token
  const supervisorId = `human-agent-${event.assignedAgentId}`;
  const call = this.streamClient.video.call(event.callType, event.callId);
  await call.updateCallMembers({
    update_members: [{ user_id: supervisorId, role: 'admin' }],
  });
  const token = this.streamClient.generateUserToken({ user_id: supervisorId });
  this.emit('agent-token-ready', {
    escalationId: event.id,
    token,
    callId: event.callId,  // same call — this is the difference
  });
  this.emitStateChange(event, TransferState.TRANSFERRING);

  // Phase 5: Wait for the supervisor to actually join, then verbal handoff
  await this.waitForSupervisorJoin(call, supervisorId);
  await this.triggerVerbalHandoff(event, context);

  // Phase 6: Disconnect the AI's Realtime session, then drop its membership
  this.emit('disconnect-ai', { escalationId: event.id });
  await call.updateCallMembers({ remove_members: ['ai-agent'] });
  this.emitStateChange(event, TransferState.HUMAN_HANDLING);
}

This goes:

• Phase 1: Build the context. Runs in parallel with the AI still talking. While the caller hears "got someone for you, one moment," GPT-4o is already chewing through the transcript and producing the structured handoff.
• Phase 2: Seed the Stream Chat channel. The context lands in a channel before the supervisor even sees the notification. The channel also doubles as the audit trail after the call ends.
• Phase 3: Notify the supervisor. A condensed slice of the context (intent, sentiment, one-line summary) is sent to the dashboard so the supervisor can decide whether to accept it before opening the channel.
• Phase 4: Invite the supervisor into the same call. This is the architectural payoff. The supervisor is added as a member of the call the caller has been in since the start, and is handed a token to join it. No routing to a queue. No moving the caller. The WebRTC session never breaks. Three participants briefly share the call, then the AI leaves. The token identifies the supervisor; access to this call is controlled separately by the membership update and by the role permissions configured for the call type.
• Phase 5: Verbal handoff. A session_participant_joined event from Stream confirms the supervisor is actually on the call, not just authorized to join. Then the AI says something like, "Hi Sarah, I've got John on the line; he's calling about a billing dispute on his April invoice, he's been a bit frustrated, and I've already confirmed his cancellation date in the CRM." This is what the caller hears. They've already heard the AI promise to bring someone up to speed; now they hear it actually happen. That's where the trust comes from.
• Phase 6: Disconnect the AI. The voice agent process closes its RealtimeClient connection, which leaves the call. Membership cleanup follows. The caller and the supervisor are alone on the line.

The script gets built dynamically and sent into the AI's session as an instruction to speak:

1
2
3
4
5
6
7
8
const handoffScript = [
  `A human agent has joined the call.`,
  `Introduce them to the caller briefly.`,
  `Mention the caller is contacting about: ${context.intent}.`,
  entities ? `Key details: ${entities}.` : '',
  `The caller's current mood is ${context.sentiment}.`,
  `After the introduction, say goodbye and let them know they're in good hands.`,
].filter(Boolean).join(' ');

The voice agent process receives this script via an internal event and feeds it to the OpenAI Realtime client, which produces a natural-sounding spoken version.

This does take time, but in this moment of the call, callers give you time - they are used to hold music or pauses for transfers in customer support calls. You just don't want dead air, the gap between the AI's "got someone for you" and the supervisor's first words. Hold audio, ambient noise, or the AI itself going "want to keep me company while we wait?" are all ways to keep the gap bearable.

Generating the Context

The context builder is what turns the transcript into something the supervisor can absorb in 10 seconds.

The design decision worth flagging is using GPT-4o standard chat completions for this, not the Realtime API. The Realtime session is busy producing voice for the caller; interrupting it to think about the summary structure would create an audible pause. A separate text completion is faster, cheaper, and produces more reliable JSON.

The prompt is structured to enforce that JSON shape:

1
2
3
4
5
6
7
8
9
10
11
12
13
const SUMMARIZATION_PROMPT = `You are analyzing a customer service conversation
that is being escalated to a human agent. Produce a JSON object with these fields:

- "summary": A concise one-paragraph summary. Lead with what the customer needs,
  what the AI tried, and why escalation is happening.
- "intent": The customer's primary intent in 2-5 words.
- "sentiment": One of "positive", "neutral", "frustrated", or "angry". Base this
  on the overall tone, not just the last message.
- "keyEntities": An object mapping entity type to value. Extract names, account
  numbers, product names, dates, order numbers, anything the human agent would
  want at a glance.

Be concise. The human agent needs to understand the situation in 10 seconds.`;

Once the JSON comes back, it gets posted to a Stream Chat channel with the structured fields attached as custom data:

1
2
3
4
5
6
7
8
9
10
11
12
const channel = this.streamChat.channel('messaging', channelId, {
  name: `Escalation: ${context.intent}`,
  members: ['system', agentUserId],
  escalation_context: {
    summary: context.summary,
    intent: context.intent,
    sentiment: context.sentiment,
    keyEntities: context.keyEntities,
    // ...
  },
});
await channel.create();

The transcript also gets posted as individual messages, so the supervisor can scroll through them naturally using Stream Chat's built-in message components. The channel stays active after the call ends, so it serves as the audit trail for post-call reviews or supervisor handoffs.

Triggering the Escalation From the Voice Agent

The voice agent itself runs as a Node process that connects an OpenAI Realtime agent to the call:

1
2
3
4
5
6
7
8
9
10
11
12
const call = streamClient.video.call('default', callId);
const realtimeClient = await streamClient.video.connectOpenAi({
  call,
  openAiApiKey: process.env.OPENAI_API_KEY!,
  agentUserId: 'ai-agent',
});

realtimeClient.updateSession({
  instructions: AGENT_INSTRUCTIONS,
  voice: 'alloy',
  // turn detection, audio transcription, etc.
});

connectOpenAi returns Stream's RealtimeClient, a wrapper around OpenAI's Realtime API that smooths over version churn in the session shape and tool-call event names. The code below uses the wrapper's methods; consult OpenAI's reference for the current raw event names if you want to drop down a layer.

The escalation gets triggered in two ways. The first is a tool the AI can call directly, registered through the wrapper's addTool helper:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
realtimeClient.addTool({
  type: 'function',
  name: 'request_human_agent',
  description:
    'Transfer the caller to a human agent. Call this when: ' +
    '(1) the caller explicitly asks for a human, ' +
    '(2) you cannot resolve their issue after trying, or ' +
    '(3) the caller is clearly frustrated and needs human empathy.',
  parameters: { /* reason, brief_summary */ },
}, async ({ reason, brief_summary }) => {
  escalationManager.handleEscalation({
    reason: reason as EscalationReason,
    confidence: 0.95,
    detectedAt: new Date(),
    triggerPhrase: brief_summary,
  }, callerId, callId, transcript);
  return 'Transfer initiated. Let the caller know you are connecting them with a specialist.';
});

The tool's description is the actual prompt the LLM sees when deciding whether to escalate. That description is what shapes the AI's behavior, so it deserves more engineering care than people usually give it. A vague description ("call this when needed") gives you flaky escalation. Specific criteria, with examples, give you reliable behavior.

The second mechanism is a passive sentiment monitor that runs after every user turn. The transcribed user speech arrives asynchronously as its own event:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
realtimeClient.on('conversation.item.input_audio_transcription.completed', (event) => {
  transcript.push({
    role: 'user',
    content: event.transcript,
    timestamp: new Date(),
  });

  if (!escalationInProgress) {
    const sentimentTrigger = shouldEscalateOnSentiment(transcript);
    if (sentimentTrigger) {
      escalationInProgress = true;
      escalationManager.handleEscalation(sentimentTrigger, /* ... */);
    }
  }
});

These transcripts are best-effort from the audio model. They won't always exactly match what the LLM inferred from the audio itself, which is part of why a moving average over recent turns is more reliable than acting on any single sample.

This catches the callers whose tone shifts without them ever saying "let me talk to a human." They go quiet. They get clipped. They start repeating themselves with rising frustration. A passive monitor analyzing recent turns catches the shift before it turns into a hangup.

What the Supervisor Sees

The supervisor's dashboard is a Stream Chat React UI with one custom panel above the message list.

1
2
3
<EscalationContextPanel context={channel.data.escalation_context} />
<MessageList />
<TransferActions onAccept={...} onDecline={...} />

When a warm transfer fires, a new channel appears in their incoming list. The channel header shows the intent and a sentiment badge color-coded by mood: green for positive, gray for neutral, amber for frustrated, red for angry. The supervisor sees those at a glance before they ever open the channel.

When they open it, the context panel renders above the message list, displaying the AI-generated summary, escalation reason, conversation duration, and any key entities the AI extracted. The full transcript scrolls below.

The supervisor reads all of that before they click accept. By the time they join the Stream call, they know who the caller is, what they want, what's already been tried, and what kind of mood they're walking into. The AI's verbal handoff is just audio reinforcement of what's already on the supervisor's screen.

This is the architectural payoff of the room-participant model. The audio session, the structured context, the persistent transcript: all of it rides on one connection layer. The supervisor isn't waiting on a screen pop from a separate CRM, hoping the data shows up before they have to start talking. The Stream Chat channel that provided the context remains available throughout the call, so it doubles as the supervisor's note-taking surface during the conversation and the searchable record afterward.

What To Measure

A few numbers will tell you whether the warm-transfer system is actually doing what it's supposed to. Most of them mislead when read in isolation, so the trick is to read them in pairs.

• Containment rate is the obvious metric and the most-gamed. CallMiner's critique is worth taking seriously: containment optimized in isolation pushes frustrated callers into automated loops, because it measures whether the customer was prevented from reaching a more expensive channel, not whether they got their problem solved. Always pair it with a downstream outcome metric.
• Repeat-context rate. What fraction of transferred customers had to re-explain their issue to the human? It's the single best signal for warm-transfer quality.
• Re-escalation rate. A handed-off interaction that gets escalated again. High values mean wrong routing or incomplete summaries.
• Handoff latency. Time from "I'll get someone" to the first words from the human. Under 8 seconds keeps the call alive.
• Post-transfer CSAT vs AI-only CSAT, as a delta. The downstream outcome that everything else is a proxy for.

Warm Is Where the World Is Going

In 2026, warm transfer is the default. Cold is the special case, for the calls where there's nothing worth carrying across.

The architecture that makes warm tractable is the room-participant model, in which the AI is a peer in the room rather than a step in a workflow. What matters is that the AI is a participant, that policy and mechanism remain separate, that the structured context appears on the supervisor's screen before the bridge completes, and that the supervisor joins the same call the caller has been on the whole time.

If you're shipping your first voice agent, lean on a built-in warm transfer primitive from whichever platform you're already on. If you're already in production, instrument repeat-context rate first; the rest follows from there. If you're at scale, move to a room-native stack and treat telephony as an edge concern rather than the architectural center.

The architectural payoff is that the caller no longer has to repeat themselves, which may sound small until you're the caller.