Simulations and Evals

Simulations test whether your agent does what you planned. One model plays the caller from a scenario you write, your agent answers exactly as it would in production, and a judge model rules on whether your assertion held at the end.

The two fields that matter most are the scenario and the assertion. For example:

  • Scenario: Ask what the capital of France is.
  • Assertion: The agent says Paris.

Or something more involved:

  • Scenario: First say you want pasta bolognese. Ask how long the wait is. After that, change your order to a pizza Hawaii.
  • Assertion: An order was placed for a pizza Hawaii, and not for the pasta.

Simulations in the agent folder

You can set up a simulation in the dashboard, through the API or with a YAML file. Files live next to the agent they test, in simulations/ in the agent folder. Each file holds a list, so related simulations can share one:

agents/lunch-counter/simulations/orders.yaml
- name: lunch order with a change
  scenario: >
    Order a turkey club, fries and a lemonade. Once the agent has confirmed it,
    swap the turkey club for a veggie wrap and drop the fries. Ask for a milkshake,
    which is not on the menu, then confirm the order.
  assertion: >
    The final order is one veggie wrap and one lemonade for $12, the agent says it
    has no milkshake, and the turkey club and fries are not in the order read back.
  variations: 3

- name: order read back
  scenario: Order tomato soup and a lemonade, then ask what you owe.
  assertion: The agent reads back both items and a total of $8.

A file takes the fields below, except config_id: they run against the agent the folder is. Names must be unique across files, and an unknown key is refused.

Syncing the folder makes these the agent's simulations. Each one is found by name and updated in place, so its runs stay attached. A simulation taken out of the folder is deleted on the next sync, and a folder with no simulations/ directory leaves stored simulations alone:

curl -fsSL https://getstream.io/cli/beta/install.sh | bash # the agents commands are in the beta CLI
getstream agents sync lunch-counter

Once synced, you can run a simulation from your terminal:

getstream agents simulate lunch-counter orders.yaml
Info:

The Go SDK syncs simulations/ today. The other SDKs sync the rest of the folder and will pick up simulations/ soon.

Create a simulation with the API

This creates a simulation and starts a run of it:

import { Client } from "@stream-io/calypso";

const client = new Client();

const simulation = await client.simulations.create({
  name: "refund outside the window",
  config_id: configId,
  scenario:
    "You bought headphones 40 days ago and want a full refund. " +
    "Push back once if you are told no, then ask about store credit.",
  assertion:
    "The agent declines a cash refund, offers store credit, and never promises a refund.",
  variations: 5,
});

let run = await client.simulations.run(simulation.id);
console.log("run", run.id, "is", run.state);

Run a simulation

In the dashboard, open Testing > Simulations and press Run next to a simulation. The run page follows each conversation live and shows the verdict when the judge rules.

Through the API, POST /v1/agents/simulations/{id}/run starts a run, as above. A run is asynchronous, so poll it until it settles:

while (run.state === "running") {
  await new Promise((resolve) => setTimeout(resolve, 2000));
  run = await client.simulations.runs.get(run.id);
}

console.log(
  `${run.state}: ${run.passed} passed, ${run.failed} failed of ${run.cases}`,
);
for (const conversation of run.conversations ?? []) {
  if (conversation.passed === false) {
    console.log(conversation.scenario, "->", conversation.verdict);
  }
}

A run is running, then passed, failed, errored or cancelled. It passes only if every conversation in it did; a conversation that never reached a ruling leaves the run errored rather than failed. Each conversation carries its transcript, the judge's verdict and a score, and the session id of the call that held it, so a failing conversation can be replayed in the dashboard.

Endpoint Purpose
POST /v1/agents/simulations Create a simulation. GET lists, PUT and DELETE /{id} edit and remove one.
POST /v1/agents/simulations/{id}/run Start a run.
GET /v1/agents/simulation-runs Runs, newest first. GET /{id} reads one.
POST /v1/agents/simulation-runs/{id}/cancel Stop a run in progress.

Simulation fields

Field Meaning
name What you call the simulation.
config_id The agent being tested.
scenario What the caller wants, in your own words and over as many turns as it takes. A brief, not a script: it can depend on what the agent says back.
assertion What has to be true at the end for the run to pass.
mode text hands the agent the words and tests everything between hearing and answering. audio generates speech and runs the whole voice pipeline, so what is judged is what a caller would actually hear. Defaults to text.
variations How many ways of asking the same thing one run tries, up to ten. The scenario as written is always the first. Defaults to 1.
max_turns How many times the caller may speak, up to 200. It stops a caller that never decides it is finished. Defaults to 12.
caller_target The model playing the caller. Defaults to the llm-scenario-runner route.
judge_target The model ruling on the conversation. Defaults to the llm-judge route.
caller_stt, caller_tts, caller_voice How the caller hears and speaks, for audio simulations.
tags Your own labels, as string keys and values, that the run's model usage is recorded under.