# Simulations and Evals

Simulations test whether your agent does what you planned. One model plays the caller from a scenario you write, your agent answers exactly as it would in production, and a judge model rules on whether your assertion held at the end.

The two fields that matter most are the `scenario` and the `assertion`. For example:

- Scenario: Ask what the capital of France is.
- Assertion: The agent says Paris.

Or something more involved:

- Scenario: First say you want pasta bolognese. Ask how long the wait is. After that, change your order to a pizza Hawaii.
- Assertion: An order was placed for a pizza Hawaii, and not for the pasta.

## Simulations in the agent folder

You can set up a simulation in the dashboard, through the API or with a YAML file. Files live next to the agent they test, in `simulations/` in the [agent folder](https://getstream.io/agents/docs/python/creating-an-agent/). Each file holds a list, so related simulations can share one:

```yaml title="agents/lunch-counter/simulations/orders.yaml"
- name: lunch order with a change
  scenario: >
    Order a turkey club, fries and a lemonade. Once the agent has confirmed it,
    swap the turkey club for a veggie wrap and drop the fries. Ask for a milkshake,
    which is not on the menu, then confirm the order.
  assertion: >
    The final order is one veggie wrap and one lemonade for $12, the agent says it
    has no milkshake, and the turkey club and fries are not in the order read back.
  variations: 3

- name: order read back
  scenario: Order tomato soup and a lemonade, then ask what you owe.
  assertion: The agent reads back both items and a total of $8.
```

A file takes the [fields](#simulation-fields) below, except `config_id`: they run against the agent the folder is. Names must be unique across files, and an unknown key is refused.

Syncing the folder makes these the agent's simulations. Each one is found by name and updated in place, so its runs stay attached. A simulation taken out of the folder is deleted on the next sync, and a folder with no `simulations/` directory leaves stored simulations alone:

```sh
curl -fsSL https://getstream.io/cli/beta/install.sh | bash # the agents commands are in the beta CLI
getstream agents sync lunch-counter
```

Once synced, you can run a simulation from your terminal:

```sh
getstream agents simulate lunch-counter orders.yaml
```

> **Info:** The Go SDK syncs `simulations/` today. The other SDKs sync the rest of the folder and will pick up `simulations/` soon.

## Create a simulation with the API

This creates a simulation and starts a run of it:

```ts label="JavaScript"
import { Client } from "@stream-io/calypso";

const client = new Client();

const simulation = await client.simulations.create({
  name: "refund outside the window",
  config_id: configId,
  scenario:
    "You bought headphones 40 days ago and want a full refund. " +
    "Push back once if you are told no, then ask about store credit.",
  assertion:
    "The agent declines a cash refund, offers store credit, and never promises a refund.",
  variations: 5,
});

let run = await client.simulations.run(simulation.id);
console.log("run", run.id, "is", run.state);
```

```ruby label="Ruby"
api = GetStream::VisionAgents::Client.new

simulation = api.post("/v1/agents/simulations", body: {
  name: "refund outside the window",
  config_id: config_id,
  scenario: "You bought headphones 40 days ago and want a full refund. " \
            "Push back once if you are told no, then ask about store credit.",
  assertion: "The agent declines a cash refund, offers store credit, and never promises a refund.",
  variations: 5
})

run = api.post("/v1/agents/simulations/{id}/run", path: { id: simulation["id"] })
puts "run #{run["id"]} is #{run["state"]}"
```

```php label="PHP"
use GetStream\VisionAgents\Client;
use GetStream\VisionAgents\Generated\{Simulation, SimulationRequest, SimulationRun};

$client = new Client();

$simulation = Simulation::fromArray($client->post('/v1/agents/simulations', body: new SimulationRequest(
    name: 'refund outside the window',
    configId: $configId,
    scenario: 'You bought headphones 40 days ago and want a full refund. '
        . 'Push back once if you are told no, then ask about store credit.',
    assertion: 'The agent declines a cash refund, offers store credit, and never promises a refund.',
    variations: 5,
)->toArray()));

$run = SimulationRun::fromArray($client->post('/v1/agents/simulations/{id}/run', ['id' => $simulation->id]));
echo "run {$run->id} is {$run->state}\n";
```

```go label="Go"
client, err := stream.Backend{}.Client()
if err != nil {
	return err
}

variations := 5
created, err := client.CreateSimulationWithResponse(ctx, acceleration.SimulationRequest{
	Name:     "refund outside the window",
	ConfigId: configID,
	Scenario: "You bought headphones 40 days ago and want a full refund. " +
		"Push back once if you are told no, then ask about store credit.",
	Assertion:  "The agent declines a cash refund, offers store credit, and never promises a refund.",
	Variations: &variations,
})
if err != nil {
	return err
}
simulation := created.JSON201

started, err := client.RunSimulationWithResponse(ctx, simulation.Id)
if err != nil {
	return err
}
run := started.JSON202
fmt.Println("run", run.Id, "is", run.State)
```

```csharp label="C#"
using var client = new VisionAgentsClient();

var simulation = await client.PostAsync<Simulation>("/v1/agents/simulations", new SimulationRequest
{
    Name = "refund outside the window",
    ConfigId = configId,
    Scenario = "You bought headphones 40 days ago and want a full refund. "
        + "Push back once if you are told no, then ask about store credit.",
    Assertion = "The agent declines a cash refund, offers store credit, and never promises a refund.",
    Variations = 5,
});

var run = await client.PostAsync<SimulationRun>($"/v1/agents/simulations/{simulation.Id}/run", null);
Console.WriteLine($"run {run.Id} is {run.State}");
```

```rust label="Rust"
use vision_agents::types::SimulationRequest;

let client = Client::from_env()?;

let simulation = client
    .create_simulation(&SimulationRequest {
        name: "refund outside the window".into(),
        config_id: config_id.clone(),
        scenario: "You bought headphones 40 days ago and want a full refund. \
                   Push back once if you are told no, then ask about store credit."
            .into(),
        assertion: "The agent declines a cash refund, offers store credit, and never promises a refund."
            .into(),
        variations: Some(5),
        ..Default::default()
    })
    .await?;

let run = client.run_simulation(&simulation.id).await?;
println!("run {} is {}", run.id, run.state);
```

## Run a simulation

In the dashboard, open **Testing > Simulations** and press **Run** next to a simulation. The run page follows each conversation live and shows the verdict when the judge rules.

Through the API, `POST /v1/agents/simulations/{id}/run` starts a run, as above. A run is asynchronous, so poll it until it settles:

```ts label="JavaScript"
while (run.state === "running") {
  await new Promise((resolve) => setTimeout(resolve, 2000));
  run = await client.simulations.runs.get(run.id);
}

console.log(
  `${run.state}: ${run.passed} passed, ${run.failed} failed of ${run.cases}`,
);
for (const conversation of run.conversations ?? []) {
  if (conversation.passed === false) {
    console.log(conversation.scenario, "->", conversation.verdict);
  }
}
```

```ruby label="Ruby"
loop do
  run = api.get("/v1/agents/simulation-runs/{id}", path: { id: run["id"] })
  break unless run["state"] == "running"

  sleep 2
end

puts "#{run["state"]}: #{run["passed"]} passed, #{run["failed"]} failed of #{run["cases"]}"
run["conversations"].each do |conversation|
  puts "#{conversation["scenario"]} -> #{conversation["verdict"]}" if conversation["passed"] == false
end
```

```php label="PHP"
while ($run->state === 'running') {
    sleep(2);
    $run = SimulationRun::fromArray($client->get('/v1/agents/simulation-runs/{id}', ['id' => $run->id]));
}

echo "{$run->state}: {$run->passed} passed, {$run->failed} failed of {$run->cases}\n";
foreach ($run->conversations ?? [] as $conversation) {
    if ($conversation->passed === false) {
        echo $conversation->scenario, ' -> ', $conversation->verdict, "\n";
    }
}
```

```go label="Go"
for {
	got, err := client.GetSimulationRunWithResponse(ctx, run.Id)
	if err != nil {
		return err
	}
	run = got.JSON200
	if run.State != acceleration.SimulationRunStateRunning {
		break
	}
	time.Sleep(2 * time.Second)
}

fmt.Printf("%s: %d passed, %d failed of %d\n", run.State, run.Passed, run.Failed, run.Cases)
for _, conversation := range *run.Conversations {
	if conversation.Passed != nil && !*conversation.Passed {
		fmt.Println(conversation.Scenario, "->", *conversation.Verdict)
	}
}
```

```csharp label="C#"
while (run.State == "running")
{
    await Task.Delay(TimeSpan.FromSeconds(2));
    run = await client.GetAsync<SimulationRun>($"/v1/agents/simulation-runs/{run.Id}");
}

Console.WriteLine($"{run.State}: {run.Passed} passed, {run.Failed} failed of {run.Cases}");
foreach (var conversation in run.Conversations ?? [])
{
    if (conversation.Passed == false)
    {
        Console.WriteLine($"{conversation.Scenario} -> {conversation.Verdict}");
    }
}
```

```rust label="Rust"
use std::time::Duration;
use vision_agents::types::SimulationRunState;

loop {
    run = client.get_simulation_run(&run.id).await?;
    if run.state != SimulationRunState::Running {
        break;
    }
    tokio::time::sleep(Duration::from_secs(2)).await;
}

println!("{}: {} passed, {} failed of {}", run.state, run.passed, run.failed, run.cases);
for conversation in run.conversations.iter().flatten() {
    if conversation.passed == Some(false) {
        println!("{} -> {}", conversation.scenario, conversation.verdict.as_deref().unwrap_or_default());
    }
}
```

A run is `running`, then `passed`, `failed`, `errored` or `cancelled`. It passes only if every conversation in it did; a conversation that never reached a ruling leaves the run `errored` rather than `failed`. Each conversation carries its transcript, the judge's verdict and a score, and the session id of the call that held it, so a failing conversation can be replayed in the dashboard.

| Endpoint                                      | Purpose                                                                         |
| --------------------------------------------- | ------------------------------------------------------------------------------- |
| `POST /v1/agents/simulations`                 | Create a simulation. `GET` lists, `PUT` and `DELETE /{id}` edit and remove one. |
| `POST /v1/agents/simulations/{id}/run`        | Start a run.                                                                    |
| `GET /v1/agents/simulation-runs`              | Runs, newest first. `GET /{id}` reads one.                                      |
| `POST /v1/agents/simulation-runs/{id}/cancel` | Stop a run in progress.                                                         |

## Simulation fields

| Field                                      | Meaning                                                                                                                                                                                                                      |
| ------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name`                                     | What you call the simulation.                                                                                                                                                                                                |
| `config_id`                                | The agent being tested.                                                                                                                                                                                                      |
| `scenario`                                 | What the caller wants, in your own words and over as many turns as it takes. A brief, not a script: it can depend on what the agent says back.                                                                               |
| `assertion`                                | What has to be true at the end for the run to pass.                                                                                                                                                                          |
| `mode`                                     | `text` hands the agent the words and tests everything between hearing and answering. `audio` generates speech and runs the whole voice pipeline, so what is judged is what a caller would actually hear. Defaults to `text`. |
| `variations`                               | How many ways of asking the same thing one run tries, up to ten. The scenario as written is always the first. Defaults to 1.                                                                                                 |
| `max_turns`                                | How many times the caller may speak, up to 200. It stops a caller that never decides it is finished. Defaults to 12.                                                                                                         |
| `caller_target`                            | The model playing the caller. Defaults to the `llm-scenario-runner` route.                                                                                                                                                   |
| `judge_target`                             | The model ruling on the conversation. Defaults to the `llm-judge` route.                                                                                                                                                     |
| `caller_stt`, `caller_tts`, `caller_voice` | How the caller hears and speaks, for audio simulations.                                                                                                                                                                      |
| `tags`                                     | Your own labels, as string keys and values, that the run's model usage is recorded under.                                                                                                                                    |

---

For the most recent version of this documentation, visit [https://getstream.io/agents/docs/python/simulations-and-evals/](https://getstream.io/agents/docs/python/simulations-and-evals/).