# Evaluations
Evaluations let you run a **fixed batch of content** through a [moderation policy](https://getstream.io/moderation/docs/python/configuration/policies/) and see how it is handled. Re-run the same set after you change rules, filters, or LLM settings to see what improved or drifted.
Use them from the dashboard (**Moderation → Configuration → Evaluations**) or from the server-side SDKs. The dashboard is the easiest way to create sets, watch a run, and compare results. The API is the same surface the dashboard calls, so you can automate evaluations from CI or your own tooling.
Evaluations are a **batch re-run** of saved content. They are not the same as **Test Policy** on a policy editor, which is a one-off check of a single sample against the policy you are editing.
## Concepts
| Term | Meaning |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Evaluation set** | A named, saved batch of up to 1000 items. The text is frozen so later runs are comparable. |
| **Run** | One execution of every item in the set against live moderation. Runs are asynchronous: you start one, then poll until it finishes. |
| **Baseline** | The first completed run of a set. Later runs are scored against it (or against seeded production outcomes, see [Replay](#replay-vs-a-single-policy)). |
| **Replay** | A set with no `config_key`. Each item is re-run under the policy it was originally moderated under. |
| **Scored row** | A row that can receive a pass/fail verdict (it has expected labels/action, or it is compared to a previous run). Unscored rows only record what the policy did. |
### Replay vs a single policy
When you create a set you pick how rows are executed:
- **A specific policy.** Every item runs against that policy's current configuration.
- **Replay.** Omit `config_key`. Seeded items keep the policy they originally ran under. Deleted policies are skipped for those rows.
Pasted (hand-written) content has no original policy, so it always needs a policy selected.
### How scoring works
1. The **first completed run** is the baseline. It has no pass/fail metrics of its own.
2. Later runs compare each row's labels and recommended action to the baseline (or, for replay sets seeded from production, to the original production outcome stored on the row).
3. Rows with **no expectations** (empty labels and empty recommended action) are **unscored**. The run still records actual labels and action so you can inspect them, but they are excluded from pass/fail totals.
A row **fails** when labels differ (`label_mismatch`), the recommended action differs (`action_mismatch`), the provider errors (`provider_error`), or the row's policy no longer exists (`policy_not_found`).
### Billing and retention
Each item in a run executes **real moderation calls** (LLM, NLP, image checks, and so on) at standard rates, the same as production traffic.
Each set keeps the **last 10 completed runs**. Starting another after that deletes the oldest run and its per-row results. In-flight runs are not deleted.
## Using the dashboard
1. Open **Moderation → Configuration → Evaluations**.
2. Click **Create evaluation**.
- Give the set a name.
- Choose a policy, or **Replay** to re-run each item under its original policy.
- Sample your newest moderated production content (up to 1000 items), or paste content (one item per line).
3. Open the set and click **Run evaluation**. Confirm the dialog: this spends real moderation usage.
4. When the run completes, compare it to the baseline or to another run. Filter to drifted rows, inspect a row's labels and recommended action, and export results as CSV.
You can also open Evaluations already filtered to a policy from that policy's page.
Creating, running, and deleting evaluations requires full access to the app's moderation configuration privilege. Read-only users can view sets and results.
## API
These methods are **server-side only**. They are not available from client-side SDKs.
Every run is billed as production moderation. Poll the run until `status` is terminal (`completed`, `error`, or `cancelled`). There is no cancel method. A run stuck in `pending`/`running` for more than about 15 minutes is marked `error` so a new run can start.
### Create an evaluation set
`POST /api/v2/moderation/policy_tests/sets`
Provide **either** `rows` **or** `seed`, not both. Names are unique per app.
The example below samples the newest production content for a policy. To paste items instead, pass `rows` (each with a `text` field). To replay each item under its original policy, omit `config_key`.
```python label="Python"
response = client.moderation().create_policy_test_set(
name="Hate-speech sample",
config_key="chat:messaging",
seed={"labels": [], "limit": 100},
)
```
#### Request parameters
| Name | Required | Type | Description |
| ------------ | -------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | yes | string | Display name. Unique within the app. |
| `config_key` | no | string | Policy to run every row against. Omit for replay. |
| `team` | no | string | Team scope for the policy (multi-tenancy). |
| `mode` | no | string | `labels` or `check`. If omitted, the server picks `labels` when the Labels API is enabled for the org, otherwise `check`. |
| `seed` | one of | object | Sample newest production content. Labels-mode sets sample stored label results; check-mode sets sample reviewed AI-text review-queue items. Mutually exclusive with `rows`. |
| `rows` | one of | array | Explicit items, max 1000. Each `text` is required, max 20,000 characters. Mutually exclusive with `seed`. |
`seed` fields:
| Name | Required | Type | Description |
| -------- | -------- | -------- | -------------------------------------------------------------------------------------------------- |
| `limit` | yes | number | How many items to sample, newest first. `1` to `1000`. |
| `labels` | no | string[] | Only sample records that carry any of these labels. Empty (or omitted) samples everything. Max 20. |
Row fields (when using `rows`, and also returned on seeded sets):
| Name | Type | Description |
| -------------------- | -------- | ------------------------------------------------------------------------------------ |
| `text` | string | Content to moderate. |
| `labels` | string[] | Expected labels (optional). |
| `recommended_action` | string | Expected action (optional). |
| `policy` | string | Original policy key. Used on replay when `config_key` is empty. |
| `content_type` | string | When `username`, labels-mode runs dispatch through the username classification path. |
#### Response
| Name | Type | Description |
| ----- | ------ | ---------------- |
| `set` | object | The created set. |
If seeding matches no production data, the request fails. Broaden `labels` or moderate more content first.
### List evaluation sets
`GET /api/v2/moderation/policy_tests/sets`
Each set in `sets` may include `last_run` so you can show last-run status and pass rate without a second request.
```python label="Python"
response = client.moderation().list_policy_test_sets(limit=50)
```
#### Request parameters
| Name | Required | Type | Description |
| -------- | -------- | ------ | ------------------------------------------------- |
| `limit` | no | number | Page size. Default is the store default, max 200. |
| `offset` | no | number | Offset for pagination. |
#### Response
| Name | Type | Description |
| ------ | ----- | ---------------------------- |
| `sets` | array | Evaluation sets for the app. |
### Get an evaluation set
`GET /api/v2/moderation/policy_tests/sets/{id}`
Returns `set` (including `rows`), `recent_runs` (newest first, capped at 10), and `baseline_run_id` once the first run has completed.
```python label="Python"
response = client.moderation().get_policy_test_set("pts_...")
```
#### Request parameters
| Name | Required | Type | Description |
| ---- | -------- | ------ | ------------------------------- |
| `id` | yes | string | The evaluation set to retrieve. |
#### Response
| Name | Type | Description |
| ----------------- | ------ | ------------------------------------------------------------ |
| `set` | object | The set, including rows. |
| `recent_runs` | array | Retained run history, newest first. |
| `baseline_run_id` | string | Earliest completed run. Absent until the first run finishes. |
### Delete an evaluation set
`DELETE /api/v2/moderation/policy_tests/sets/{id}`
Cascades to runs and results. Fails if a run is still `pending` or `running`.
```python label="Python"
client.moderation().delete_policy_test_set("pts_...")
```
#### Request parameters
| Name | Required | Type | Description |
| ---- | -------- | ------ | ----------------------------- |
| `id` | yes | string | The evaluation set to delete. |
### Start a run
`POST /api/v2/moderation/policy_tests/sets/{id}/runs`
The body is empty. Only one run may be in progress per set. When a new run is created, the oldest terminal runs beyond the 10-run cap are deleted.
```python label="Python"
response = client.moderation().start_policy_test_run("pts_...")
```
#### Request parameters
| Name | Required | Type | Description |
| ---- | -------- | ------ | ------------------------------ |
| `id` | yes | string | The evaluation set to execute. |
#### Response
| Name | Type | Description |
| ----- | ------ | ------------------------------------------------ |
| `run` | object | The new run, typically with `status: "pending"`. |
### Get a run (and results)
`GET /api/v2/moderation/policy_tests/runs/{id}`
Poll this while `status` is `pending` or `running`. `rows_completed` / `rows_total` track progress. Per-row `results` are **omitted until the run is terminal**, so polling stays cheap.
```python label="Python"
response = client.moderation().get_policy_test_run("ptr_...")
```
#### Request parameters
| Name | Required | Type | Description |
| ---- | -------- | ------ | -------------------- |
| `id` | yes | string | The run to retrieve. |
#### Response
| Name | Type | Description |
| --------- | ------ | -------------------------------------------------------- |
| `run` | object | The run, including progress and metrics when available. |
| `results` | array | Per-row results. Present only once the run has finished. |
#### Run object
| Field | Type | Description |
| ------------------- | ------ | -------------------------------------------------------------------------- |
| `id` | string | Run id. |
| `set_id` | string | Parent set. |
| `status` | string | `pending`, `running`, `completed`, `error`, or `cancelled`. |
| `config_key` | string | Policy snapshot used for this run. |
| `config_updated_at` | string | When that policy was last updated, if it exists. |
| `rows_total` | number | Items in the set. |
| `rows_completed` | number | Items processed so far. |
| `metrics` | object | Set on non-baseline completed runs. Pass/fail totals plus per-label drift. |
| `error_message` | string | Present when `status` is `error`. |
| `triggered_by` | string | Who started the run. |
| `started_at` | string | When the worker picked up the run. |
| `completed_at` | string | When it reached a terminal status. |
`metrics.totals`:
| Field | Description |
| ---------- | ------------------------------------------- |
| `rows` | All rows. |
| `scored` | Rows included in pass/fail. |
| `unscored` | Rows recorded without a verdict. |
| `passed` | Scored rows that matched the baseline/seed. |
| `failed` | Scored rows that drifted or errored. |
`metrics.by_label` maps each label to `{ same, changed }`. `changed` counts rows where that label was added or dropped versus the baseline. Rows where the label was absent in both runs are not counted.
#### Result object (terminal runs)
| Field | Type | Description |
| ----------------- | -------- | ----------------------------------------------------------------------------- |
| `row_index` | number | Index in the set. |
| `message_text` | string | The item's text. |
| `expected_labels` | string[] | Baseline or seeded labels. |
| `expected_action` | string | Baseline or seeded recommended action. |
| `actual_labels` | string[] | Labels this run produced. |
| `actual_action` | string | Recommended action this run produced. |
| `scored` | boolean | Whether the row received a verdict. |
| `passed` | boolean | `true`/`false` when scored; `null` when not. |
| `failure_reason` | string | `label_mismatch`, `action_mismatch`, `provider_error`, or `policy_not_found`. |
| `severity` | string | AI-text severity from the provider, display only. Not part of pass/fail. |
Compare two runs client-side: fetch both, join on `row_index`.
## Recommended workflow
1. Create a set from recent production traffic for the policy you are about to change.
2. Run it once to capture a baseline (or rely on seeded production outcomes for replay).
3. Change the [policy](https://getstream.io/moderation/docs/python/configuration/policies/), [rules](https://getstream.io/moderation/docs/python/configuration/rules/), or [filters](https://getstream.io/moderation/docs/python/configuration/filters/).
4. Run the same set again and inspect drifted rows before you roll the change out further.
---
This page was last updated at 2026-09-02T15:00:35.051Z.
For the most recent version of this documentation, visit [https://getstream.io/moderation/docs/python/configuration/evaluations/](https://getstream.io/moderation/docs/python/configuration/evaluations/).