Agent monitoring

After simulations are setup the next topic is how to monitor your agents.

The following monitoring is built-in by default.

Agent Sessions

After each session is completed we report on the following things.

Summary & Review

  • session.summary provides a quick summary/recap of what was discussed
  • session.review_score the judge LLM assigns a score. This can be configured in the agent.yaml
  • session.review_notes: problems encountered during this session, we leave notes in the review notes

See title, description and review for when they're written, getting a session to read them, and filter and sort to find the low scoring sessions.

The full conversation is also stored so you can take the time to review it if needed.

Agent Metrics

At the session level we also track many metrics.

  • Costs
  • Response time on various models
  • Token usage. Split by input, cache input and output. Also split by instructions, messages and tool definitions.
  • Tool response times and token usage
  • Skill usage

You can see this data on the session detail screen, or read it from your server with the session's id:

  • sessions.get returns the session with its usage: cost, input, cached input and output tokens and requests, over every model it called. It is written once the session ends. See getting a session.
  • sessions.tokens is tokens and cost per model, with the prompt split into instructions, messages, tool definitions, tool use, images and video. It is answered while the session runs.
  • sessions.timeline is every exchange, with the transcription, model and voice timings kept apart, and each model call's time to first token, duration and tokens. Kept for an hour.
  • sessions.decisions is what the agent decided, in order: why it waited rather than answering, why it read something as not meant for it, why it stopped mid-sentence. Kept for an hour.
Node.js
const tokens = await agent.sessions.tokens(id);
console.log(tokens.input_tokens, tokens.input_parts.tool_definitions);
for (const entry of await agent.sessions.timeline(id))
  console.log(entry.roundtrip_ms);
for (const decision of await agent.sessions.decisions(id))
  console.log(decision.kind, decision.reason);
Python
tokens = await agent.sessions.tokens(session_id)
print(tokens.input_tokens, tokens.input_parts.tool_definitions)
for entry in await agent.sessions.timeline(session_id):
    print(entry.roundtrip_ms)
for decision in await agent.sessions.decisions(session_id):
    print(decision.kind, decision.reason)
Ruby
tokens = agent.sessions.tokens(id)
p tokens.values_at("input_tokens", "input_parts")
agent.sessions.timeline(id).each { |entry| p entry["roundtrip_ms"] }
agent.sessions.decisions(id).each { |decision| p decision.values_at("kind", "reason") }
PHP
$tokens = $agent->sessions->tokens($id);
echo $tokens->inputTokens, ' ', $tokens->inputParts->toolDefinitions, PHP_EOL;
foreach ($agent->sessions->timeline($id) as $entry) {
    echo $entry->roundtripMs, PHP_EOL;
}
foreach ($agent->sessions->decisions($id) as $decision) {
    echo $decision->reason, PHP_EOL;
}
Go
tokens, err := agent.Sessions.Tokens(ctx, id)
if err != nil {
	log.Fatal(err)
}
fmt.Println(tokens.InputTokens, tokens.InputParts.ToolDefinitions)

timeline, err := agent.Sessions.Timeline(ctx, id)
if err != nil {
	log.Fatal(err)
}
for _, entry := range timeline {
	if entry.RoundtripMs != nil {
		fmt.Println(*entry.RoundtripMs)
	}
}

decisions, err := agent.Sessions.Decisions(ctx, id)
if err != nil {
	log.Fatal(err)
}
for _, decision := range decisions {
	fmt.Println(decision.Kind, decision.Reason)
}
.NET
var tokens = await agent.Sessions.TokensAsync(id);
Console.WriteLine($"{tokens.InputTokens} {tokens.InputParts.ToolDefinitions}");
foreach (var entry in await agent.Sessions.TimelineAsync(id)) Console.WriteLine(entry.RoundtripMs);
foreach (var decision in await agent.Sessions.DecisionsAsync(id)) Console.WriteLine($"{decision.Kind} {decision.Reason}");
Rust
let tokens = agent.sessions.tokens(id).await?;
println!("{} {}", tokens.input_tokens, tokens.input_parts.tool_definitions);
for entry in agent.sessions.timeline(id).await? {
    println!("{:?}", entry.roundtrip_ms);
}
for decision in agent.sessions.decisions(id).await? {
    println!("{} {}", decision.kind, decision.reason);
}

Costs are in millionths of a dollar. To cap them, see spend limits. All of these are server-side only.

Aggregated Metrics

  • Cost are broken down by model/ what you're using it on
  • Session/ number of sessions are tracked
  • Tool usage & performance
  • Skill usage and performance