Collection of deterministic test cases
AI Models & Assistants / Knowledge library
Agent Evaluation Loop
Runs representative tasks, scores traces and outcomes, then feeds failures into improvements
Why learn Agent Evaluation Loop?
It detects regressions and shows whether changes improve real task outcomes rather than isolated model responses.
The problem it solves
It gives agent development a repeatable evidence cycle instead of relying on demos and anecdotes.
Where it came from
An evaluation loop samples tasks, records versions and traces, applies deterministic or judged metrics, reviews failures, and reruns after changes. Evaluating only final prose misses tool misuse, excess cost, and unsafe intermediate actions.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Define Agent Evaluation Loop and identify its place in an agent architecture.
- Choose a task where Agent Evaluation Loop improves an agent system.
- Diagnose one failure caused by misusing Agent Evaluation Loop.
- Interpret a trace or state transition involving Agent Evaluation Loop.
- Explain one operational tradeoff of Agent Evaluation Loop.
Documentation and sources
- OpenAI Agents SDK: Testing (openai.github.io)
Connect the ideas
Explore next
Agent TracingEvaluator-Optimizer LoopAI Red TeamingParallel Agent WorkflowAgent Termination Condition
Real World Example
Automated Regression for Logistics Routing Agents
Logistics companies managing automated delivery dispatch agents.
The business problem
Agents occasionally misinterpret complex delivery constraints, leading to costly manual rerouting and delayed shipments.
How this topic solves it
Implement an evaluation loop that runs representative delivery scenarios through a deterministic test harness, scoring agent trace outcomes against expected constraints and feeding failures back into prompt refinement.
- Define a suite of representative delivery scenarios as deterministic ScriptedModel sequences.
- Execute agent runs within the testing harness to capture normalized interaction traces.
- Compare agent tool-use outputs against expected delivery constraints using assertion logic.
- Identify failed traces where agents violated routing rules or failed to call required tools.
- Update system prompts or tool definitions based on the specific failure patterns identified in the traces.
The cycle of testing agent logic against defined scenarios and using the results to improve agent performance.
Executes agent with ScriptedModel
Compares output against constraints
Refines prompts or tool definitions
- Scenario Repository → feeds test data → Agent Test Harness
- Agent Test Harness → generates interaction traces → Trace Analyzer
- Trace Analyzer → identifies failures → Improvement Loop
- Improvement Loop → updates logic → Scenario Repository
Implementation example
This code maps to the business scenario by using ScriptedModel to enforce a specific, repeatable routing decision for the agent, allowing the developer to verify if the agent correctly follows the routing constraints.
from agents.testing import ScriptedModel, assistant_message
# Define a deterministic scenario for the routing agent
model = ScriptedModel([[assistant_message("Route: Zone A to Zone B")]])
# The agent uses this model to simulate a specific routing decision
agent = Agent(name="Router", model=model)
# Run the agent and assert the outcome matches the expected routing constraint
result = await Runner.run(agent, "Optimize route", run_config=RunConfig(tracing_disabled=True))
assert result.final_output == "Route: Zone A to Zone B" # Validates agent logicExpected business outcome
Higher reliability in complex routing decisions and reduced manual intervention for edge-case delivery scenarios.
Limitations and checks
The evaluation loop relies on deterministic scripting; it cannot simulate the stochastic nature of real-world traffic or unpredictable API latency, requiring separate integration testing.