Contract requirements
AI Models & Assistants / Knowledge library
Evaluator-Optimizer Loop
Alternates a generator with an evaluator that supplies criteria-based feedback
Why learn Evaluator-Optimizer Loop?
It improves outputs when evaluation criteria are clearer than a one-shot generation path.
The problem it solves
It turns quality feedback into repeated targeted revisions rather than a final unacted-on score.
Where it came from
One component produces or revises an artifact while another evaluates it against explicit criteria. The loop stops when quality passes or a budget is reached; weak or correlated evaluators can reward polished but incorrect work.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Define Evaluator-Optimizer Loop and identify its place in an agent architecture.
- Choose a task where Evaluator-Optimizer Loop improves an agent system.
- Diagnose one failure caused by misusing Evaluator-Optimizer Loop.
- Interpret a trace or state transition involving Evaluator-Optimizer Loop.
- Explain one operational tradeoff of Evaluator-Optimizer Loop.
Documentation and sources
- Building Effective AI Agents (www.anthropic.com)
Connect the ideas
Explore next
Real World Example
Legal Contract Clause Refinement
Legal departments drafting high-volume standardized contracts.
The business problem
Initial drafts often contain inconsistent terminology or missing clauses that require multiple manual reviews, causing significant delays in deal finalization.
How this topic solves it
An Evaluator-Optimizer loop automates the iterative refinement of contract clauses by using a generator LLM to draft content and an evaluator LLM to critique the draft against a predefined legal checklist, looping until the criteria are met.
- Define the specific legal criteria or checklist that a compliant contract clause must satisfy.
- Configure the generator LLM to produce an initial draft based on the contract requirements.
- Configure the evaluator LLM to compare the draft against the checklist and output specific feedback.
- Implement a loop that passes the evaluator's feedback back to the generator for revision.
- Set a termination condition, such as a maximum number of iterations or a successful evaluation pass.
A cyclical process where a generator creates content and an evaluator provides feedback until the criteria are satisfied.
Drafts the clause
Checks against criteria
Check termination condition
Refined clause
- Input Request → triggers → Generator
- Generator → sends draft → Evaluator
- Evaluator → provides feedback → Pass/Fail
- Pass/Fail → if fail, re-draft → Generator
- Pass/Fail → if pass, finalize → Final Output
Expected business outcome
Higher quality, consistent contract drafts that require minimal human intervention before final review.
Limitations and checks
The loop can increase latency and cost; check this by monitoring the average number of iterations required to reach a 'pass' state and ensuring it stays within acceptable time bounds.