Skip to content
Last Dictionary
Plane
Agents & HarnessesEvaluation & Benchmarks

evaluator–optimizer

nounAlso: evaluator–optimizer loop, generator–critic loop, critic loop

In plain wordsA pairing in which one AI produces a draft and another AI grades it and suggests fixes, repeating the cycle until the work is good enough.

  1. A workflow pattern, named in Anthropic's 2024 guide to building agents, that pairs a generator (the optimizer) with a critic (the evaluator): one model call drafts an output, a second scores it against stated criteria and sends back comments, and the draft is revised until the critic is satisfied or an iteration cap is hit. The critic may be a different model, the same model under a reviewing prompt, a rubric-based LLM judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers., or a programmatic check such as a compiler or test suite; programmatic checks make the loop most dependable. It suits work whose quality can be judged clearly and where each revision measurably helps — code that must pass tests, nuanced translation, research needing several rounds of searching — and fails when the critic shares the generator's blind spots or rewards cosmetic changes. It is the workflow counterpart of agentic 'reflection'.

    We wrapped the translator in an evaluator–optimizer loop; three rounds of critique fixed most of the register errors.

Origin

Named as one of five workflow patterns in 'Building effective agents' by Erik Schluntz and Barry Zhang (Anthropic, December 2024). It generalizes generate-and-critique methods such as Self-Refine (Aman Madaan et al., March 2023) and ReflexionReflexion, prop. n.: A technique in which an AI agent that fails at a task writes itself a short note on what went wrong, then reads that note before trying again — learning from mistakes without retraining. (2023), and the much older generate-and-test strategy of classical AI.

Related

See also
  • ReflexionReflexion, prop. n.: A technique in which an AI agent that fails at a task writes itself a short note on what went wrong, then reads that note before trying again — learning from mistakes without retraining.
  • self-correctionself-correction, n.: An AI model noticing and fixing its own mistakes — by rereading its answer, running a test or checking a source — before it gives a final result.
  • LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
  • agentic workflowagentic workflow, n.: A multi-step process in which AI models draft, check, use tools and revise their work over several rounds, instead of answering in one shot; often the steps are laid out by developers in advance.
  • verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
  • prompt chainingprompt chaining, n.: A way of getting a big job done with a relay of AI requests: each handles one stage and passes its result to the next — pull out the facts, write, polish — with automatic checks between stages.
Contrast with
  • best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.

Sources

  1. Schluntz, E. & Zhang, B. (2024). Building effective agents. Anthropic. anthropic.com/research/building-effective-agents
  2. Madaan, A. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651 (NeurIPS 2023). arxiv.org/abs/2303.17651

Mentioned in

agentic workflowagentic workflow, n.: A multi-step process in which AI models draft, check, use tools and revise their work over several rounds, instead of answering in one shot; often the steps are laid out by developers in advance., prompt chainingprompt chaining, n.: A way of getting a big job done with a relay of AI requests: each handles one stage and passes its result to the next — pull out the facts, write, polish — with automatic checks between stages., self-correctionself-correction, n.: An AI model noticing and fixing its own mistakes — by rereading its answer, running a test or checking a source — before it gives a final result., ReflexionReflexion, prop. n.: A technique in which an AI agent that fails at a task writes itself a short note on what went wrong, then reads that note before trying again — learning from mistakes without retraining.