Skip to content
Last Dictionary
Plane
Evaluation & BenchmarksLanguage Models & TransformersTraining & Post-training

IFEval

proper nounAlso: Instruction-Following Evaluation, Instruction-Following Eval

In plain wordsA test of whether AI models obey precise formatting and content rules — such as 'use exactly three bullet points' or 'answer in capital letters' — that a computer program can check automatically.

  1. An instruction-following benchmark limited to constraints that a short program can verify — word and paragraph counts, required or banned keywords, letter case, punctuation, JSON output, a fixed number of bullet points and the like, 25 kinds in all, which its authors call 'verifiable instructions'. Its roughly 500 prompts each impose at least one such constraint, as when a story must be written entirely in lowercase and end with a specified phrase. Scores are reported as prompt-level accuracy (every instruction in a prompt followed) and instruction-level accuracy, each under a strict check and a 'loose' check that tolerates superficial variations such as markdown. Because it needs no human or model judge, IFEval gives an objective, reproducible measure of instruction followinginstruction following, n.: An AI's ability to do exactly what it is asked, including details such as length, format, language and things to avoid, rather than something merely related.; it was adopted by Hugging Face's Open LLM Leaderboard in 2024 and its style of checkable constraint became a source of rewards in reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds..

    Our RLVR run raised IFEval strict prompt accuracy from 78% to 86% without hurting chat quality.

Origin

Introduced by Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou and Le Hou, researchers at Google, in November 2023.

Usage note

IFEval tests surface compliance with mechanical constraints, not whether a model understands and carries out the substance of a request.

Related

See also
  • instruction followinginstruction following, n.: An AI's ability to do exactly what it is asked, including details such as length, format, language and things to avoid, rather than something merely related.
  • reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
  • benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
  • structured outputsstructured outputs, n.: Getting an AI model to answer in an exact, machine-readable format — like a form with fixed fields — so other software can use the answer without guesswork.
  • LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.

Sources

  1. Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D. & Hou, L. (2023). Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911. arxiv.org/abs/2311.07911