HealthBench
In plain wordsAn OpenAI test of how well AI chatbots handle health conversations, grading 5,000 realistic chats against checklists written by 262 doctors from 60 countries.
An open benchmark of model behavior in health care built from 5,000 realistic multi-turn chats, in which the person talking to the model is either a member of the public or a clinician; each chat is graded against its own rubric, written by physicians — 262 of them, with practice experience across 60 countries — for 48,562 distinct criteria overall. Each criterion awards or deducts points for a specific behavior, such as advising emergency care when warranted or avoiding an unsupported claim, and a model-based grader checks each one, so a response's score is the points earned divided by the maximum possible. Conversations fall into seven themes, including emergency referrals, responding under uncertainty, global health and context seeking, and criteria are tagged by axes such as accuracy, completeness, context awareness and instruction following. Subsets include HealthBench Consensus, whose criteria were validated by multiple physicians, and HealthBench Hard. At release, OpenAI reported that physicians could improve on responses from its September 2024 models but no longer on those from its April 2025 models.
On HealthBench Hard the model still misses emergency-referral criteria in about a fifth of conversations.
Origin
Released by OpenAI on 12 May 2025, developed with its network of physicians; the accompanying paper was led by Rahul K. Arora.
Usage note
Rubric scores measure agreement with physicians' expectations in simulated chats, as judged by a model; they are not clinical-outcome data and do not establish safety for medical use. Because a model developer built it, independent replication matters.
Related
- LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
- rubric rewardrubric reward, n.: A training score built from a checklist of specific things a good answer should do, with an AI grader ticking off each one — a way to reward quality where no single right answer exists.
- human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well.
- AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable.
- abstentionabstention, n.: When an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake.
- evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews.
Sources
- OpenAI (2025). Introducing HealthBench. openai.com/index/healthbench
- Arora, R. K. et al. (2025). HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv preprint.