GPQA
In plain wordsA set of very hard multiple-choice science questions written by PhD-level experts, designed so that looking things up online does not help much: capable non-specialists with web access still mostly fail.
A benchmark of 448 multiple-choice questions in biology, physics and chemistry, each written and checked by people who hold or are working toward a PhD in the field, and built to resist lookup ('Google-proof'). Skilled validators from outside a question's field, free to search the web and averaging more than half an hour per question, got only 34% right, against 65% for specialists in the field (74% once errors they later conceded are set aside); at release the best baseline, built on GPT-4GPT-4, prop. n.: OpenAI's 2023 language model, a big step up from the one behind the first ChatGPT. It scored well on many professional and academic exams and could also read images., reached 39%. GPQA Diamond, a 198-question subset that expert validators answered correctly and most non-experts missed, became the standard version in model reports. In September 2024 OpenAI reported 77% for its o1 reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning., above a panel of PhD experts it had recruited, and by 2026 the benchmark was nearing saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.. The authors designed it partly as a testbed for scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., in which non-experts must supervise systems that know more than they do.
The model scores 87% on GPQA Diamond, above the PhD-expert baseline, yet still botches simple lab protocols.
Origin
Introduced by David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael and Samuel R. Bowman, mainly at New York University, on arXiv in November 2023; it was presented at the first Conference on Language Modeling (COLM) in 2024.
Usage note
Beating the expert baseline means outperforming PhD-level people answering under time limits, often outside their exact specialty; it does not show that a model can do research. The authors ask that examples not be posted online in plain text.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge.
- Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%.
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
Sources
- Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J. & Bowman, S. R. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv:2311.12022. arxiv.org/abs/2311.12022
- OpenAI (2024). Learning to Reason with LLMs. openai.com/index/learning-to-reason-with-llms
Mentioned in
Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%., MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge.