vibe check
In plain wordsTrying out a new AI model informally — chatting with it, asking your favorite test questions — to get a gut feeling for how good it is, rather than measuring it with formal tests.
An informal, qualitative assessment of a model formed by interacting with it directly — posing personal test prompts, probing its tone, reasoning and failure modes — rather than by running standardized benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.. Vibe checks became an accepted practice because benchmark scores saturate, leak into training data and reward narrow optimization (see benchmaxxingbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks.), while practitioners care about qualities such as instruction-following, judgment, taste and conversational feel that numbers capture poorly. Aggregated over many users, the same idea underlies preference leaderboards such as LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.. Critics note that vibe checks are unrepeatable, prone to confirmation bias and easily swayed by style — length, formatting and flattery.
The scores look great, but it failed my vibe check: it can't admit when it doesn't know.
Origin
'Vibe check' was general internet slang, widespread by 2019, for gauging the mood of a person or situation. Engineers building on large language models applied it to informal model testing from about 2023, alongside the half-joking phrase 'vibes-based evaluation'.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.
- benchmaxxingbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks.
- pelican on a bicyclepelican on a bicycle, n.: A joke test in which an AI model is asked to draw, by writing code, a pelican riding a bicycle; comparing the results became a popular, lighthearted way to judge each new model.
- leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models.
- vibe codingvibe coding, n.: Programming by conversation: you ask an AI for what you want, try out what it produces and ask again until it works, trusting the code without really reading it.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
Mentioned in
benchmaxxingbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks., evals-driven developmentevals-driven development, n.: Building an AI product by first writing tests that show what good answers look like, then accepting changes to prompts, models or code only when the test scores improve., nerfednerfed, adj.: Gaming slang for something made weaker by an update; applied to AI models that users believe got worse after a provider changed them, often without saying so., pelican on a bicyclepelican on a bicycle, n.: A joke test in which an AI model is asked to draw, by writing code, a pelican riding a bicycle; comparing the results became a popular, lighthearted way to judge each new model.