prompt sensitivity
In plain wordsThe way an AI's answers, and its test scores, can change a lot after small changes to a prompt that should not matter — a different word, an extra space or examples in another order.
The degree to which a model's outputs or measured performance change under semantically irrelevant variations of the promptprompt, n.: The message or instructions you give an AI system — a question, a request, some examples or a document — that it responds to. Clearer prompts generally get better results.: paraphrased instructions, punctuation and whitespace, separators and capitalization, the choice and order of few-shotfew-shot prompting, n.: Showing an AI a handful of examples of a task — inputs with the answers you want — inside the prompt, so it can copy the pattern for a new case. examples, the order of answer options, or the position of key information. Documented effects include large swings in accuracy from formatting alone, recency and majority-label biases introduced by demonstrations, and preferences for particular answer positions in multiple-choice tests. Sensitivity makes single-prompt benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. scores unreliable and model comparisons fragile, so careful evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews. reports results over several prompt variants. Instruction tuning and scale reduce it without eliminating it.
Before trusting the three-point gain, check prompt sensitivity: rerun with five paraphrases.
Origin
Documented as soon as GPT-3 made prompting central: Tony Zhao and colleagues (2021) showed that few-shot accuracy depended on the choice and order of examples and proposed calibrating outputs, and Yao Lu and colleagues (2022) found that ordering alone could move results between near state of the art and near chance. Melanie Sclar and colleagues (2023) measured differences of up to 76 accuracy points from formatting changes with LLaMA-2-13B.
Related
- prompt engineeringprompt engineering, n.: The skill of writing requests to an AI — picking the words, examples, format and background information it receives — so that it reliably gives the result you want, without retraining it.
- few-shot promptingfew-shot prompting, n.: Showing an AI a handful of examples of a task — inputs with the answers you want — inside the prompt, so it can copy the pattern for a new case.
- evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews.
- robustnessrobustness, n.: An AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- construct validityconstruct validity, n.: Whether high marks on a test actually reflect the skill the test is named after — for example, whether a 'reasoning' test rewards reasoning rather than memory or test-taking tricks.
Sources
- Sclar, M., Choi, Y., Tsvetkov, Y. & Suhr, A. (2024). Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting. ICLR 2024. arxiv.org/abs/2310.11324
- Zhao, T. Z., Wallace, E., Feng, S., Klein, D. & Singh, S. (2021). Calibrate Before Use: Improving Few-Shot Performance of Language Models. ICML 2021.
- Lu, Y., Bartolo, M., Moore, A., Riedel, S. & Stenetorp, P. (2022). Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. ACL 2022.
Mentioned in
prompt engineeringprompt engineering, n.: The skill of writing requests to an AI — picking the words, examples, format and background information it receives — so that it reliably gives the result you want, without retraining it., evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically., few-shot promptingfew-shot prompting, n.: Showing an AI a handful of examples of a task — inputs with the answers you want — inside the prompt, so it can copy the pattern for a new case.