robustness
In plain wordsAn AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it.
- 1
Machine learning
The extent to which a model's performance holds up when its inputs differ from those it was trained and tested on — under distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested., noise, corruption, rare events or changed conditions. It is usually measured as the gap between in-distribution and shifted performance, using benchmarks of natural shifts (new hospitals, later years, new image styles) or synthetic corruptions. Sources of fragility include reliance on spurious correlations and overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.; remedies include data augmentation, more diverse training data, domain-generalization methods and large-scale pretraining. Resistance to deliberately crafted inputs is the separate topic of adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up..
The model's accuracy dropped twelve points on scans from the new hospital — a robustness problem, not a bug.
- 2
AI safety
The reliability of a system's safety-relevant behavior — refusals, honesty, adherence to its specificationmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict. and intended goals — under novel situations, optimization pressure and adversarial users, including jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. and prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger.. In 'Unsolved Problems in ML Safety' (2021), robustness ('withstanding hazards') is one of four pillars, with monitoring, alignment and systemic safety. Researchers distinguish capability robustness from goal robustness: a system that stays competent off-distribution but pursues the wrong goal shows goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine., which is more dangerous than simply failing.
- 3
Statistics
The insensitivity of an estimator or test to departures from its assumptions, such as outliers or non-normal errors. Robust statistics — the median rather than the mean, for example — remain reliable when a fraction of the data is corrupted. This is the oldest technical sense.
Origin
The statistician George E. P. Box introduced 'robustness' in 1953 for tests insensitive to violated assumptions, and John Tukey and Peter Huber developed robust statistics in the 1960s. In machine learning the word spread with studies of distribution shift and adversarial examples in the 2010s; 'Concrete Problems in AI Safety' (2016) listed robustness to distributional shift among its five problems.
Usage note
Always ask 'robust to what?' — distribution shift, noise, adversaries or optimization pressure. Robustness of one kind does not imply another; adversarially trained image classifiers, for example, can be less accurate on clean data.
Related
- adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
- distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
- out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
- goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine.
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- certified robustnesscertified robustness, n.: A mathematical guarantee — not just a test result — that no change to an input within a set limit can alter a model's answer.
- AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable.
Sources
- Box, G. E. P. (1953). Non-normality and tests on variances. Biometrika 40(3/4), 318–335.
- Hendrycks, D., Carlini, N., Schulman, J. & Steinhardt, J. (2021). Unsolved Problems in ML Safety. arXiv:2109.13916. arxiv.org/abs/2109.13916
Mentioned in
AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable., adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up., adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them., distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested., HELMHELM, prop. n.: A large, open testing framework from Stanford that grades many language models on many kinds of tasks and qualities — not only accuracy but also fairness, toxicity and efficiency., out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable., prompt sensitivityprompt sensitivity, n.: The way an AI's answers, and its test scores, can change a lot after small changes to a prompt that should not matter — a different word, an extra space or examples in another order., trustworthy AItrustworthy AI, n.: AI that deserves people's trust because it works reliably, obeys the law, treats people fairly and can be checked, rather than AI that people merely happen to trust., domain randomizationdomain randomization, n.: Training a robot's AI in a simulation whose colors, lighting, weights, friction and other details are shuffled at random on every run, so that the real world holds few surprises it has not already practiced for., ISO/IEC 22989ISO/IEC 22989, prop. n.: An international standard that fixes agreed meanings for basic AI words — such as AI system, machine learning and trustworthiness — so that other standards, contracts and laws can use the same vocabulary., safetywashingsafetywashing, n.: Making AI work sound as if it improves safety when it mostly just makes AI more capable — like 'greenwashing' for environmental claims., shortcut learningshortcut learning, n.: When an AI system gets right answers for the wrong reasons — noticing which hospital an X-ray came from rather than signs of disease — so it fails once that accidental clue disappears.