explainability
In plain wordsHow well the reasons behind an AI system's outputs can be put into terms that people can understand, check and, if needed, challenge.
The degree to which the grounds for an AI system's outputs can be made clear to a particular audience, whether because the system is transparent by design or because a separate method generates explanations of it. Frameworks draw finer lines: the US NIST AI Risk Management Framework (2023, §3.5) uses 'explainable' for accounts of how a system works and 'interpretable' for what its outputs mean in the context of its purpose, while the international terminology standard ISO/IEC 22989 treats explainability as a system's capacity to convey the main factors behind its results in humanly understandable form. Explanations are judged by faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given. to the model's actual computation, comprehensibility to the intended audience, and usefulness for a task such as debugging, auditing or contesting a decision. Explainability is therefore audience-relative: an explanation adequate for a data scientist may be useless to a loan applicant.
Explainability was a procurement requirement, so the vendor shipped feature attributions with every risk score.
Origin
Used of expert systems by the 1980s. It became a policy keyword through the DARPA XAI program (2016) and the European Commission High-Level Expert Group's 'Ethics Guidelines for Trustworthy AI' (2019), which listed explicability among four ethical principles and explainability as part of its transparency requirement.
Usage note
Whether explainability and interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box. differ is unsettled. Distinguish both from AI transparencyAI transparency, n.: Openness about AI: telling people when they are dealing with one, how a system was built and tested, and why it reached a particular decision., which concerns disclosure about a system — its data, purpose, limitations and the fact that AI is in use — rather than reasons for particular outputs.
Related
- explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
- interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
- AI transparencyAI transparency, n.: Openness about AI: telling people when they are dealing with one, how a system was built and tested, and why it reached a particular decision.
- faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given.
- right to explanationright to explanation, n.: The idea that a person affected by a decision made or shaped by an algorithm — a refused loan, a rejected job application — is entitled to a meaningful account of why.
- post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable.
- AI transparencyAI transparency, n.: Openness about AI: telling people when they are dealing with one, how a system was built and tested, and why it reached a particular decision.
Sources
- National Institute of Standards and Technology (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
- ISO/IEC 22989:2022. Information technology — Artificial intelligence — Artificial intelligence concepts and terminology.
Defined in law or standards
ISO/IEC 22989:2022 (defines 'explainability'); NIST AI 100-1 (AI RMF 1.0), §3.5
Mentioned in
AI ethicsAI ethics, n.: The study of what is right and wrong in how AI is built and used: who it may harm, who is responsible when it fails, and which values it should respect., explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., AI transparencyAI transparency, n.: Openness about AI: telling people when they are dealing with one, how a system was built and tested, and why it reached a particular decision., faithfulnessfaithfulness, n.: How accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given., model risk managementmodel risk management, n.: The rules banks and other firms follow to make sure the mathematical models and AI systems they rely on are built correctly, checked independently and used only for what they are fit for., UNESCO Recommendation on the Ethics of AIUNESCO Recommendation on the Ethics of AI, prop. n.: A worldwide set of ethical principles for AI that all of UNESCO's member countries agreed in 2021, covering human rights, fairness, privacy, the environment, gender equality and education., Ethics Guidelines for Trustworthy AIEthics Guidelines for Trustworthy AI, prop. n.: Guidelines written for the European Commission by an expert group in 2019, listing seven requirements AI should meet to be trustworthy; they paved the way for the EU AI Act., ISO/IEC 22989ISO/IEC 22989, prop. n.: An international standard that fixes agreed meanings for basic AI words — such as AI system, machine learning and trustworthiness — so that other standards, contracts and laws can use the same vocabulary.