model-agnostic

Of an explanation or analysis method: applicable to any predictive model because it relies only on querying the model with inputs and observing outputs, without access to its parameters, gradients or architecture. LIMELIME, abbr.: A method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example., KernelSHAP, AnchorsAnchors, prop. n.: A method that explains a prediction with a simple if-then rule that locks in the result — for example, if an email contains both 'invoice' and 'urgent', the filter almost always flags it., partial dependencepartial dependence plot, n.: A chart showing how a model's average prediction changes as one input is varied while everything else stays as it is in the data — for example, predicted house price against floor area., individual conditional expectationindividual conditional expectation, n.: A chart that traces, for each example separately, what a model would predict as one input is swept across its range, exposing differences between examples that an overall average would hide., accumulated local effectsaccumulated local effects, n.: A chart of how a model's predictions change with one input, built from small local comparisons so it stays reliable when inputs are related to each other, unlike simpler average-based charts., permutation feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect., global surrogatessurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply. and many counterfactual explanationcounterfactual explanation, n.: An explanation that tells someone the smallest change that would have produced a different outcome — for example, that the application would have been accepted if it had listed two more years of work experience. generators are model-agnostic. This lets them compare different model types on a common footing and explain models reachable only through an API. The price is cost and fidelity: they need many queries, may probe unrealistic inputs, and cannot exploit internal structure. Model-specific methods — gradients for neural networks (integrated gradientsintegrated gradients, n.: A way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way., Grad-CAMGrad-CAM, abbr.: A technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label.), TreeSHAP for tree ensembles, probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere. and activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer. for transformers — are faster or more precise but tied to one model family.

Since the vendor exposed only a scoring API, we were limited to model-agnostic explanations.