decision theory

The study of choice under uncertainty, combining probabilityprobability, n.: How likely something is, on a scale from 0 for the impossible to 1 for the certain; rolling a six with a fair die scores one-sixth. AI systems state their predictions, even each next word, as probabilities. (what is likely) with utilityutility function, n.: A way of scoring every possible outcome by how much someone, or some AI, wants it. A rational agent is often modeled as choosing whatever maximizes its expected score. (what is wanted). Normative decision theory prescribes the act with the highest expected utility; its foundations, the axioms of von Neumann and Morgenstern and of Leonard Savage, also imply that a coherent decision-maker's beliefs behave like probabilities. Statistical decision theory, due to Abraham Wald, treats estimation and testing as choices that minimize expected loss, giving the vocabulary of loss, risk and Bayes-optimal decisions used throughout machine learning — for example, setting a classification threshold to minimize expected cost. In AI, decision theory underlies rational-agent design, Markov decision processesMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice. and reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next. and value-of-information calculations in active learningactive learning, n.: A way to train a model with fewer labeled examples by letting it choose which unlabeled examples a person should label next, usually the ones it is most unsure about.; descriptive decision theory studies how people actually choose, including systematic deviations such as loss aversion.

Decision theory says flag the transaction when the expected cost of fraud exceeds the cost of a review.

AI alignmentcontested

A debate over which decision rule an idealized agent should follow when its choices are correlated with predictions of them, as in Newcomb's problem. Causal decision theory (CDT) evaluates acts by their causal consequences, evidential decision theory (EDT) by what they indicate, and functional decision theory (FDT), proposed by Eliezer Yudkowsky and Nate Soares in 2017, by the output of the decision procedure itself. How far this matters for real AI systems is disputed.