i.i.d.

Describing a collection of random variablesrandom variable, n.: A quantity whose value depends on chance, such as the number rolled on a die or the next word a chatbot will write. that are mutually independent and share one probability distributionprobability distribution, n.: A complete description of how likely each possible outcome is — for a fair die, one-sixth for each face. A language model produces one of these over its whole vocabulary at every step.. The i.i.d. assumption underlies most classical statistics and learning theory: the law of large numberslaw of large numbers, n.: The mathematical rule that averages of chance events steady out over many repetitions: flip a fair coin a few times and the share of heads can be far from one-half, but over thousands of flips it gets very close., the central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like., PAC learningPAC learning, n.: A mathematical definition of what it means to learn from examples: given enough data, a learner should, with high probability, find a rule that is almost always right. bounds and the use of a held-out test settest set, n.: A set of examples kept completely apart from training and used at the end to measure how well a model works on data it has never seen, like a sealed final exam. to predict future performance all assume that training and test examples are i.i.d. draws from a single distribution. Practice routinely violates it: time series and conversations are dependent, data arrive in clusters, and deployment data drift away from training data (distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested., out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable. inputs). Learning from non-i.i.d. streams is the concern of online learningonline learning, n.: Learning from data that arrive one piece at a time, updating the model after each new example instead of training once on a fixed collection, as a spam filter does while emails stream in. and continual learningcontinual learning, n.: The ability of an AI to keep learning new things over time from a stream of experience, without forgetting what it already knew and without being retrained from scratch., and reinforcement learning breaks the assumption because an agent's actions shape its own data — one reason experience replayexperience replay, n.: Storing an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened. shuffles stored transitions.

Splitting by user rather than by row keeps the test set closer to i.i.d. with respect to future traffic.