survival analysis
In plain wordsMethods for predicting how long until something happens — a patient's relapse, a machine's failure, a customer's cancellation — that properly use the cases where it has not happened yet.
A family of statistical and machine-learning methods for modeling the time until an event occurs, set apart by how they handle censoring: for many subjects the event has not been observed by the end of follow-up (right censoring), so only a lower bound on their time is known, and discarding or mislabeling those cases biases results. Its central quantities are the survival function, the probability of lasting beyond time t, and the hazard function, the instantaneous event rate at t among those still at risk. Classic tools are the nonparametric Kaplan–Meier estimator of the survival curve; the Cox proportional hazards model, which relates covariates to the hazard through a partial likelihood without specifying the baseline hazard's shape; and parametric models such as Weibull regression. Machine-learning extensions include random survival forests, boosting with survival losses and neural models such as DeepSurv, evaluated with the concordance index (C-index) and time-dependent Brier scoresBrier score, n.: A score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better.. Uses span medicine, reliability engineering, customer churn and credit default.
A third of the patients were still relapse-free at the study's end, so we used survival analysis instead of a plain classifier.
Origin
Life tables go back to John Graunt's 1662 analysis of London's bills of mortality. Edward Kaplan and Paul Meier's product-limit estimator (Journal of the American Statistical Association, 1958) and David Cox's 'Regression Models and Life-Tables' (Journal of the Royal Statistical Society, Series B, 1972) founded the modern field; Hemant Ishwaran and colleagues introduced random survival forests in 2008.
Usage note
Treating censored cases as non-events, or dropping them, is a common error when general-purpose classifiers are applied to time-to-event data. 'Survival' is literal in medicine but figurative elsewhere: the event may be a purchase or a repair.
Related
- regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known.
- generalized linear modelgeneralized linear model, n.: A family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve.
- random forestrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree.
- Brier scoreBrier score, n.: A score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better.
- quantile regressionquantile regression, n.: Predicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average.
- gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it.
Sources
- Kaplan, E. L. & Meier, P. (1958). Nonparametric Estimation from Incomplete Observations. Journal of the American Statistical Association, 53(282), 457–481.
- Cox, D. R. (1972). Regression Models and Life-Tables. Journal of the Royal Statistical Society, Series B, 34(2), 187–220.
Mentioned in
generalized linear modelgeneralized linear model, n.: A family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve.