feature selection

A step in model building that narrows a pool of candidate inputs to the variables that actually help predict the target, dropping those that carry no signal or merely repeat what others supply. Because the survivors are original measurements, not new combinations as in projection-based dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from., the model stays interpretable. Methods fall into three families. Filter methods rank features by a statistic computed independently of any model, such as correlation, mutual informationmutual information, n.: How much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated. or a chi-squared test. Wrapper methods search over subsets by repeatedly training a model and measuring validation performance, as in forward selection, backward elimination and recursive feature elimination; they are accurate but expensive and can overfit the validation data. Embedded methods select features as part of training, as with the L1 penalty of the lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model. or tree-based feature importancefeature importance, n.: A score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect.. Feature selection reduces overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., computation and data-collection costs and counters the curse of dimensionalitycurse of dimensionality, n.: The problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much.. It must be performed inside each cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn. fold: selecting features on the full dataset before splitting is a classic form of data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints..

Recursive feature elimination cut the model from 340 inputs to 25 with no loss in validation AUC.