one-hot encoding
In plain wordsTurning a category such as 'red', 'green' or 'blue' into a row of zeros with a single one marking which category it is, so that a model can use it as numbers.
A representation of a categorical variable with K possible values as a binary vector of length K in which exactly one element is 1 (the 'hot' position marking the category) and all others are 0. It lets algorithms that require numeric inputs use categories without implying a false order or spacing, as integer codes (label encoding) would. In linear models with an intercept, one category is usually dropped (dummy coding) to avoid perfect collinearity. One-hot vectors are sparse and grow with the number of categories, so high-cardinality variables such as postcodes or product identifiers are often handled instead by target encoding (replacing each category with a smoothed statistic of the outcome, computed out-of-fold to avoid data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints.), feature hashing or learned embeddingsembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.. In deep learning, one-hot vectors select rows of an embedding matrix and represent classification targets: a language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own.'s training target is a one-hot vector over its vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list., compared with the predicted distribution by cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text..
We one-hot encoded the twelve store formats but used a learned embedding for the 40,000 product IDs.
Origin
Borrowed from digital electronics, where a one-hot code has exactly one line 'hot' (high), as in the state encoding of finite-state machines. Statisticians long used the equivalent dummy or indicator variables in regression, discussed by Daniel Suits in 1957; the electronics name spread through neural-network and machine-learning practice.
Usage note
Statistics usually keeps K − 1 columns (dummy coding); machine-learning libraries often keep all K, which is harmless for regularized models and trees.
Related
- feature engineeringfeature engineering, n.: Turning raw data into the inputs a model learns from, using knowledge of the problem; for example, converting a timestamp into 'hour of day' and 'is it a weekend'.
- embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
- categorical distributioncategorical distribution, n.: The probabilities for a single pick among a fixed set of options, like a weighted die. A language model's guess about the next word is one of these, spread over its whole vocabulary.
- cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
- vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list.
- tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this.
Sources
- Suits, D. B. (1957). Use of Dummy Variables in Regression Equations. Journal of the American Statistical Association 52(280).
Mentioned in
embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance., CatBoostCatBoost, prop. n.: A free gradient-boosting library from the tech company Yandex, known for handling categories such as city names or product types directly, without first converting them into numbers by hand., feature engineeringfeature engineering, n.: Turning raw data into the inputs a model learns from, using knowledge of the problem; for example, converting a timestamp into 'hour of day' and 'is it a weekend'., tabular datatabular data, n.: Data arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this.