latent Dirichlet allocation
In plain wordsA popular topic-finding method that assumes every document is a blend of a few themes and every theme favors certain words, then works backward from the words to find both.
A Bayesian topic modeltopic model, n.: A method that reads a large pile of documents and discovers the themes running through them, describing each theme by its typical words and each document by its mix of themes. for text and other count data, built on a simple story of how documents could be generated at random from a small set of shared topics. It assumes K topics, each a distribution over the vocabulary drawn from a Dirichlet prior; each document draws its own topic proportions from a second Dirichlet prior, and each word is produced by first picking a topic from those proportions and then a word from that topic. Given only the words, inference recovers the topics, each document's topic mixture and each word's topic assignment. Exact posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces. inference is intractable, so LDA is fitted by variational Bayes (as in the original paper), by collapsed Gibbs sampling, a form of Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability., or by online stochastic variational methods for streaming corpora. The Dirichlet priors, which favor sparse mixtures when their concentration parameters are small, give LDA a proper generative model for unseen documents, which probabilistic latent semantic analysis lacked. LDA treats each document as a bag of words, ignoring word order, and needs the number of topics in advance.
We fit LDA with 50 topics to the support forum and used each post's topic proportions as features.
Origin
Introduced by David Blei, Andrew Ng and Michael Jordan in a paper presented at the NIPS conference in December 2001 (proceedings 2002) and in the 2003 Journal of Machine Learning Research article 'Latent Dirichlet Allocation'. Jonathan Pritchard, Matthew Stephens and Peter Donnelly had published an essentially identical model for inferring population structure from genotype data in 2000. The collapsed Gibbs sampler of Thomas Griffiths and Mark Steyvers (2004) made LDA easy to implement.
Usage note
Not to be confused with linear discriminant analysislinear discriminant analysis, n.: A statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few., a supervised method also abbreviated LDA.
Related
- topic modeltopic model, n.: A method that reads a large pile of documents and discovers the themes running through them, describing each theme by its typical words and each document by its mix of themes.
- latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together.
- Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
- Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability.
- natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence.
- clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size.
- linear discriminant analysislinear discriminant analysis, n.: A statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few.
Sources
- Blei, D. M., Ng, A. Y. & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research 3: 993–1022. jmlr.org/papers/v3/blei03a.html
- Pritchard, J. K., Stephens, M. & Donnelly, P. (2000). Inference of Population Structure Using Multilocus Genotype Data. Genetics 155: 945–959.
- Griffiths, T. L. & Steyvers, M. (2004). Finding Scientific Topics. Proceedings of the National Academy of Sciences 101 (suppl. 1).
Mentioned in
Andrew NgAndrew Ng, prop. n.: A British-born American computer scientist whose online courses have taught machine learning to millions; he also co-founded Google's deep learning team, Google Brain, and the online-course company Coursera., linear discriminant analysislinear discriminant analysis, n.: A statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few., topic modeltopic model, n.: A method that reads a large pile of documents and discovers the themes running through them, describing each theme by its typical words and each document by its mix of themes., latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together., Michael I. JordanMichael I. Jordan, prop. n.: An American computer scientist and statistician who did much to bring probability and statistics into machine learning, and who trained many of the field's leaders., variational inferencevariational inference, n.: A way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem.