data mining
In plain wordsSearching large collections of data with computers to discover useful patterns, such as products often bought together or signs that a transaction may be fraudulent.
Automated search of large datasets for regularities worth acting on, such as items bought together, groups of similar customers, rules that predict an outcome or records that do not fit. Since Usama Fayyad and colleagues' 1996 framing, it is usually treated as the analysis stage of knowledge discovery in databases (KDD), a larger process that also selects, cleans and transforms the data beforehand and interprets the results afterward. Standard techniques include classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled., regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known., clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size., association rule learningassociation rule learning, n.: Finding 'if this, then often that' patterns in large records of transactions, such as noticing that shoppers who buy pasta and tomatoes often buy parmesan too. (market-basket analysis), sequence mining and anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network.. Growing out of database research as well as statistics and machine learningmachine learning, n.: A way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats., the field emphasized algorithms that scale to data too large for memory. In law, text and data mining denotes automated analysis of works to extract information, the activity covered by copyright exceptions that matter for AI training (text and data mining exceptiontext and data mining exception, n.: A rule in many countries' copyright laws that lets people use computers to copy and analyze large amounts of text, images or data without asking permission — sometimes only for research, sometimes unless owners opt out.).
Data mining of five years of maintenance logs revealed which part failures tended to occur together.
Origin
Statisticians and economists long used 'data mining', 'data dredging' and 'data fishing' pejoratively for searching data until something significant turns up. In 1989 Gregory Piatetsky-Shapiro coined 'knowledge discovery in databases' for the first KDD workshop, and 'data mining' appeared with positive connotations in the database community around 1990. Usama Fayyad, Piatetsky-Shapiro and Padhraic Smyth's 1996 overview framed data mining as a step within KDD; the workshops grew into an annual conference (from 1995), later run by ACM SIGKDD.
Usage note
The pejorative sense survives in statistics: searching many hypotheses without correction yields spurious findings. As a label the term is less fashionable now, its territory largely absorbed by data sciencedata science, n.: The practice of collecting, cleaning, analyzing and interpreting data, using statistics, programming and machine learning, to answer questions and guide decisions. and machine learningmachine learning, n.: A way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats..
Related
- data sciencedata science, n.: The practice of collecting, cleaning, analyzing and interpreting data, using statistics, programming and machine learning, to answer questions and guide decisions.
- association rule learningassociation rule learning, n.: Finding 'if this, then often that' patterns in large records of transactions, such as noticing that shoppers who buy pasta and tomatoes often buy parmesan too.
- clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size.
- anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network.
- big databig data, n.: Data that piles up too fast, in too many forms or in too great a quantity for one ordinary computer to store and study, and the methods invented to cope with it.
- text and data mining exceptiontext and data mining exception, n.: A rule in many countries' copyright laws that lets people use computers to copy and analyze large amounts of text, images or data without asking permission — sometimes only for research, sometimes unless owners opt out.
- machine learningmachine learning, n.: A way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats.
Sources
- Fayyad, U., Piatetsky-Shapiro, G. & Smyth, P. (1996). From Data Mining to Knowledge Discovery in Databases. AI Magazine, 17(3), 37–54.
Mentioned in
big databig data, n.: Data that piles up too fast, in too many forms or in too great a quantity for one ordinary computer to store and study, and the methods invented to cope with it., data sciencedata science, n.: The practice of collecting, cleaning, analyzing and interpreting data, using statistics, programming and machine learning, to answer questions and guide decisions., KDDKDD, prop. n.: The leading yearly conference on data mining — finding useful patterns in large datasets — where many widely used machine learning tools were first presented.