big data
In plain wordsData that piles up too fast, in too many forms or in too great a quantity for one ordinary computer to store and study, and the methods invented to cope with it.
Data whose volume, rate of arrival or diversity of form overwhelms a conventional database running on a single machine, together with the distributed technologies and practices built to cope with it. Doug Laney's 'three Vs' (volume, velocity and variety) are the usual characterization, to which later writers added veracity (reliability) and value. Characteristic tools were distributed storage and processing frameworks such as Google's MapReduce (2004) and its open-source counterpart Hadoop, followed by Apache Spark and cloud data warehouses. Big data supplied the web crawls, click logs and image collections that made deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract. effective, and the term was the slogan of a 2010s wave of data-driven business and science, criticized for surveillance, loss of privacy and overconfidence in correlations.
In the big data era we kept everything; now we ask which data actually improve the model.
Origin
In use since the 1990s; John Mashey, chief scientist at Silicon Graphics, is often credited with popularizing it in talks on computing infrastructure. Analyst Doug Laney's 2001 note on data volume, velocity and variety supplied the 'three Vs', and the term peaked as a buzzword in the early 2010s.
Usage note
Largely superseded as a buzzword by 'AI' and 'data science', though the infrastructure it named underpins modern AI. Bigger is not automatically better: sampling bias, poor data qualitydata quality, n.: How well a set of data does its job: whether it is accurate, complete, up to date, consistent and representative — and, for AI training, informative enough to teach useful skills. and leakage scale too.
Related
- data sciencedata science, n.: The practice of collecting, cleaning, analyzing and interpreting data, using statistics, programming and machine learning, to answer questions and guide decisions.
- data miningdata mining, n.: Searching large collections of data with computers to discover useful patterns, such as products often bought together or signs that a transaction may be fraudulent.
- training datatraining data, n.: The examples an AI system learns from. Whatever patterns, skills, gaps and biases are in the training data tend to show up in the finished system.
- data qualitydata quality, n.: How well a set of data does its job: whether it is accurate, complete, up to date, consistent and representative — and, for AI training, informative enough to teach useful skills.
- data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up.
- AI surveillanceAI surveillance, n.: Using AI to watch, identify, track or profile people on a scale no team of humans could manage, for example by scanning camera feeds or analyzing phone and online data.
- data pipelinedata pipeline, n.: The chain of automated steps that collects raw data, cleans and transforms it, and delivers it in a usable form — the factory line that turns messy inputs into training material.
Mentioned in
datadata, n.: Recorded facts, measurements, text, images or sounds that a computer can store and process. AI systems learn their skills from large amounts of data rather than from rules written by hand., data miningdata mining, n.: Searching large collections of data with computers to discover useful patterns, such as products often bought together or signs that a transaction may be fraudulent., data sciencedata science, n.: The practice of collecting, cleaning, analyzing and interpreting data, using statistics, programming and machine learning, to answer questions and guide decisions.