data labeling
In plain wordsThe work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them.
The production of labelslabel, n.: The correct answer attached to an example in a dataset — 'cat' on a photo of a cat, 'positive' on an upbeat review — that a learning system is trained to predict. for training and evaluation data as an organized activity, usually at scale and often outsourced. A labeling operation combines task design and guidelines, a workforce (in-house staff, crowdsourcingcrowdsourcing, n.: Getting many people over the internet to each do a small piece of a big job, such as labeling pictures for a few cents apiece, with anyone free to join in. platforms, managed vendors or credentialed experts), tooling, and quality control: gold questions with known answers, redundant labeling with majority vote or statistical aggregation, inter-annotator agreementinter-annotator agreement, n.: A measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable. tracking and audits. Since the early 2020s demand has shifted from simple image and text classification toward skilled work — writing demonstrations, ranking and critiquing model responses, and building reinforcement-learning tasksRL environment, n.: A practice world built for training an AI by trial and error: it hands the AI tasks and tools, lets it act, and scores the results so the AI can learn what works. and grading rubrics — performed by domain experts for post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner.. Costs are cut with active learningactive learning, n.: A way to train a model with fewer labeled examples by letting it choose which unlabeled examples a person should label next, usually the ones it is most unsure about., weak supervisionweak supervision, n.: Training AI with labels that are cheap but imperfect, such as rough rules of thumb, partial tags or noisy guesses, instead of carefully hand-checked answers., model-assisted pre-labeling and fully automated labeling by models.
Labeling the 40,000 radiology images required board-certified readers, not crowdworkers.
Origin
Paid labeling at scale took off with Amazon Mechanical Turk (2005) and projects such as ImageNet, labeled by crowdworkers from 2007. A commercial labeling industry formed in the 2010s around autonomous driving, then large language models, whose training on human feedback created demand for expert labelers.
Usage note
Labeling work is often low-paid and invisible, done through subcontractors in lower-income countries, a concern discussed under ghost workghost work, n.: The hidden human labor behind AI: people, often poorly paid and far away, who label data, check outputs and handle cases the software cannot, while the product appears fully automatic.; safety and content-moderation labeling can expose workers to disturbing material.
Related
- annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it.
- labellabel, n.: The correct answer attached to an example in a dataset — 'cat' on a photo of a cat, 'positive' on an upbeat review — that a learning system is trained to predict.
- crowdsourcingcrowdsourcing, n.: Getting many people over the internet to each do a small piece of a big job, such as labeling pictures for a few cents apiece, with anyone free to join in.
- Amazon Mechanical TurkAmazon Mechanical Turk, prop. n.: A website Amazon opened in 2005 that acts as a hiring hall for tiny online jobs, such as labeling images for AI, which registered workers pick up and finish for a few cents each.
- ghost workghost work, n.: The hidden human labor behind AI: people, often poorly paid and far away, who label data, check outputs and handle cases the software cannot, while the product appears fully automatic.
- active learningactive learning, n.: A way to train a model with fewer labeled examples by letting it choose which unlabeled examples a person should label next, usually the ones it is most unsure about.
- human-in-the-loophuman-in-the-loop, adj.: Describes AI systems in which a person checks, corrects or approves the machine's work at key points, rather than letting it act entirely on its own.
Mentioned in
labellabel, n.: The correct answer attached to an example in a dataset — 'cat' on a photo of a cat, 'positive' on an upbeat review — that a learning system is trained to predict., active learningactive learning, n.: A way to train a model with fewer labeled examples by letting it choose which unlabeled examples a person should label next, usually the ones it is most unsure about., annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it., crowdsourcingcrowdsourcing, n.: Getting many people over the internet to each do a small piece of a big job, such as labeling pictures for a few cents apiece, with anyone free to join in., data-centric AIdata-centric AI, n.: An approach to building AI that focuses on improving the data — fixing labels, adding missing cases, cleaning inconsistencies — rather than endlessly tweaking the model., ghost workghost work, n.: The hidden human labor behind AI: people, often poorly paid and far away, who label data, check outputs and handle cases the software cannot, while the product appears fully automatic., human-in-the-loophuman-in-the-loop, adj.: Describes AI systems in which a person checks, corrects or approves the machine's work at key points, rather than letting it act entirely on its own., preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., Scale AIScale AI, prop. n.: A company that pays large numbers of people to label examples, write model answers and grade chatbot replies, producing the data AI developers train and test on. Meta bought nearly half of it in 2025., Alexandr WangAlexandr Wang, prop. n.: An American entrepreneur who founded Scale AI, a major supplier of human-labeled data for training AI, and became Meta's chief AI officer in 2025., Amazon Mechanical TurkAmazon Mechanical Turk, prop. n.: A website Amazon opened in 2005 that acts as a hiring hall for tiny online jobs, such as labeling images for AI, which registered workers pick up and finish for a few cents each., data flywheeldata flywheel, n.: A cycle in which a product's users generate data that improves its AI, the better AI attracts more users, and those users generate still more data., inter-annotator agreementinter-annotator agreement, n.: A measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable., weak supervisionweak supervision, n.: Training AI with labels that are cheap but imperfect, such as rough rules of thumb, partial tags or noisy guesses, instead of carefully hand-checked answers.