crowdsourcing
In plain wordsGetting many people over the internet to each do a small piece of a big job, such as labeling pictures for a few cents apiece, with anyone free to join in.
The practice of getting work done by posting it online for anyone to take up, usually split into small, independent microtasks that a large, mostly anonymous pool of contributors completes for per-task pay or as volunteers. In AI it became the standard way to build labeled datasets cheaply in the late 2000s: platforms such as Amazon Mechanical TurkAmazon Mechanical Turk, prop. n.: A website Amazon opened in 2005 that acts as a hiring hall for tiny online jobs, such as labeling images for AI, which registered workers pick up and finish for a few cents each. paid workers per task to classify images, transcribe audio, judge search relevance or write short texts, and ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom.'s millions of images were labeled this way. Because individual crowdworkers are unvetted and paid by the piece, projects add redundancy (several workers per item), hidden gold-standard checks, qualification tests and statistical aggregation of answers. Studies found that pooled non-expert labels can match expert quality on many language tasks, but crowdsourcing copes poorly with specialized judgment, which has moved much AI data work to vetted experts.
We crowdsourced five judgments per sentence and kept the majority label.
Origin
Coined in 2006 by Wired editors Jeff Howe and Mark Robinson, blending 'crowd' and 'outsourcing'; Howe's article 'The Rise of Crowdsourcing' (Wired, June 2006) popularized it. Rion Snow and colleagues' 2008 study 'Cheap and Fast — But is it Good?' helped make crowdsourced annotation routine in natural language processing.
Usage note
Low pay, often a few dollars an hour, and the absence of employment protections are long-standing criticisms; see ghost workghost work, n.: The hidden human labor behind AI: people, often poorly paid and far away, who label data, check outputs and handle cases the software cannot, while the product appears fully automatic.. Freely given contributions, as in Wikipedia or citizen science, are also called crowdsourcing but rarely supply paid AI labels.
Related
- Amazon Mechanical TurkAmazon Mechanical Turk, prop. n.: A website Amazon opened in 2005 that acts as a hiring hall for tiny online jobs, such as labeling images for AI, which registered workers pick up and finish for a few cents each.
- data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them.
- annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it.
- ghost workghost work, n.: The hidden human labor behind AI: people, often poorly paid and far away, who label data, check outputs and handle cases the software cannot, while the product appears fully automatic.
- inter-annotator agreementinter-annotator agreement, n.: A measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable.
- ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom.
Sources
- Howe, J. (2006). The Rise of Crowdsourcing. Wired 14(6).
- Snow, R., O'Connor, B., Jurafsky, D. & Ng, A. Y. (2008). Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. EMNLP 2008.
Mentioned in
annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it., data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them., Fei-Fei LiFei-Fei Li, prop. n.: A Chinese-American computer scientist whose ImageNet project, millions of hand-labeled photos, gave deep learning the proving ground that launched its boom; she now champions 'human-centered' AI., ghost workghost work, n.: The hidden human labor behind AI: people, often poorly paid and far away, who label data, check outputs and handle cases the software cannot, while the product appears fully automatic., human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well., ImageNetImageNet, prop. n.: A huge collection of more than 14 million labeled photos, sorted into object categories, whose yearly recognition contest from 2010 to 2017 sparked the deep learning boom., Amazon Mechanical TurkAmazon Mechanical Turk, prop. n.: A website Amazon opened in 2005 that acts as a hiring hall for tiny online jobs, such as labeling images for AI, which registered workers pick up and finish for a few cents each., Mechanical TurkMechanical Turk, prop. n.: A famous fake chess-playing machine from 1770: a robed figure that seemed to play chess by itself but hid a human player inside. Its name now stands for 'AI' that is really people working behind the scenes.