task (machine learning)
In plain wordsThe job an AI model is meant to do, such as translating sentences, spotting spam or answering questions, defined by its inputs, its expected outputs and how success is scored.
A specification of what a model should do: the space of inputs, the space of desired outputs and a criterion for success, usually a metric computed on held-out data. Tasks are grouped into types (classification, regression, translation, summarization, question answering, object detection) and instantiated by datasetsdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system. and benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.. With foundation modelsfoundation model, n.: A very large AI model, built once at great cost from vast amounts of text, images or code gathered mostly from the internet, that many products then reuse as their starting point, from chatbots to coding tools., practitioners distinguish the pretraining task, such as next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers., from the downstream tasks to which a model is adapted by fine-tuning or prompting; multi-task learningmulti-task learning, n.: Training one model to do several related jobs at once, such as spotting objects and judging distance in the same photo, so that practice on each job can sharpen the others. trains on several tasks at once, and zero- and few-shotfew-shot learning, n.: Getting an AI system to handle a new task or recognize a new kind of thing from just a handful of examples, the way a person can learn a new word after hearing it once or twice. settings test tasks never explicitly trained. In NLP a shared task is a community competition with a common dataset and metric. For AI agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done., a task is a goal pursued over many steps in an environment and judged by its outcome, as in long-horizon taskslong-horizon task, n.: A job that takes many steps and a long time — hours or days of work rather than a single answer — so an AI must stay on track and recover from its own mistakes..
On the summarization task the fine-tuned small model beat the general one, but it was useless on everything else.
Origin
Tom Mitchell's 1997 definition of machine learning, framed in terms of a task, a performance measure and experience, made 'task' a basic term. 'Downstream task' spread with transfer learning and pretrained language models in the late 2010s, and agent evaluations increasingly also rate tasks by how long they take skilled humans (METR time horizonMETR time horizon, n.: A way of measuring AI progress by asking how long a task — timed by how long it takes a skilled human — an AI agent can complete with even odds of success.).
Usage note
Success on a benchmark task may not carry over to the real-world job the benchmark is meant to represent; see construct validityconstruct validity, n.: Whether high marks on a test actually reflect the skill the test is named after — for example, whether a 'reasoning' test rewards reasoning rather than memory or test-taking tricks.. Agent researchers use 'task' for whole assignments, such as fixing a bug, rather than single input–output mappings.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- datasetdataset, n.: An organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system.
- multi-task learningmulti-task learning, n.: Training one model to do several related jobs at once, such as spotting objects and judging distance in the same photo, so that practice on each job can sharpen the others.
- transfer learningtransfer learning, n.: Reusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data.
- few-shot learningfew-shot learning, n.: Getting an AI system to handle a new task or recognize a new kind of thing from just a handful of examples, the way a person can learn a new word after hearing it once or twice.
- long-horizon tasklong-horizon task, n.: A job that takes many steps and a long time — hours or days of work rather than a single answer — so an AI must stay on track and recover from its own mistakes.
- METR time horizonMETR time horizon, n.: A way of measuring AI progress by asking how long a task — timed by how long it takes a skilled human — an AI agent can complete with even odds of success.
- evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews.
- capabilitycapability, n.: Something an AI system is able to do, such as writing working code or solving math problems, and how well. Measuring capabilities is how researchers track progress and spot dangerous skills.
Mentioned in
machine learningmachine learning, n.: A way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., learninglearning, n.: Getting better at something through experience. For a machine, it means changing its internal settings based on data so that it performs a task better than before.