Message Understanding Conference
In plain wordsA series of US government-funded contests held from 1987 to 1998 in which research teams built programs to pull key facts, such as who did what, where and when, out of news and military messages.
A series of seven US government-sponsored evaluations of information extractioninformation extraction, n.: Automatically pulling structured facts out of ordinary text — who did what, when, where and for how much — and putting them into a table or database. systems (MUC-1 to MUC-7, 1987–1998). Participating teams received a task definition, training texts and answer keys, then ran their systems blind on new texts, and outputs were scored automatically against templates filled in by people. Domains moved from naval operations messages (MUC-1 and MUC-2) to Latin American terrorism reports (MUC-3 and MUC-4), joint ventures and microelectronics (MUC-5), executive succession (MUC-6) and air crashes and rocket launches (MUC-7). MUC carried the information-retrieval measures precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder. and recallrecall, n.: Of all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer. into NLP evaluation, combining them from MUC-4 (1992) in the F-measure, and MUC-6 (1995) split extraction into reusable subtasks, namely named-entity recognitionnamed-entity recognition, n.: Software that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date., coreference, template elements and scenario templates, that shaped the field's research agenda. Its blind shared-task format became a model for later NLP and AI evaluation campaigns.
Origin
Organized from 1987 by Beth Sundheim and colleagues at the US Naval Ocean Systems Center in San Diego with DARPA support; from MUC-5 (1993) the evaluations ran within DARPA's TIPSTER text program. The term 'named entity' is generally traced to the MUC-6 task (Ralph Grishman and Sundheim, 1996). The series ended with MUC-7, whose conference met in 1998; the Automatic Content Extraction (ACE) program succeeded it from about 1999.
Usage note
Despite the name, each 'conference' was mainly a workshop where teams reported blind-test results. The MUC entity types (person, organization, location, dates, times, money and percentages) remain a common default label set for named-entity recognition.
Related
- information extractioninformation extraction, n.: Automatically pulling structured facts out of ordinary text — who did what, when, where and for how much — and putting them into a table or database.
- named-entity recognitionnamed-entity recognition, n.: Software that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date.
- TRECTREC, prop. n.: A yearly contest, run by a US government standards agency since 1992, in which search-engine researchers test their systems on the same documents and questions, with people judging which results are actually relevant.
- F1 scoreF1 score, n.: A single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.
- precisionprecision, n.: Of all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
Sources
- Grishman, R. & Sundheim, B. (1996). Message Understanding Conference-6: A Brief History. Proc. COLING 1996.
- Chinchor, N. (1992). MUC-4 Evaluation Metrics. Proc. Fourth Message Understanding Conference (MUC-4).
Mentioned in
TRECTREC, prop. n.: A yearly contest, run by a US government standards agency since 1992, in which search-engine researchers test their systems on the same documents and questions, with people judging which results are actually relevant.