document understanding
In plain wordsUsing AI to read documents such as invoices, forms, contracts and scientific papers the way a person does, taking in layout, tables and charts as well as words, to pull out the information needed.
The automatic extraction of structure and meaning from documents whose information is carried by layout and visual elements as well as by text: forms, invoices, receipts, contracts, scientific papers, slides and reports. Component tasks include layout analysis (finding titles, paragraphs, figures and tables, and their reading order), table structure recognition, key information extraction (for example invoice number, date and total), document classification and document question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words.. Pipelines first ran optical character recognitionoptical character recognition, n.: Software that picks out the letters and words in a picture — a road sign, a handwritten note, a page of an old book — and turns them into real text a computer can copy, search or read aloud. and then models combining each word's text with its position on the page and the page image (LayoutLM, 2019, and its successors). OCR-free models such as Donut (2021) read pixels directly, and since 2023 multimodal language modelsmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type. parse whole pages into text, Markdown or JSON and answer questions about charts and tables. Benchmarks include FUNSD and CORD (forms and receipts), DocVQA, ChartQA and DocLayNet. Document understanding feeds retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from. systems, where mistakes in tables or reading order quietly degrade answers.
The parser gets the line items right, but tables that span two pages still come out as separate fragments.
Origin
Grew out of document image analysis, which has had its own conference, the International Conference on Document Analysis and Recognition (ICDAR), since 1991. The modern multimodal approach began with Yiheng Xu and colleagues' LayoutLM at Microsoft Research Asia (arXiv December 2019), and the commercial label 'document AI' spread from about 2020.
Usage note
Distinct from optical character recognitionoptical character recognition, n.: Software that picks out the letters and words in a picture — a road sign, a handwritten note, a page of an old book — and turns them into real text a computer can copy, search or read aloud., which only transcribes characters, and from text-only natural language processingnatural language processing, n.: The field that teaches computers to handle human language — to read, interpret, translate, summarize and produce text and speech. It is one of the oldest and largest areas of artificial intelligence., which ignores layout.
Related
- optical character recognitionoptical character recognition, n.: Software that picks out the letters and words in a picture — a road sign, a handwritten note, a page of an old book — and turns them into real text a computer can copy, search or read aloud.
- visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words.
- multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type.
- retrieval-augmented generationretrieval-augmented generation, n.: Giving an AI an open-book exam: before answering, it looks up relevant passages in a set of documents and bases its reply on them, often citing where each point came from.
- chunkingchunking, n.: Cutting long documents into smaller pieces before storing them for AI search, so the system can find and hand the model just the relevant part instead of a whole manual.
- structured outputsstructured outputs, n.: Getting an AI model to answer in an exact, machine-readable format — like a form with fixed fields — so other software can use the answer without guesswork.
Sources
- Xu, Y., Li, M., Cui, L., Huang, S., Wei, F. & Zhou, M. (2020). LayoutLM: Pre-training of Text and Layout for Document Image Understanding. KDD 2020.
- Mathew, M., Karatzas, D. & Jawahar, C. V. (2021). DocVQA: A Dataset for VQA on Document Images. WACV 2021.
Mentioned in
optical character recognitionoptical character recognition, n.: Software that picks out the letters and words in a picture — a road sign, a handwritten note, a page of an old book — and turns them into real text a computer can copy, search or read aloud., information extractioninformation extraction, n.: Automatically pulling structured facts out of ordinary text — who did what, when, where and for how much — and putting them into a table or database., robotic process automationrobotic process automation, n.: Software 'robots' that copy the clicks and typing a person would do in office computer programs — such as moving data between forms — to automate repetitive paperwork., vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words.