Document automation: extraction, evidence and validation

When you automate incoming document processing — invoices, purchase orders, letters, forms — the question is not "can AI read?" but "will you be able to justify every retained value?". A serious pipeline keeps the provenance of every data point (page, table, checkbox), distinguishes model confidence from documentary evidence, applies explicit business rules and plans for human correction when the rules cannot decide. This article describes the corresponding decision architecture, with a fictional example of a purchase order in two incompatible versions.

Why provenance changes everything

A value extracted without provenance is unverifiable. "Total amount: 4,250.00 €" is worth nothing if nobody can tell which page, which table and which cell that figure comes from. Provenance is not a technical nicety: it is what allows a colleague, an auditor or a control system to replay the extraction and verify it in seconds.

Concretely, every extracted value should carry at least four pieces of information: the source document (file identifier and, if possible, its fingerprint), the page or zone involved, the structure read (table, row, column, text block, checkbox) and the version of the model or extraction module that produced the value. Without the version, there is no way to understand why two runs on the same document produced different results.

This requirement ties into overall case tracking: an extraction is never isolated, it sits within a journey you must be able to trace end to end, as described in our guide on following an administrative case from start to finish. If your documents arrive by email, provenance starts with the attachment and its sender, a point developed in our article on validating data extracted from email attachments.

OCR, types and units: extracting is not interpreting

OCR produces text, not data. "1,250" can be an amount in euros, a quantity or an item number depending on context. The first reliability step is therefore to type every field: date, amount, quantity, reference, free text. Typing triggers mechanical checks: a date must be a valid date, an amount must follow a decimal format, a quantity must be a positive integer or decimal according to your nomenclature.

Units are the second source of silent errors. "12" without a unit can mean 12 units, 12 cases of 6, 12 kilograms or 12 hours. A well-designed pipeline keeps the unit as it appears in the document and normalises it separately: never overwrite "12 cases" with "12" without keeping a trace of the original unit. Same logic for currencies: an amount without a currency is not an amount.

Three design rules to remember:

• Keep the raw OCR reading next to the normalised value, so you can compare when in doubt.
• Treat separators (comma, dot, space) as format data, not as an obvious fact: "1.250" is ambiguous across countries.
• Explicitly mark fields not found rather than leaving them empty without explanation, so you can distinguish "absent from the document" from "not extracted".

Model confidence and evidence: two different things

This is the most costly confusion in document automation. Model confidence is an internal score: it says the extraction engine believes it read well. Evidence is external to the model: it is the document zone you can display next to the value, the cross-check between two fields of the same document, or consistency with an external reference such as a registered order.

A high confidence score proves nothing. A model can be very sure of a wrong reading, for instance when a stamp overlaps a figure or when the document's font makes "1" and "7" nearly identical. Conversely, a low score does not mean the value is wrong, only that the reading is uncertain. Both dimensions must therefore be handled separately:

• Confidence is for triage: above a threshold you set, automatic processing; below it, human review.
• Evidence is for deciding: it is what lets a human validate or correct in seconds, and lets a later audit understand why the value was retained.

Setting that threshold is a business decision, not a universal constant. It depends on the cost of an error, the volume processed and how easy the review is. A threshold set too low drowns the team in useless reviews; one set too high lets wrong readings through automatically. Start cautious, measure, adjust.

The reference pipeline, step by step

Here is an actionable procedure for an incoming document flow, broken into steps with the control attached to each:

1. Reception and sealing. On arrival, assign a unique identifier to the document and keep the original intact (file, date, arrival channel, sender if known). This sealed original is the reference for everything else; you never rework the source file.
2. Classification. Determine the document type (invoice, purchase order, letter, form). An unrecognised document goes to human review rather than being processed by default as a common type.
3. Extraction with provenance. The extraction module produces, for each field: raw value, normalised value, type, unit, page, zone and confidence score. A field not found is marked as such.
4. Format checks. Valid dates, amount formats, currency present, references matching a known pattern. Any failure sends the document to human correction with the zone highlighted.
5. Business validation. Internal consistency rules (totals, positive quantities, dates in a plausible range) and external cross-checks (matching against an existing order or case). Every rule has a business owner, not just a developer.
6. Tooled human correction. The reviewer sees the document and the value side by side, corrects, and the correction is recorded as an explicit decision with their name or identifier and a timestamp.
7. Output traceability. The finalised case references the sealed original, all extracted values with their provenance, human corrections and the rules triggered.

This structure echoes the access governance principles described in our checklist on AI agent permissions: a document pipeline must know who may see what, who may correct what, and who is entitled to give final validation.

Main example: two versions of a purchase order

Take a fictional case, designed for illustration. You receive a customer's purchase order BC-2026-0417, then, two days later, a second document from the same customer carrying the same reference. The pipeline extracts:

Field	Version A (received on the 12th)	Version B (received on the 14th)
Reference	BC-2026-0417	BC-2026-0417
Line 1	40 units — 60.00 €/unit	50 units — 60.00 €/unit
Line 2	10 cases of 6 — 45.00 €/case	10 cases of 6 — 45.00 €/case
Displayed total	3,000.00 €	3,450.00 €

Let us check the internal arithmetic. Version A: 40 × 60 = 2,400.00 €, plus 10 × 45 = 450.00 €, i.e. 2,850.00 €. Yet the displayed total is 3,000.00 €: an internal inconsistency of 150.00 €. Version B: 50 × 60 = 3,000.00 €, plus 450.00 €, i.e. 3,450.00 €, which matches the displayed total. Version B is therefore the only one consistent with its own lines.

The pipeline must not decide on its own that "latest wins". It must first detect the conflict (same reference, incompatible values on line 1's quantity and on the total), check the internal consistency of each version, then apply an authority rule defined in advance. Three possible authority rules, to choose according to your business:

• The most recent version prevails, unless it is internally inconsistent — the case here: B is more recent and consistent, A is not.
• The signed document or the one confirmed through an official channel prevails, regardless of date.
• Systematic escalation: any reference conflict goes to human review, whatever the confidence scores.

In our fictional example, the "most recent and consistent" rule retains version B, but the case keeps both documents, the value comparison, the rule applied and, had the rule been escalation, the named human decision. If nobody defined an authority rule in advance, the pipeline must block the case rather than invent a priority. An unresolved, unescalated conflict is the worst outcome: it produces an order at 2,850.00 € or 3,450.00 € without anyone having chosen.

Decision table: what to do based on control results

Situation	Model confidence	Evidence available	Recommended decision
Value read, valid format, internal consistency OK	High	Source zone displayable	Automatic processing, traceability kept
Value read, but total inconsistent with lines	Irrelevant	Inconsistency documented	Mandatory human correction
Two documents, same reference, incompatible values	Irrelevant	Conflict documented	Apply the authority rule or escalate
Field absent from the document	None	Verifiable absence	Mark "not found", decide if the field is mandatory
Uncertain reading (stamp, degraded font)	Low	Source zone displayable	Human review with side-by-side display
Unclassified document	None	—	Human review, do not guess the type
External matching impossible (no linked order)	Irrelevant	Cross-check documented	Block or escalate per the defined rule

This table is a design guide: thresholds and escalation rules belong to you, and none of these criteria replaces your own obligations regarding internal control.

Human correction: tool it, don't endure it

Human correction fails for one main reason: the tool does not show the evidence. If a reviewer must open the PDF, find the page, locate the table and mentally compare with the extracted value, they will rush and default to trusting the model. The correction interface must therefore display the document and the value together, highlight the source zone, pre-fill the normalised value and allow correction in one or two clicks.

Second requirement: record the correction as a decision, not as a silent overwrite. The corrected value, the original value, the author and the timestamp stay in the case file. This is what then lets you measure where the pipeline errs most often and fix the rules or the extraction instead of retyping the same errors forever.

Third point: do not give correction and final validation to the same people when financial stakes are real. Whoever corrects a quantity should not be the only one able to trigger the resulting commitment. This role-separation principle applies to any system where an automated agent prepares actions, as we detail for organising knowledge and access for enterprise chatbots.

Governance: retention, access rights and the limits of compliance

A document pipeline accumulates personal data and data sensitive to your activity: amounts, contact details, supporting documents. Three governance questions arise at design time.

How long to retain? Define retention periods per document category and per data type, distinguishing the sealed original, the extractions and the technical logs. A single, unlimited retention period is rarely defensible; a documented period per category is far more so.

Who may see and correct what? Access rights follow the case and its content, not only the person's job title. An order manager does not need access to documents from another perimeter. The CNIL describes the main principles of personal data protection that structure these choices: purpose limitation, minimisation, controlled retention periods.

What can be said about compliance? One honest point: the recommendations in this article are design choices, not legal guarantees. Designing a pipeline with provenance, traceability and differentiated rights puts you on a serious path, but it does not "guarantee" your processing compliance on your behalf. Compliance depends on your purposes, your legal bases and your own situation — topics outside this article's scope that belong to your own analysis and, if needed, your advisers.

Common mistakes and how to avoid them

• Confusing confidence score with truth. The score triages, the evidence decides. Never validate a value on its score alone.
• Overwriting the raw value. Without the original reading, you cannot replay an extraction or understand a correction.
• Ignoring units and currencies. "12" and "1,250" without context are traps; keep unit and currency as data.
• Letting the pipeline choose in a conflict. The authority rule is defined cold, not at the moment of conflict.
• Correcting without a trace. An anonymous, overwriting correction makes the case unauditable and prevents improving the system.
• Treating an unrecognised document as a common type. Better a human review than an invoice processed as a delivery note.

If you build this kind of pipeline with AI automation components, keep in mind that an exchange protocol between tools carries context and calls, but does not decide for you the business rules or the permissions: those choices remain yours, as we explain in our article on chatbot development, from acceptance testing to handover. And if your documents feed an internal knowledge base, the same provenance requirement applies, see our guide on an AI agent's knowledge base.

If you would like to define a document automation mission suited to your flows, you can describe your needs on our missions page.

Frequently asked questions

Should the page and table of origin of an extracted value always be displayed?

Yes, whenever the value feeds a decision. Provenance is what allows verification, correction and audit without redoing the work. For purely decorative or unused fields, the cost of traceability is not justified, but it is better to apply it by default than to have to add it afterwards.

What confidence threshold should be chosen for routing to human review?

There is no universal threshold. Start with a cautious setting (many reviews), measure the actual error rate in automatic mode and the time spent reviewing, then adjust. The right threshold depends on the cost of an error in your business, not on the extraction engine's advertised capabilities.

How to handle two contradictory documents with no defined authority rule?

Block the case and escalate. The worst choice is letting the system decide by default, for instance in favour of the most recent document, without checking each one's internal consistency. Then define a written authority rule so the next conflict is handled automatically or through a clear procedure.

Does keeping the sealed original raise access rights issues?

It mainly raises questions: retention period, access, personal data contained in the document. Keeping the original is useful for evidence, but you must govern who accesses it and for how long. These trade-offs belong to your analysis of your purposes and obligations, not to an automatic technical rule.

Can everything be automated if confidence scores are good?

No. A high score covers neither conflicts between documents, nor business inconsistencies, nor never-seen cases. Automation handles the routine flow; exceptions, conflicts and committing decisions remain human control points, precisely because they are rare and costly.