99.2%
Structured-corpus reconciliation
124 of 125 purchase orders
← All case studies
AI & agentic workflows Sanitised case study
Supplier pricing benchmark with independent validation
Produced a defensible supplier pricing benchmark with 99.2% structured-corpus reconciliation and 98.2–98.3% agreement between two independent extraction engines.
98.2–98.3%
Cross-engine agreement
1,847
Price points cross-validated
722 lines · 333 tests
Validation depth
0
Quality flags raised
across 8,508 extracted rows
Evidence inputs
- A supplier pricing benchmark across a structured purchase-order corpus and a harder catalogue set.
- Python · regex · pdfplumber
Transformation
- Parsed structured purchase orders deterministically and reserved model vision for the harder catalogue.
- Ran a second independent engine and cross-validated results price point by price point.
- Selected the lowest-cost method that met the accuracy threshold for each document type.
Controls & assurance
- Matched method to document structure instead of applying one extraction technique everywhere.
- Used independent-engine agreement as the accuracy evidence.
Output
- The structured corpus reconciled to 99.2% (124 of 125 purchase orders, 722 lines, 333 tests); the two engines agreed to 98.2–98.3% across 1,847 price points; automated quality checks — unit, currency, and magnitude anomalies — raised zero flags across 8,508 extracted rows.
Business value
- Produced a defensible pricing benchmark with accuracy evidenced by cross-validation rather than assumed.
- Problem
Pricing sat across structured purchase orders and unstructured catalogues, without a reliable normalised benchmark.
- Approach
Parsed structured purchase orders deterministically and reserved model vision for the harder catalogue.
- Outcome
The structured corpus reconciled to 99.2% (124 of 125 purchase orders, 722 lines, 333 tests); the two engines agreed to 98.2–98.3% across 1,847 price points; automated quality checks — unit, currency, and magnitude anomalies — raised zero flags across 8,508 extracted rows.
Business value
Produced a defensible pricing benchmark with accuracy evidenced by cross-validation rather than assumed.
Transformation route
- 01
Parsed structured purchase orders deterministically and reserved model vision for the harder catalogue.
- 02
Ran a second independent engine and cross-validated results price point by price point.
- 03
Selected the lowest-cost method that met the accuracy threshold for each document type.
Decision log
- Matched method to document structure instead of applying one extraction technique everywhere.
- Used independent-engine agreement as the accuracy evidence.
This case study proves
- Regex-vs-LLM routing AI & Automation Strong
What I learned
- Independent agreement is stronger evidence than one engine validating itself.
Future improvements
- Extend labelled evaluation across the full catalogue set and track accuracy by document type.