Hybrid Document Extraction (VLM + OCR)
An extraction pipeline that turns messy real-world documents into structured data, with every value linked back to its location on the page. It uses only open-source models and runs on the client's own infrastructure.
- Industry
- Fintech & Document Automation
- Category
- Document AI / OCR
- Models
- Open-source only
- Deployment
- On-prem, GPU or CPU
The challenge
Standard OCR struggles with skewed photos, crumpled paper, mixed layouts and handwriting. Cloud document-AI services also require sending sensitive financial documents to third parties.
The objective
Reliably extract line items, totals, dates, payees, account numbers and other custom fields from messy or handwritten documents without sending pages to an external API.
Our approach
Computer-vision preprocessing (detection, deskewing, region proposals) feeds PaddleOCR, which reads text and its layout. A vision-language model then interprets the page structure and outputs clean data. Combining exact OCR text with the model's layout understanding makes the system more robust than either method alone, and bounding boxes let reviewers verify every value quickly.
How it works
- 01
Ingest a scan, phone photo or PDF page.
- 02
Preprocess and propose regions.
- 03
Run PaddleOCR to read text and layout.
- 04
Structure the content with the vision-language model.
- 05
Return structured data with each value mapped to its source box.
The build
- Vision
- OpenCV, deskewing, region proposals
- OCR
- PaddleOCR
- Model
- Self-hosted open-source vision-language model
- Service
- Python, on-prem GPU or CPU inference
The result
"Every cloud OCR we tried either choked on a crumpled receipt or wanted us to upload our documents somewhere."
Want results like this?
Tell us about your project and get a fixed quote within 24 hours.
Get a free quote ->