In Document QA Project, we used LayoutLM to read documents. But that required a separate OCR step to find the text first. Now, Hugging Face Donut offers an end-to-end solution that eliminates the need for separate OCR.
Donut (Document Understanding Transformer) changes the game. It is an “end-to-end” model. It looks at the raw image pixels and outputs structured JSON, skipping the OCR engine entirely. It’s perfect for parsing messy receipts, invoices, or charts.
Step 1: Installation
We need transformers and protobuf.
pip install transformers torch protobuf sentencepiece
Step 2: The Code
We will use a pre-trained model fine-tuned on receipts (naver-clova-ix/donut-base-finetuned-cord-v2).
from transformers import DonutProcessor, VisionEncoderDecoderModel
from PIL import Image
import torch
import requests
import re
import json
# 1. Load Model and Processor
model_name = "naver-clova-ix/donut-base-finetuned-cord-v2"
processor = DonutProcessor.from_pretrained(model_name)
model = VisionEncoderDecoderModel.from_pretrained(model_name)
# Move to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
# 2. Load an Image (Receipt)
url = "https://huggingface.co/datasets/naver-clova-ix/cord-v2/resolve/main/test/image/0004.png"
image = Image.open(requests.get(url, stream=True).raw)
# 3. Pre-process the image
pixel_values = processor(image, return_tensors="pt").pixel_values.to(device)
# 4. Generate Output (Decoder)
task_prompt = "<s_cord-v2>" # Special start token for this task
decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids.to(device)
outputs = model.generate(
pixel_values,
decoder_input_ids=decoder_input_ids,
max_length=512,
early_stopping=True,
pad_token_id=processor.tokenizer.pad_token_id,
eos_token_id=processor.tokenizer.eos_token_id,
use_cache=True,
bad_words_ids=[[processor.tokenizer.unk_token_id]],
return_dict_in_generate=True,
)
# 5. Decode to JSON
sequence = processor.batch_decode(outputs.sequences)[0]
sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
# Remove the start token using regex
sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()
# Convert the structured string to a real Dict
print("--- Extracted Data ---")
print(processor.token2json(sequence))Step 3: The Result
The output is a perfectly clean Python dictionary:
{
"menu": [
{"nm": "ICED LATTE", "cnt": "1", "price": "15,000"},
{"nm": "CROISSANT", "cnt": "2", "price": "10,000"}
],
"total_price": "25,000"
}You just extracted structured data from a photo without running OCR!
Production Scale: Asynchronous Orchestration
VisionEncoderDecoder models like Donut are computationally heavy. If you drop this code directly into a synchronous web endpoint (like a standard Flask or FastAPI route), a single request will block your server while the GPU churns through the image pixels.
To deploy this in production, you must offload the inference to an asynchronous task queue using Celery and Redis.
# tasks.py
from celery import Celery
# Initialize a Redis-backed task queue to handle heavy vision workloads
app = Celery('document_pipeline', broker='redis://localhost:6379/0')
@app.task(bind=True, max_retries=3)
def process_receipt_async(self, image_url):
try:
# 1. Fetch image
image = Image.open(requests.get(image_url, stream=True).raw)
# 2. Run Donut Inference (Assuming model is pre-loaded globally)
pixel_values = processor(image, return_tensors="pt").pixel_values.to(device)
outputs = model.generate(pixel_values, max_length=512) # Truncated for brevity
# 3. Decode
sequence = processor.batch_decode(outputs.sequences)[0]
cleaned_sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()
return processor.token2json(cleaned_sequence)
except Exception as exc:
# Re-queue the task if the image download fails or GPU is OOM
self.retry(exc=exc, countdown=5)The Truncated Sequence Crash
A classic issue professional engineers encounter when moving Donut from testing to production is the catastrophic failure of processor.token2json(). If a user uploads an exceptionally long invoice, the model hits its token limit and stops generating midway through a JSON block.
The Error:
# The model stops generating before closing the JSON brackets
sequence = '{"menu": [{"nm": "COFFEE", "cnt": "1"'
processor.token2json(sequence)
# Crash! The sequence is missing closing tags, breaking the internal regex parser.The Fix:
You must defend against malformed sequences. Always increase your max_length for long documents, but more importantly, wrap the token parser in a robust fallback mechanism that attempts to salvage the partial data using a fuzzy JSON repair library, or gracefully returns the raw string if the structure is completely broken.
import json_repair # pip install json-repair
def safe_parse_donut(sequence, max_retries=2):
try:
# First attempt: Use the native Donut processor
return processor.token2json(sequence)
except Exception:
# Fallback: The model hallucinated or truncated the closing brackets.
# Use a repair library to salvage the extracted key-value pairs.
print("Warning: Malformed Donut output. Attempting to repair JSON structure.")
repaired_json_string = json_repair.repair_json(sequence)
return json.loads(repaired_json_string)
# Usage
extracted_data = safe_parse_donut(sequence)Key Takeaways
- Hugging Face Donut is an end-to-end model that outputs structured JSON from raw image pixels, eliminating the need for OCR.
- Installation requires the libraries transformers and protobuf for setup.
- The pre-trained model ‘naver-clova-ix/donut-base-finetuned-cord-v2’ efficiently extracts structured data without OCR.
- For production use, implement asynchronous orchestration with Celery and Redis to manage computational load.
- To address token limit issues, increase ‘max_length’ and use a robust fallback mechanism for partial data extraction.
