Open-Weight LLM · Private & Custom AI
dots.ocr
Unified document parser that replaces multi-model pipelines with a single VLM for layout detection, text extraction, table parsing, and formula recognition—enabling ops teams to automate document workflows and reduce infrastructure complexity in self-hosted environments.
dots.ocr is a 3B-parameter vision-language model that extracts text, layout, tables, and formulas from documents in a single forward pass, maintaining reading order across 11+ languages. For ops and AI teams, it eliminates the need to chain separate OCR, layout detection, and table recognition tools—reducing latency, operational overhead, and data movement in document-heavy workflows.
Model facts
Private deployment
Run dots.ocr in your own environment
Self-hosting dots.ocr keeps document images and extracted content within your infrastructure, eliminating third-party API calls and vendor lock-in. The model runs on enterprise GPU (16–24 GB VRAM in bfloat16), using HuggingFace transformers with flash_attention_2 support. This is critical for regulated industries (finance, legal, healthcare) where document data cannot leave the organization; deployment is straightforward via standard ML serving (vLLM, TGI, or custom inference servers).
Operational AI use cases
Invoice & Receipt Processing
Automate AP/AR workflows: extract vendor, amount, date, and table structures from invoices in bulk. Running privately means sensitive pricing and vendor data stays internal; reading-order preservation ensures line-item accuracy for reconciliation.
Compliance Document Triage
Route contracts, regulatory filings, and internal policies to the right department by parsing headers, sections, and layout. A self-hosted instance avoids exposing confidential documents to external APIs while maintaining audit trails in your own logs.
Knowledge Base Ingestion
Feed PDFs, scanned documents, and emails into internal knowledge systems or RAG pipelines. Accurate layout and reading order prevent hallucinated document structure; multilingual support handles global subsidiary docs without API hops.
Custom AI
As a base for custom AI
dots.ocr is a capable foundation for building document-intelligent agents and retrieval systems. Use it as the document understanding layer in custom workflows: wrap it in an agent that classifies documents, extracts metadata, triggers downstream tasks (e.g., send invoice to approval queue), or feeds structured output into a search index. The single-model design makes prompt-based task switching feasible for custom parsing logic (e.g., 'extract only budget tables' or 'find all risk clauses').
In the operating system
Where it fits
In an LLM.co ops AI stack, dots.ocr sits at the perception/ingestion layer: it bridges unstructured document images into structured knowledge. Use it upstream of a RAG retriever (feeds context into vector DB), a workflow automation engine (triggers ops decisions), or an internal LLM agent (provides document context for reasoning). Its multilingual and layout-aware output is ideal for feeding downstream NLU models or rule-based extraction systems.
Data control & security
Self-hosting eliminates document exposure to third-party cloud APIs. No document images are transmitted, logged externally, or used for model retraining. However, the model itself contains learned patterns from training data—review the training data composition in the repository if handling highly sensitive content. Data security is an architecture choice (your VPC, your controls), not a property of the model itself.
Hardware footprint
Estimate: ~16 GB VRAM (bfloat16, single GPU A100/H100), ~22 GB (float32). Batch inference on 8 GPUs feasible with model parallelism. CPU-only inference possible but slow (~2–5 min per document); not recommended for production ops.
Integration
dots.ocr outputs JSON with bounding boxes, layout categories, and formatted text (Markdown, HTML for tables, LaTeX for formulas). Integrate via: (1) batch APIs using HuggingFace inference containers or vLLM; (2) REST wrappers (FastAPI, Flask) for real-time document upload workflows; (3) message queues (Kafka, RabbitMQ) for high-volume document processing; (4) direct Python calls in orchestration tools (Airflow, Prefect). Requires vision encoder + LLM inference—plan for 20–60 sec latency per document depending on length and hardware.
When it's not the right fit
- —Real-time, single-document latency is critical (20–60 sec per doc may exceed SLA for chat-like UI); consider lighter models like SmolDocling-256M for speed-over-accuracy tradeoffs.
- —Handling ultra-long documents (100+ pages): context length unknown and not specified; may require chunking/sliding-window approaches increasing complexity.
- —Extracting structured data from highly custom layouts (forms, specialized templates) without extensive prompt engineering; performance on out-of-distribution formats not benchmarked.
- —Running on edge/mobile devices; 3B model size requires cloud or robust on-premise infrastructure.
Alternatives to consider
Qwen-VL-OCR (2B–7B variants)
Comparable multilingual VLM with layout/OCR; lighter Qwen-2B option exists; similar self-hosting story but less OCR-specific fine-tuning.
Claude 3.5 Sonnet (API-only)
Superior general reasoning and layout understanding, but requires cloud API calls, higher cost per document, and no self-hosting; better for complex reasoning, not bulk ops.
Marker (open-source pipeline)
Modular multi-model tool (Surya layout detector + LM-based text recognition); more flexible for custom pipelines but higher latency and infrastructure complexity than unified dots.ocr.
Related open models
FAQ
Can I run dots.ocr entirely on-premise without cloud APIs?
Yes. Self-host on your own GPU infrastructure using HuggingFace transformers or vLLM. No external API calls required. Ensure you have 16+ GB VRAM and handle your own inference serving, scaling, and monitoring.
What's the commercial license for dots.ocr?
MIT license permits commercial use, modification, and redistribution with attribution. You can build products on top and deploy privately. Review the full license terms in the repository; no royalty or usage restrictions.
How does dots.ocr compare for non-English documents?
Designed for multilingual support (EN, ZH, +9 more languages). Benchmarked on low-resource language datasets with competitive performance. Exact language coverage requires checking the model card; assumes Qwen-VL base supports your target language.
What if I need higher throughput or lower latency?
Batch inference via vLLM, TGI, or Ray Serve can process 10–50 docs/sec on a single 8-GPU cluster. For sub-second latency on single docs, SmolDocling-256M or quantized variants may be necessary; trade off accuracy.
Build Document Intelligence Into Your Ops Stack
dots.ocr is ready to deploy on LLM.co. Let us help you architect a private document automation system—no vendor APIs, full data control, integrated into your ops workflows. Talk to our team about custom fine-tuning or multi-model orchestration.