/ Paper arXiv 2610.39301
Curator's notes
Youtu-Parsing-Omni uses a single encoder and schema to parse text, images, audio, and video into structured records.
Curator's notesWritten by AI from the paperEvery number here appears in the abstractReport a mistake
What's new
It introduces a unified encoder for all modalities and a self‑evolving distillation training loop (OSAD) that shares objectives across ten parsing tasks.
Key result
With a single 5 B‑parameter checkpoint it scores 96.96 Overall on OmniDocBench v1.6 and 75.08 Avg. on OmniParsingBench, leading open‑weight models.
Why it matters
Builders can obtain a single model that extracts structured, time‑aligned information from any media type, simplifying multimodal perception pipelines.
Abstract
From the paperThe authors' own words.
We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.
From the full paper
From the paperCopied from the paper itself by our code.
Reading the paper on arXiv…
What you could build
Written by AIAdvancedRequires large‑scale model training and multimodal data handling
- Create a multimodal search engine that indexes and cites evidence across text, images, audio, and video
- Develop an AI assistant that reads documents and videos, extracts key facts, and presents them as structured tables
- Build a tool that automatically transcribes and parses sheet music, chemical formulas, and charts from scanned images and videos
Try it
Code, models and demos that use this paper.
- Models:
- tencent/Youtu-Parsing-Omni ↗1 model on Hugging Face
- Demos:
- mrfakename/youtu-parsing-omni ↗1 demo on Hugging Face
Ask this paper
/ Ask a questionAnswers come from the full paper and name the section
/ Quiz yourself
What training method allows the parser and its supervision to co‑evolve in Youtu-Parsing-Omni?
The method is OmniSchema‑Aware On‑Policy Distillation (OSAD), where the model trains on its own rollouts against a reference‑conditioned copy that is refreshed each round.
/ Your notes
Select text in the abstract to highlight it or add a note. They stay in this browser.
