# Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

**Authors:** Yunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou, Zekai Zhang, Kun Zhang, Zezhong Wu, Jian Hu, Haihan Gao, Heting Gao, Xin Li, Yinsong Liu, Deqian Jiang, Di Yin, Haoyu Cao, Xing Sun, Bing Liu, Yanqiu Qu, Yunfei Wu, Kun Yin, Zhixiang Wei, Yongxiang Hua, Qianyu Li, Xiaoxing Hu, Peiyuan Zhang, Lijiang Li, Dong Xu, Zhenchuan Yang, Yiliang Jiang  
**Lab:** Tencent · **Published:** 28 Oct 2026 · **Upvotes:** 163 · **Level:** Advanced  
**Links:** [arXiv](https://arxiv.org/abs/2610.39301) · [PDF](https://arxiv.org/pdf/2610.39301) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2610.39301.html)

## Curator's notes

**TL;DR:** Youtu-Parsing-Omni uses a single encoder and schema to parse text, images, audio, and video into structured records.

- **What's new:** It introduces a unified encoder for all modalities and a self‑evolving distillation training loop (OSAD) that shares objectives across ten parsing tasks.
- **Key result:** With a single 5 B‑parameter checkpoint it scores 96.96 Overall on OmniDocBench v1.6 and 75.08 Avg. on OmniParsingBench, leading open‑weight models.
- **Why it matters:** Builders can obtain a single model that extracts structured, time‑aligned information from any media type, simplifying multimodal perception pipelines.

## Abstract

We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.

## What you could build

1. Create a multimodal search engine that indexes and cites evidence across text, images, audio, and video
2. Develop an AI assistant that reads documents and videos, extracts key facts, and presents them as structured tables
3. Build a tool that automatically transcribes and parses sheet music, chemical formulas, and charts from scanned images and videos

## Quiz yourself

What training method allows the parser and its supervision to co‑evolve in Youtu-Parsing-Omni?

**Answer:** The method is OmniSchema‑Aware On‑Policy Distillation (OSAD), where the model trains on its own rollouts against a reference‑conditioned copy that is refreshed each round.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.