/ Paper arXiv 2611.04418
In short
In shortReport a mistake
Key numbers
From the paperCopied from the abstract, not written by AI.
Abstract
From the paperThe authors' own words.
The advancement of multimodal learning is heavily constrained by the scarcity and high annotation cost of high-quality instruction data. While raw videos offer an abundant and dynamic source of visual knowledge, transforming them into precise instruction data remains highly challenging. To bridge this gap, we present VidUEU-Agent, an automated pipeline that synthesizes instruction data from raw videos for multimodal understanding, editing, and unified tasks. VidUEU-Agent follows a four-stage pipeline: it performs global perception to filter low-quality shots, mines keyframes and multimodal signals, constructs metadata and routes the metadata to synthesize instruction samples. More importantly, our agent framework can autonomously iterate the data distribution based on downstream training results, thereby optimizing data quality, balance, and training effectiveness. Scaling this pipeline, we construct VidUEU-100K, a large-scale dataset of 100K training samples spanning multiple domains and tasks, providing a robust foundation for developing comprehensive multimodal capabilities. Extensive experiments demonstrate that fine-tuning baseline models on VidUEU-100K yields significant and consistent performance gains across diverse representative benchmarks, validating the effectiveness of our proposed agent framework and dataset.
From the full paper
From the paperCopied from the paper itself by our code.
Reading the paper on arXiv…
Try it
Code, models and demos that use this paper.
- Datasets:
- Lruca/VidUEU100k ↗1 dataset on Hugging Face
Ask this paper
/ Ask a questionAnswers come from the full paper and name the section
/ Your notes
Select text in the abstract to highlight it or add a note. They stay in this browser.