All papersGenerative AI & DiffusionTaoLive AIGC

TaoMate-HD: Progressive-Resolution Autoregressive Audio-Video Generation

/ Paper arXiv 2611.03394

Key numbers

1.6×Experiments show a 1.6x speedup over direct high-resolution generation with the same model and hardware
1.9×Optimized stage-parallel deployment provides a further 1.9x overall speedup over serial progressive generation
11.18Our continuation training reduces character error rate by 11.18 percentage points relative to the baseline

From the paperCopied from the abstract, not written by AI.

Abstract

From the paperThe authors' own words.

Streaming digital humans produce detailed video while continuing speech as new text arrives. High-resolution denoising is costly, and text updates can cause repeated or interrupted speech. We present TaoMate-HD, an autoregressive model for high-resolution audio-video streaming. For each short segment, or chunk, two denoising steps establish coarse video structure at low resolution. A final step refines the video to high resolution, with audio generated throughout the same trajectory. Each stage passes its audiovisual predictions to the same stage of the next chunk as history. Across text updates, the model retains this history to infer what has already been spoken. Continuation training utilizes asymmetric text conditioning, with separate text windows for generation and distillation. Experiments show a 1.6x speedup over direct high-resolution generation with the same model and hardware. Optimized stage-parallel deployment provides a further 1.9x overall speedup over serial progressive generation. Our continuation training reduces character error rate by 11.18 percentage points relative to the baseline. The model also supports hour-long audiovisual generation. Our project page is https://taoliveaigc.github.io/TaoMate-HD.

From the full paper

From the paperCopied from the paper itself by our code.

Reading the paper on arXiv…

Try it

Code, models and demos that use this paper.

Models:
TaoLiveAIGC/TaoMate-HD-preview ↗1 model on Hugging Face
Datasets:
TaoLiveAIGC/TaoMate-HD-demos ↗1 dataset on Hugging Face

Ask this paper

/ Ask a questionAnswers come from the full paper and name the section

    / Your notes

    Select text in the abstract to highlight it or add a note. They stay in this browser.

      Read the PDF