AI PAPERS/DAILY
Subscribe free Open sourceView on GitHub Star This project is open sourceRead the code or use it in your own project. If you like it, don’t forget to give it a ★ star on GitHub.

All papersAudio, Speech & MusicTop pick on Nov 14Ajou university

WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning

/ Paper arXiv 2611.12527

Curator's notes

WaveGen directly generates speech waveforms from text using a diffusion model that learns internal spectral trajectories.

Curator's notesWritten by AI from the paperEvery number here appears in the abstractReport a mistake

What's new

It removes all intermediate modules like spectrogram vocoders, duration predictors, and self‑supervised embeddings, training a single end‑to‑end model.

Key result

WaveGen achieves a word error rate of 1.47 and a similarity score of 0.75 on the Seed‑en benchmark.

Why it matters

Builders can simplify TTS pipelines to a single model, reducing engineering overhead and latency.

Key numbers

0.6B…the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters.

From the paperCopied from the abstract, not written by AI.

Why this ranked

MeasuredPublic numbers, not AIHow ranking works

62 / 100#1 of 198 papers that day, from GitHub, Hugging Face, Hacker News and Semantic Scholar, looked up the morning it was ranked.

Builder evidence27.2 / 40

  • Code released
  • ★ 56 GitHub stars
  • +4 stars since the day before
  • 1 model, 1 demo on Hugging Face

Community attention12.5 / 30

  • ▲ 17 upvotes on Hugging Face
  • 0 points on Hacker News

Author track record9.9 / 15

  • Authors' median h-index 12 (Semantic Scholar)
  • From Ajou university

What the paper shows12 / 15

  • 2 benchmark results, each found word for word in the abstract
  • Less like recent papers than 80% of that day's candidates

Extracted by AI, checked by codeEach result was found by the model and kept only because its sentence and number appear word for word in the abstract.

  1. Seed-en 1.47 WERWaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark.
  2. Seed-en 0.75 SIMWaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark.

Abstract

From the paperThe authors' own words.

This work proposes WaveGen, a fully end-to-end diffusion-baseddiffusion-basedA generative approach that iteratively denoises random noise to produce data. text-to-waveform framework based on Spectral ForcingSpectral ForcingA model architecture that jointly learns spectral and waveform trajectories with disentangled diffusion heads., which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addition, WaveGen performs internal text-speech alignment within a single model, eliminating the need for external alignment modules such as duration predictors. To preserve a truly end-to-end formulation, our framework further avoids dependence on self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, as well as semantic distillation methods for training optimization. To achieve this, we carefully design Spectral ForcingSpectral ForcingA model architecture that jointly learns spectral and waveform trajectories with disentangled diffusion heads., a model architecture with disentangled diffusion heads, and a training objective based on a multi-scale spectral $v$-loss. We further validate the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters. Using a single architecture, open-source data, and one-stage training, without external modules such as duration predictor or REPA, WaveGen demonstrates fully end-to-end waveform generation achieving state-of-the-art performance on the Emilia training dataset. In particular, WaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen

From the full paper

From the paperCopied from the paper itself by our code.

Reading the paper on arXiv…

What you could build

Written by AIAdvancedRequires diffusion training expertise and substantial GPU resources.

  1. Create a low‑latency voice assistant using WaveGen for on‑device speech synthesis.
  2. Develop a multilingual TTS service by fine‑tuning WaveGen on new language data.
  3. Build an audio book generator that directly converts text chapters to high‑quality waveforms.

Try it

Code, models and demos that use this paper.

Models:
LeeSangHoon/WaveGen ↗1 model on Hugging Face
Demos:
LeeSangHoon/WaveGen ↗1 demo on Hugging Face

Ask this paper

/ Ask a questionAnswers come from the full paper and name the section

    / Quiz yourself

    What two performance metrics does WaveGen report on the Seed‑en benchmark?

    / Your notes

    Select text in the abstract to highlight it or add a note. They stay in this browser.

      / Next pickDon't Just Trust Your Memory: Evidence-Linked Memory Reduces Memory-Induced Degradation in LLM Agents

      / Read next, same idea

      Read the PDF