# CRESiST: Self-Enhancing Explorative Policy Learning for Simultaneous Speech Translation

**Authors:** Chenyang Le, Pei Zhang, Yuxiang Chen, Bing Han, Derek F. Wong, Baosong Yang, Yanmin Qian  
**Lab:** Shanghai Jiao Tong University · **Published:** 4 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.06501) · [PDF](https://arxiv.org/pdf/2611.06501) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.06501.html)

## Abstract

Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from redundant Key-Value cache recomputation and necessitates external Voice Activity Detection to process unbounded streams. To overcome these bottlenecks, we propose an end-to-end interleaved read-write framework built upon an omni-modal LLM. By interleaving speech chunks and text tokens, our architecture achieves full KV cache reuse, natively supporting segmentation-free, infinite-length inference. Furthermore, to learn optimal streaming policies without rigid offline alignments, we introduce a novel Actor-Critic self-enhancement pipeline. An RNN-T Actor dynamically samples read-write trajectories, while an LLM-based Critic evaluates these paths via intrinsic semantic and latency rewards, distilling the optimal policy back to the Actor. Extensive experiments on sentence-level (FLEURS, CoVoST2) and document-level (MuST-C, ACL 60/60) benchmarks demonstrate that our model establishes new state-of-the-art quality-latency Pareto frontiers, substantially outperforming existing cascaded and static-alignment baselines.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.