# ARLoop: Fully-Open Pre-trained Looped Transformers with Attention Residuals

**Authors:** Yuekun Yao, Yupei Du, Dawei Zhu, Huan Sun  
**Lab:** Saarland University · **Published:** 3 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.05234) · [PDF](https://arxiv.org/pdf/2611.05234) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.05234.html)

## Key numbers

- **1.2B**: we further scale ARLoop to 0.6B and 1.2B parameters under a realistic LLM pretraining recipe

## Abstract

Looped transformers increase effective depth by repeatedly applying a shared set of transformer layers, offering a parameter-efficient way to scale computation. However, standard residual connections aggregate information across recurrent depth into a single stream, providing limited control over which intermediate representations from preceding layers should be reused. In this paper, we introduce ARLoop, a looped transformer architecture based on attention residuals, where the input of each layer is selectively aggregated through attention over model depth. Distinct from standard transformers, we find that looped transformers do not require access to the full residual history: sliding-window depth attention matches or outperforms full-history attention at larger recurrent depths. Across controlled experiments with multiple parameter scales (140M and 370M) and recurrent depths (R = 1 to 5), ARLoop consistently outperforms prior residual designs for looped transformers. More importantly, we further scale ARLoop to 0.6B and 1.2B parameters under a realistic LLM pretraining recipe, demonstrating that the gains transfer to realistic pretraining, with particularly strong improvements on math and coding tasks. Our 1.2B model, pretrained on only 0.5T tokens with three recurrences, outperforms Huginn-3.5B (the only fully-open looped LLM as of our release date) across all evaluated tasks and is competitive with similar-sized non-looped models such as Qwen3.5-2B despite being trained on substantially less data.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.