# Breakdown of Autoregressive Multi-Step Reasoning Under Concentrated vs. Diffuse Attention Head Dropout

**Authors:** Karol Struniawski  
**Published:** 10 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.14209) · [PDF](https://arxiv.org/pdf/2611.14209) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.14209.html)

## Key numbers

- **5.7%**: scheduling ~ 5.7% of vulnerable head-token volume
- **91.2%**: Qwen3-8B accuracy drops from 91.2% under diffuse noise to 78.8% under burst dropout
- **78.8%**: Qwen3-8B accuracy drops from 91.2% under diffuse noise to 78.8% under burst dropout

## Abstract

During extended wakefulness, mammalian neocortical networks experience localized periods of transient neuronal silence termed off-states. Here, we examine how analogous spatiotemporally concentrated disruptions affect sequential reasoning in autoregressive Large Language Models (LLMs). Evaluating 250 paired derivations on the GSM8K benchmark across two 8B models (Qwen3-8B and LLaMA-3.1-8B-Instruct), we contrast clustered bursts of attention head masking against budget-matched diffuse i.i.d. dropout. Concentrated bursts induce substantial reasoning degradation: at onset parameter B=0.10 (scheduling ~ 5.7% of vulnerable head-token volume), Qwen3-8B accuracy drops from 91.2% under diffuse noise to 78.8% under burst dropout ($Δ= -12.4$ percentage points [pp], exact McNemar p = 5.5 × 10^-6), while LLaMA-3.1-8B-Instruct drops from 84.0% to 76.8% ($Δ= -7.2$ pp, $p = 0.0198$). Conversely, dispersing the identical cumulative budget uniformly across generation steps via stratified quintile matching yields substantially milder degradation (retaining 91.2% in Qwen and 84.0% in LLaMA). In an exploratory factorial dissociation ($N=100$), single-step dropout density is associated with an acute drop in a single schedule draw ($-10.0$ pp on Qwen, p_holm = 0.0425), while multi-token continuity exhibits an inconclusive compounding trend ($-7.0$ pp, p_holm = 0.1892). Error analysis reveals that Qwen failures reflect derivation errors within formatted responses (conditional accuracy drop of $-11.4$ pp), whereas in LLaMA 22.2% of discordant losses lack a final answer format line (conditional gap narrows to $-5.0$ pp). These findings demonstrate that autoregressive reasoning chains are particularly vulnerable to concentrated structural disruptions, providing empirical benchmarks for fault tolerance in transformer architectures.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.