# Massive Activations Confound Sparse Autoencoders

**Authors:** Nathan Delisle, Chenhao Tan  
**Lab:** Hugging Face · **Published:** 10 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.13548) · [PDF](https://arxiv.org/pdf/2611.13548) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.13548.html)

## Key numbers

- **99.9%**: Massive activations cause up to 99.9% of activation variance to lie along the first principal component
- **99.96%**: A dictionary can also report 99.96% of variance explained while leaving 33% of the variance outside…
- **33%**: …can also report 99.96% of variance explained while leaving 33% of the variance outside PC1 unexplained.

## Abstract

Sparse autoencoders (SAEs) are commonly evaluated by variance explained and ablation damage. We show how massive activations undermine both evaluations, and propose a correction. Massive activations cause up to 99.9% of activation variance to lie along the first principal component (PC1). Because the PC1 component has no unique sparse decomposition, each SAE run rebuilds it from a different set of features. These features dominate importance rankings: an average of 47 of the 50 most important features are unmatched in a separate run, compared with about 27 of a random 50. A dictionary can also report 99.96% of variance explained while leaving 33% of the variance outside PC1 unexplained. We bypass PC1 by training the SAE on activations with PC1 removed, then adding PC1 to the reconstruction. In affected models, the top-50 feature sharing rate rises from 4-39.5% to 73-99.5%. At the same TopK sparsity, the resulting SAE degrades the model up to 44% less and reduces content reconstruction error by up to 33.9%.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.