# A Geometric Invariant of Attention Predicts Head Specialization from Weights Alone

**Authors:** Miguel P. Bento  
**Published:** 9 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.12206) · [PDF](https://arxiv.org/pdf/2611.12206) · [Code](https://github.com/MPedraBento/rho-invariant) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.12206.html)

## Abstract

Transformers are built with continuous symmetries in the attention mechanism. We derive $ρ$, a scalar invariant that measures each head's capacity to couple to rotary positional encodings. This framework allows us to analyze heads without forward passes, opening a new research direction into data-independent analysis of modern LLMs. Empirically, we find that $ρ$ predicts induction heads, the most-studied circuit type in mechanistic interpretability, with Spearman correlation between $-0.55$ and $-0.74$ across four bias-free RoPE models spanning $135$M to $7$B parameters. As a preliminary proof of concept, we use $ρ$ to apply YaRN to only the top-50% positional heads, which degrades perplexity less than uniform application. Code is available at https://github.com/MPedraBento/rho-invariant.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.