# ConvPE: Convolutional Position Embedding Improves Transformers

**Authors:** Zehao Wang  
**Lab:** Nanjing University · **Published:** 5 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.07749) · [PDF](https://arxiv.org/pdf/2611.07749) · [Code](https://github.com/wzhagi/convpe) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.07749.html)

## Abstract

ConvPE (Convolutional Position Embedding) improves both the performance and efficiency of Transformer models in language modeling, long-context extension, and vision. It injects a windowed, content-dependent bias into attention scores through a lightweight causal convolution over a low-dimensional projection of the hidden states. As a lightweight plug-in, ConvPE is compatible with RoPE, attention masks, and KV concatenation. It also supports high-dimensional position embeddings via high-dimensional convolutions. In a reproduction, ConvPE outperforms five 1D baselines and eight 2D baselines. The code is publicly available at https://github.com/wzhagi/convpe.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.