/ Paper arXiv 2611.10866
Curator's notes
Zeroing less than 0.05% of LLM weights via a learned binary maskbinary maskA vector of 0/1 values that selects which weights to keep or delete. matches full fine‑tuning across several post‑training tasks.
Delete a tiny fraction of a language model's weights to get the same results as full retraining, using a small mask.Zeroing less than 0.05% of LLM weights via a learned binary maskbinary maskA vector of 0/1 values that selects which weights to keep or delete. matches full fine‑tuning across several post‑training tasks.Deletion Tuning learns a binary maskbinary maskA vector of 0/1 values that selects which weights to keep or delete. over transformer gate projections, zeroing <0.05% of parameters to match full fine‑tuning across RL, SFT, and distillation.Curator's notesWritten by AI from the paperEvery number here appears in the abstractReport a mistake
What's new
Introduces Deletion Tuning, a mask‑only method that achieves full fine‑tuning performance while using a tiny checkpoint and causing minimal forgetting.
Key result
Zeroing fewer than 0.05% of weights matches full fine‑tuning performance across reinforcement learning, supervised fine‑tuning, and on‑policy distillation.
Why it matters
Developers can adapt large models with negligible storage and reduced catastrophic forgetting.
Key numbers
From the paperCopied from the abstract, not written by AI.
Why this ranked
MeasuredPublic numbers, not AIHow ranking works
63 / 100#2 of 198 papers that day, from GitHub, Hugging Face, Hacker News and Semantic Scholar, looked up the morning it was ranked.
Builder evidence32 / 40
- Code released
- ★ 63 GitHub stars
- +63 stars since the day before
- 2 models on Hugging Face
Community attention20.4 / 30
- ▲ 19 upvotes on Hugging Face
- 30 points on Hacker News
Author track record2.5 / 15
- From agentic learning ai lab
What the paper shows7.9 / 15
- 1 benchmark result, each found word for word in the abstract
- Less like recent papers than 78% of that day's candidates
Extracted by AI, checked by codeEach result was found by the model and kept only because its sentence and number appear word for word in the abstract.
- prior-task accuracy 5.4 points loss compared to full fine-tuning
Under reinforcement learning, its prior-task accuracy stays near the base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points.
Tie-break. This paper was within 3 points of the next one, so an AI editor compared the leaders and preferred it in 1 of 1 comparisons: “Deletion Tuning offers a genuinely new, lightweight fine‑tuning method with strong quantitative results and released code.”
Abstract
From the paperThe authors' own words.
How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities? We find that zeroing fewer than 0.05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Our method, Deletion Tuning, learns a binary mask over the gate projection of transformer blocks. Two practical advantages follow: the trained mask is hundreds to thousands of times smaller than the corresponding FFT checkpoint, and Deletion Tuning forgets far less, which addresses a central challenge in continual learning. Under reinforcement learning, its prior-task accuracy stays near the base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points. To understand why masking alone suffices, we analyze the geometry of the learned weight changes under reinforcement learning using stable rankstable rankA measure of matrix complexity based on its singular values., spectrum drift, and principal-weight overlapprincipal-weight overlapSimilarity between dominant directions of weight changes in different training regimes.. Compared with LoRA and random sparse fine-tuning, Deletion Tuning is closest to FFT across the three measures.
From the full paper
From the paperCopied from the paper itself by our code.
Reading the paper on arXiv…
What you could build
Written by AIIntermediateRequires understanding of transformer gating and mask optimization, but no massive compute beyond standard fine‑tuning.
- Create a tool that generates deletion masks for custom downstream tasks.
- Implement a continual‑learning system that swaps masks instead of full model updates.
- Build a lightweight model‑serving pipeline that stores only masks for multiple task versions.
Where it fits
The most cited papers it builds on, and the papers that build on it.
/ Builds on 3 most cited
- 201733.5k citedProximal Policy Optimization Algorithms ↗
- 202225.9k citedTraining language models to follow instructions with human feedback ↗
- 202125.7k citedLoRA: Low-Rank Adaptation of Large Language Models ↗
/ Built on by None yet
No papers cite it yet. We check again every week.
From Semantic Scholar, updated weekly.
Try it
Code, models and demos that use this paper.
- Models:
- Jacklu0831/deletion-tuning-masks ↗2 models on Hugging Face
Ask this paper
/ Ask a questionAnswers come from the full paper and name the section
/ Quiz yourself
What proportion of weights does Deletion Tuning zero to match full fine‑tuning performance?
It zeros fewer than 0.05% of the model's weights.
/ Your notes
Select text in the abstract to highlight it or add a note. They stay in this browser.