# CANDO: Conservative Action-Noise Directional Optimization for Residual Reinforcement Learning

**Authors:** Zeyu Gao, Chengkai Hou, Chaoyue Li, Zhuangzhe Wu, Jiaming Liu, Yao Mu, Peng Jia, Li Zhang, Shanghang Zhang  
**Lab:** Fudan University · **Published:** 3 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.04543) · [PDF](https://arxiv.org/pdf/2611.04543) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.04543.html)

## Key numbers

- **91.4%**: achieving 76.1 NAUC and a 91.4% final success rate
- **4.1%**: surpassing the best single-space results by 6.6 NAUC and 4.1% success rate

## Abstract

Pretrained generative policies provide strong behavior priors for robot control, but adapting them efficiently with online reinforcement learning remains difficult. A central challenge is how to improve a frozen generative policy without destroying the structure learned from offline demonstrations. Noise-space residual methods address this by modifying the policy's input noise, but existing noise-space methods typically regulate adaptation through fixed latent feasible regions, which may exclude valuable noise edits. In contrast, action-space residual methods are more sample-efficient, but their efficiency can still be further improved. We propose Conservative Noise Directional Optimization (CNDO), a noise-space residual RL method that improves a frozen generative policy by learning state- and noise-conditioned directions around Gaussian latent samples. CNDO removes predefined latent feasible regions and instead regulates noise-space adaptation through the soft conservative constraint, which consists of conservative value estimation and gated execution. We further introduce Conservative Action-Noise Directional Optimization (CANDO), which adds an action-space residual, combining structured latent-space improvement with local action correction. Experiments show that, once CNDO narrows the gap between noise-space and action-space residual learning, CANDO further improves over single-space optimization, achieving 76.1 NAUC and a 91.4% final success rate, surpassing the best single-space results by 6.6 NAUC and 4.1% success rate. More demos and info are available on https://cando-rl.github.io/.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.