# R^2Orch: Recursive Self-Improvement for Agent Orchestration via Self-Play within a Skill-Internalization Loop

**Authors:** Mingda Zhang, Yun Huang, Qiang Huang, Yanjin Li, Rui Qian, Qichao Zhao, Tiesunlong Shen, Na Zhao, Jian Wang, Xiaoying Tang  
**Lab:** Chinese University of Hong Kong, Shenzhen · **Published:** 21 Oct 2026 · **Upvotes:** 89 · **Level:** Advanced  
**Links:** [arXiv](https://arxiv.org/abs/2610.22740) · [PDF](https://arxiv.org/pdf/2610.22740) · [Code](https://github.com/beita6969/R2Orch) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2610.22740.html)

## Curator's notes

**TL;DR:** R^2Orch uses recursive self‑play and skill withdrawal to train an orchestrator that learns tasks without relying on hand‑crafted curricula.

- **What's new:** It introduces a recursive self‑play loop with a learnability objective and an outer loop that withdraws skills once performance is maintained without them.
- **Key result:** R^2Orch consistently outperforms baselines across twelve datasets covering question answering, math, code, and interactive tasks.
- **Why it matters:** Builders can create more autonomous agents that reduce reliance on manually designed skill sets and curricula.

## Abstract

In recent years, agentic orchestration has been widely applied to solve complex human tasks. However, existing orchestration learning still faces key challenges, including hand-designed curricula, advantage collapse when a task's rollouts all reach the same outcome, and unreliable evaluation signals that judge each skill by one noisy score. To address these challenges, we propose R^2Orch, a new paradigm of Recursive Self-Play Orchestration: an inner loop of self-play that improves the orchestrator on self-chosen tasks, and an outer loop that improves how the inner loop improves. To support this paradigm, we introduce Recursive Learnability, a training objective that rewards a Proposer for choosing tasks whose rollouts keep reaching different outcomes as skills are withdrawn. Built on this objective, we further propose Skill Withdrawal, in which the outer loop withdraws each skill once a controlled evaluation within self-play shows that the orchestrator performs as well without it. Moreover, the two loops form one cycle: failures are distilled into skills, skills into parameters, and withdrawals lead the Proposer to new failures, so the orchestrator moves toward zero-skill deployment. Experimental results on twelve datasets show that R^2Orch consistently outperforms baselines across question answering, math, code, and interactive tasks. Our code is available at https://github.com/beita6969/R2Orch.

## What you could build

1. Develop a zero‑skill chatbot that self‑improves via recursive task selection
2. Create an automated code‑generation pipeline that withdraws redundant modules
3. Build an interactive tutoring system that learns new problem types through self‑play

## Quiz yourself

What training objective does R^2Orch use to reward the Proposer for choosing tasks with diverse outcomes?

**Answer:** It uses the Recursive Learnability objective, which rewards the Proposer when rollouts keep reaching different outcomes as skills are withdrawn.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.