# Accelerating Safe Reinforcement Learning with Massive Parallelism

**Authors:** Joonyoung Lim, Younghwan Yoo  
**Lab:** Pusan National University · **Published:** 5 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.08273) · [PDF](https://arxiv.org/pdf/2611.08273) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.08273.html)

## Abstract

We investigate the problem of scaling safe reinforcement learning (safe RL) to massively parallel training regimes, where a large number of environments are exe- cuted with short rollout horizons. Although this setting improves data throughput and wall-clock efficiency, it creates a mismatch with constrained Markov decision processes (CMDPs), whose safety constraints are defined over full-episode cost returns. We address this mismatch by analyzing staggered environment resets as a phase-mixture distribution over short trajectory segments. This analysis charac- terizes the bias induced by phase coverage and motivates a phase-aggregated cost estimator that reconstructs episode-level cost estimates from short rollout batches. We further incorporate finite-sample uncertainty and distribution mismatch into a constraint-tightening framework, yielding a scalable Safe RL method compatible with on-policy optimization. Using an MJX-based implementation of Safety Gym, our experiments show that the proposed method achieves performance comparable to non-massively parallel implementations while substantially reducing wall-clock training time.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.