# L3G-World: Real-Time Infinite World Generation with Latent 3D Gaussian Memory

**Authors:** Jinhyeok Choi, JoungBin Lee, Jongmin Lee, Minkyeong Jeon, Jiyoung Kim, Kaeun Han, Yoonjin Yoon, Seungryong Kim  
**Lab:** KAIST · **Published:** 29 Oct 2026 · **Upvotes:** 44 · **Level:** Advanced  
**Links:** [arXiv](https://arxiv.org/abs/2610.40614) · [PDF](https://arxiv.org/pdf/2610.40614) · [Code](https://github.com/cvlab-kaist/L3G-World) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2610.40614.html)

## Curator's notes

**TL;DR:** L3G-World uses a latent 3D Gaussian memory to generate spatially consistent worlds in real time.

- **What's new:** Introduces a compact latent 3D Gaussian memory that grows with scene extent and renders directly into diffusion latent space, avoiding re-encoding.
- **Key result:** Achieves real-time world generation at 9 FPS with up to 38.6× fewer 3D primitives, 3.2× faster updates, and 359× faster rendering versus prior baselines.
- **Why it matters:** Builders can create interactive, large-scale virtual environments without prohibitive memory or latency costs.

## Key numbers

- **38.6×**: with up to 38.6× fewer 3D primitives
- **3.2×**: with up to 38.6× fewer 3D primitives, 3.2× faster memory updates
- **359×**: and 359× faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric

## Abstract

Interactive world models generate visual content conditioned on user actions, yet maintaining spatial consistency over long-horizon exploration in real time remains challenging. Existing approaches rely on either temporal context or explicit 3D memory, both of which scale poorly because memory grows with the length of the generated stream. Moreover, explicit 3D memories condition latent video generators by re-encoding RGB renderings, which precludes real-time generation. In this paper, we introduce a Latent 3D Gaussian Memory (L3G) that combines the efficiency of feed-forward reconstruction with a compact, rapidly renderable scene representation. Building on this memory, we propose L3G-World, in which a memory updater and a video diffusion transformer (DiT) alternate autoregressively to generate spatially consistent worlds. The memory updater spawns Gaussians in newly observed regions and refines existing ones, so that the memory grows with scene extent rather than stream length. Each Gaussian further stores a VAE latent feature, allowing the memory to be rendered directly into the DiT's latent space without VAE re-encoding. Experiments show that L3G-World achieves state-of-the-art generation quality and long-horizon spatial consistency, with up to 38.6× fewer 3D primitives, 3.2× faster memory updates, and 359× faster memory rendering relative to the most efficient prior 3D-memory baseline for each metric, enabling real-time world generation at 9 FPS.

## What you could build

1. Create a real-time game level generator that expands as the player explores.
2. Develop an interactive VR scene editor that updates instantly with user actions.
3. Build a streaming visual storytelling tool that maintains spatial consistency over long narratives.

## Quiz yourself

How does the L3G memory grow compared to previous methods?

**Answer:** The memory grows with the scene extent rather than the length of the generated stream, keeping it compact as the world expands.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.