# BirdDreams: Joint Perspective and BEV Video Generation for a Plausible, Reactive Driving World Model

**Authors:** Junyoung Seo, Jaewoo Seo, Seungryong Kim, Laura Leal-Taixé, Xiuming Zhang, Seung Wook Kim  
**Lab:** NVIDIA · **Published:** 10 Nov 2026 · **Upvotes:** 50 · **Level:** Intermediate  
**Links:** [arXiv](https://arxiv.org/abs/2611.14815) · [PDF](https://arxiv.org/pdf/2611.14815) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.14815.html)

## Curator's notes

**TL;DR:** BirdDreams jointly generates perspective and BEV videos, predicting layout and enabling ego‑trajectory control without maps or vehicle‑state inputs.

- **What's new:** It combines two video diffusion transformers to co‑generate front‑view and top‑down streams, using the predicted BEV as a soft condition for ego control.
- **Key result:** BirdDreams follows input ego trajectories more faithfully than prior driving world models on PhysicalAI‑AV and nuScenes and yields more plausible ego motion.
- **Why it matters:** You can simulate realistic driving scenes and steer the ego vehicle using only a single camera image, saving the need for HD maps or extra sensors.

## Abstract

Driving world models aim to simulate how a scene unfolds under different ego actions. This requires plausible motion of surrounding vehicles as well as precise ego-trajectory control. Learning vehicle motion from perspective video alone is challenging because motion is entangled with appearance and vehicles can be occluded. Prior work therefore conditions video generation on layouts that specify vehicle motion, providing control over surrounding vehicles but requiring their future trajectories as inputs. We introduce BirdDreams, an autoregressive driving world model that jointly generates perspective video and a schematic bird's-eye-view (BEV) video, making scene layout part of the predicted future. The BEV, rendered from scene annotations for training, depicts roads and vehicles as simple shapes at a fixed metric scale, providing a top-down view of their positions and motion with reduced occlusion between vehicles. Perspective video, by contrast, retains scene details that the BEV omits, such as traffic signals and object appearance. Joint attention between two video diffusion transformers, one per stream, lets each stream inform the other during generation. The predicted BEV also provides a geometric interface for ego control: we translate and rotate its latest frame along an input ego trajectory and use the resulting frames as a soft condition that shows how the scene would appear to move if only the ego vehicle moved. At inference, BirdDreams generates both videos from a single camera image without requiring an HD map or vehicle-state inputs. On PhysicalAI-AV and nuScenes, BirdDreams follows input ego trajectories more faithfully than prior driving world models and generates more plausible ego motion without ego-action conditioning. Human evaluation shows that BirdDreams outperforms a perspective-only baseline in generating realistic surrounding-vehicle behavior.

## What you could build

1. Generate front‑view driving videos from a dashcam image for autonomous policy testing
2. Create BEV layout videos to augment perception training data
3. Build an interactive demo where users sketch ego paths and see joint perspective‑BEV video output

## Quiz yourself

How does BirdDreams incorporate ego trajectory control without using HD maps or vehicle‑state inputs?

**Answer:** It translates and rotates the latest predicted BEV frame along the input ego trajectory and feeds those transformed frames as a soft condition to guide generation.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.