# WayPOP: A Panoramic Open-Set Panoptic Tracking Benchmark

**Authors:** Rohit Mohan, Maximilian Luz, Swarnava Chowdhury, Yao Lu, Florian Drews, Thomas Nürnberg, Yakov Miron, Federico Tombari, Stefano Gasperini, Abhinav Valada  
**Lab:** University of Freiburg · **Published:** 4 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.05801) · [PDF](https://arxiv.org/pdf/2611.05801) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.05801.html)

## Key numbers

- **53,452**: yielding 53,452 unknown-object mask annotations across 21,300 camera images
- **21,300**: yielding 53,452 unknown-object mask annotations across 21,300 camera images

## Abstract

Holistic perception for autonomous agents requires reliably segmenting and associating scene elements across time and camera views, including objects outside a fixed semantic taxonomy. Existing open-set panoptic benchmarks focus on individual images or views, while panoramic tracking benchmarks assume closed-set taxonomies. Consequently, these benchmarks do not evaluate whether unknown objects can be consistently segmented and tracked across synchronized multi-camera sequences. We introduce WayPOP, a panoramic open-set panoptic tracking benchmark built on real Waymo PVPS sequences augmented with diverse out-of-taxonomy 3D assets from Objaverse. Inserted objects are placed at fixed locations in the 3D scene and rendered using calibrated vehicle-mounted cameras as the ego vehicle and surrounding traffic move, providing temporally and cross-view consistent panoptic annotations and auxiliary LiDAR range observations. WayPOP contains approximately 670 curated assets instantiated as 980 unique unknown-object tracks, yielding 53,452 unknown-object mask annotations across 21,300 camera images. Our evaluation protocol defines a hierarchy of tasks from view-independent open-set panoptic segmentation to full panoramic open-set panoptic tracking, using known/unknown PQ and (w)STQ metrics. We benchmark representative open-set and open-vocabulary segmentation methods with temporal and cross-camera association. Our results reveal substantial limitations of current methods in truly open-set scenarios, highlighting robust unknown-object localization and identity association across time and views as major open challenges.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.