/ Paper arXiv 2611.06918
Curator's notes
NEO-VLA lets action policies use visual features at all network layers, outperforming fixed-encoder models.
The new model lets robots use visual information from any stage, not just the final picture, making them act more accurately.NEO-VLA lets action policies use visual features at all network layers, outperforming fixed-encoder models.NEO-VLA replaces static visual encoders with a patch‑CNN token stream that co‑evolves with language, enabling layer‑wise visual access for action policies.Curator's notesWritten by AI from the paperReport a mistake
What's new
Introduces an encoder-free architecture where vision and language interact throughout the network, unlike prior fixed-visual-encoder pipelines.
Key result
Native architectures consistently outperform encoder-based counterparts across manipulation tasks and distribution shifts.
Why it matters
Builders get more reliable robot control by accessing visual cues early in the model, not just at the final encoder output.
Abstract
From the paperThe authors' own words.
Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expertflow-matching action expertA module that predicts actions by matching predicted visual flow conditioned on language instructions. to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. More strikingly, layer-wise probinglayer-wise probingAnalyzing which network layers contain information most relevant to a downstream task. shows that the action-critical signals often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.
From the full paper
From the paperCopied from the paper itself by our code.
Reading the paper on arXiv…
What you could build
Written by AIAdvancedRequires custom model architecture and joint vision‑language training.
- Create a simulated robot arm that follows language commands using NEO-VLA for real-time grasp adjustments.
- Develop a vision-language game agent that manipulates objects based on textual hints, leveraging layer-wise visual cues.
- Implement a home‑assistant robot that adapts to new objects by fine‑tuning NEO-VLA without a separate visual encoder.
Try it
Code, models and demos that use this paper.
- Models:
- weepiess2383/NEO-VLA ↗1 model on Hugging Face
- Datasets:
- weepiess2383/NEO-VLA-Action-Pretrain ↗1 dataset on Hugging Face
Ask this paper
/ Ask a questionAnswers come from the full paper and name the section
/ Quiz yourself
What architectural change does NEO-VLA make compared to traditional vision-language-action models?
It removes the fixed visual encoder and lets visual tokens be processed jointly with language throughout the network, allowing access to intermediate visual representations.
/ Your notes
Select text in the abstract to highlight it or add a note. They stay in this browser.