/ Paper arXiv 2611.07765
In short
In shortReport a mistake
Abstract
From the paperThe authors' own words.
Graphical User Interface (GUI) grounding aims to map natural language instructions to precise operational coordinates, serving as a critical task for autonomous agent interaction. Current approaches formalize GUI grounding as predicting action-coordinate pairs from instructions, with end-to-end training supervised solely by final coordinates. Training datasets typically provide only a small number of descriptions per target coordinate, whereas real-world scenarios involve a much larger set of varied descriptions for the same target, which limits robustness to linguistic variation. While recent GUI grounding paradigms have begun employing reinforcement learning to enhance localization accuracy, models with limited exploratory capability before reinforcement yield constrained improvements even with identical training data during the reinforcement phase. To overcome these limitations, we propose a bidirectional reward paradigm that enables model self-reinforcement. During SFT, bidirectional training equips the model to generate action descriptions from target coordinates. During RL, the model samples semantically consistent instruction variations on-policy and grounds each variation back to its target. A cycle-consistency gate couples semantic quality to grounding success, turning the model's own instruction policy into a source of continually refreshed reinforcement trajectories. Our 4B-parameter model achieves competitive accuracy across five grounding benchmarks in our evaluation, indicating the effectiveness of the proposed training paradigm. https://github.com/Spell-shi/BiGrounder
From the full paper
From the paperCopied from the paper itself by our code.
Reading the paper on arXiv…
Try it
Code, models and demos that use this paper.
- Code:
- ★ 2 on GitHub ↗
Ask this paper
/ Ask a questionAnswers come from the full paper and name the section
/ Your notes
Select text in the abstract to highlight it or add a note. They stay in this browser.