# Hybrid Computer Use: A Unified Programmatic, Semantic, and Visual Interface for General Agents

**Authors:** Haoqing Wang, Xiaoyi Zhang, Sicheng Fan, Zhexin Zhang, Xinjie Zhang, Jiahao Li, Bin Li, Yan Lu  
**Lab:** Microsoft · **Published:** 30 Oct 2026 · **Upvotes:** 15 · **Level:** Intermediate  
**Links:** [arXiv](https://arxiv.org/abs/2610.41432) · [PDF](https://arxiv.org/pdf/2610.41432) · [Code](https://github.com/microsoft/Hybrid-Computer-Use) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2610.41432.html)

## Curator's notes

**TL;DR:** Hybrid Computer Use (HCU) unifies code, accessibility, and GUI interactions for agents, boosting efficiency and success on OSWorld 2.0.

- **What's new:** HCU combines programmatic, semantic (AX), and visual channels in a single interface, unlike prior single-modality approaches.
- **Key result:** On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among compared systems.
- **Why it matters:** Developers can build agents that choose the most effective interaction mode, reducing steps and improving task completion.

## Abstract

Computer-use agents increasingly tackle tasks ranging from everyday desktop activities to long-horizon professional workflows. Yet interfaces centered on screenshots and mouse-and-keyboard actions can impose a human-interaction prior, requiring repeated screen interpretation while leaving the computer's semantic and programmatic capabilities underexposed. We argue for exposing these capabilities and letting models choose and combine them. We introduce Hybrid Computer Use (HCU), a unified interface for general agents that combines three complementary channels: programmatic interaction through code and command-line tools, semantic interaction through accessibility (AX), and visual interaction through graphical user interfaces (GUI). Our HCU supports efficient AX interaction and guards against stale-state actions during channel switching. A supplementary HCU skill guides agents in selecting and coordinating channels to use the unified interface effectively. Our designed AX tools return compact observations, execute actions in batches, and report their effects as AX diffs, so the model needs fewer round trips. On OSWorld 2.0, HCU achieves the highest Partial and Binary scores among the compared systems. Analyzing agent trajectories, we find that all evaluated models improve with HCU yet allocate work across the channels differently and consistently outperform compared baselines in terms of both efficiency and task success, highlighting the value of unified computer-use interfaces that allow flexible channel selection and combination.

## What you could build

1. Create a desktop assistant that switches between code scripts and GUI clicks based on task complexity.
2. Develop a workflow automation tool that uses AX diffs to batch actions and verify outcomes.
3. Implement a tutoring bot that demonstrates tasks via combined code snippets and visual highlights.

## Quiz yourself

What three interaction channels does HCU combine?

**Answer:** HCU combines programmatic interaction through code/CLI, semantic interaction via accessibility (AX), and visual interaction through the GUI.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.