All papersReasoning & Agents

ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces

/ Paper arXiv 2611.14364

Key numbers

66.2%the strongest passes every checklist item on only 66.2% of tasks
69%Three models pass about 69% of tasks when only delivery contracts and general norms are checked
28.4%but just 28.4% to 40.0% once historical and material or workspace requirements are also checked

From the paperCopied from the abstract, not written by AI.

Abstract

From the paperThe authors' own words.

Research agents achieve increasingly strong benchmark scores, yet these scores do not necessarily translate into reliable delivery in ongoing research projects. Many research-agent benchmarks emphasize standalone tasks and relatively self-contained workflows in clean or curated workspaces. However, researchers ask agents to complete one step within an ongoing project, where requests depend on earlier session decisions and workspaces contain unfinished files, outdated versions, and failed attempts. We introduce ARE-bench to evaluate whether research agents advance ongoing projects while respecting prior decisions and existing workspace state, grounded in real session context and in-progress workspaces. ARE-bench includes 75 tasks built on agent sessions and full workspaces from recruited researchers' own projects, spanning seven research stages, 33 requirements, and 150 sub-requirements, with 1,173 binary checklist items in total. Each task is anchored at a real user request, preserves its requirements, and summarizes the earlier requests still in effect as background. The complete workspace is replayed to that boundary without cleanup, preserving artifacts from earlier turns. Each checklist is compiled from an intent-aligned rubric and credits any valid solution. Across seven frontier models, the strongest passes every checklist item on only 66.2% of tasks, and Design and Execution rank among the hardest stages for every model. Three models pass about 69% of tasks when only delivery contracts and general norms are checked, but just 28.4% to 40.0% once historical and material or workspace requirements are also checked. Evaluating delivery without these contextual requirements can therefore overestimate research agents and mask differences between them.

From the full paper

From the paperCopied from the paper itself by our code.

Reading the paper on arXiv…

Where it fits

The most cited papers it builds on, and the papers that build on it.

/ Builds on 3 most cited

  1. 20028.2k citedA Revision of Bloom's Taxonomy: An Overview
  2. 20234.9k citedSWE-bench: Can Language Models Resolve Real-World GitHub Issues? ↗
  3. 20241.8k citedτ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains ↗

/ Built on by None yet

No papers cite it yet. We check again every week.

From Semantic Scholar, updated weekly.

Ask this paper

/ Ask a questionAnswers come from the full paper and name the section

    / Your notes

    Select text in the abstract to highlight it or add a note. They stay in this browser.

      Read the PDF