/ Paper arXiv 2611.06910
Curator's notes
FrontierPhysics benchmarks AI agents on full-cycle physics research tasks, revealing current models struggle to produce correct, publishable work.
The new test checks if AI can do all steps of real physics research, and current AIs still fail most of the time.FrontierPhysics benchmarks AI agents on full-cycle physics research tasks, revealing current models struggle to produce correct, publishable work.FrontierPhysics provides an end‑to‑end physics research benchmark; 22 models in 26 configs never exceed 46/168 successful rolloutsrolloutsIndividual runs of an agent attempting a benchmark task..Curator's notesWritten by AI from the paperEvery number here appears in the abstractReport a mistake
What's new
Introduces an end-to-end benchmark covering literature review, planning, experimentation, and manuscript writing across nine physics domains, unlike prior fixed-question tests.
Key result
Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rolloutsrolloutsIndividual runs of an agent attempting a benchmark task..
Why it matters
Builders need realistic evaluation of AI's ability to conduct genuine scientific research, not just answer isolated questions.
Key numbers
From the paperCopied from the abstract, not written by AI.
Abstract
From the paperThe authors' own words.
AI agents are playing an increasingly important role in physics research. However, existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. We introduce FrontierPhysics, an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering nine domains across theoretical, computational, and experimental physics. Each task requires agents to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Combining deterministic verifiersdeterministic verifiersAutomated checks that give the same result each time for a given input. with expert-authored rubrics, FrontierPhysics evaluates whether agents' research is correct and publishable. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. Manual review of agent trajectories shows that frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.
From the full paper
From the paperCopied from the paper itself by our code.
Reading the paper on arXiv…
What you could build
Written by AIAdvancedRequires large‑scale model evaluation and integration of deterministic verifiers with expert rubrics.
- Create a tool that uses the benchmark to fine‑tune language models for research planning.
- Develop a visual dashboard that tracks agent performance on each benchmark sub‑task.
- Implement a curriculum that gradually increases task complexity using FrontierPhysics tasks.
Ask this paper
/ Ask a questionAnswers come from the full paper and name the section
/ Quiz yourself
What is the maximum number of rollouts any model configuration passes in the FrontierPhysics benchmark?
No configuration passes more than 46 of its 168 rollouts, as reported in the abstract.
/ Your notes
Select text in the abstract to highlight it or add a note. They stay in this browser.