# Answer or Defer? Testing VLMs' Stated Confidence on Brain MRI

**Authors:** Amir Sabbaghziarani, Parsa Ghazvinian, Sepehr Salem, Sergey Plis  
**Published:** 9 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.11833) · [PDF](https://arxiv.org/pdf/2611.11833) · [Code](https://github.com/amir-sbg/VLM-Answer-or-Defer) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.11833.html)

## Key numbers

- **5%**: no threshold selects a nonempty set with at most 5% empirical error for any model
- **10%**: At a 10% error budget
- **24.4%**: Gemma-4-12B retains 24.4% of slices and every other model at most 2.5%

## Abstract

Vision-language models (VLMs) could support medical-image interpretation, but their use requires deciding which interpretations to accept and which images to refer to a clinician. We test whether thresholding stated confidence can retain useful coverage while limiting errors among accepted answers, and how this depends on the population. Eight open-weight VLMs, including two reasoning models, assess 3,888 brain MRI slices from post-treatment glioma patients (BraTS 2024) and healthy volunteers (IXI) for tumor or gross abnormality. BraTS labels are mask-derived; IXI slices are treated as negatives. On BraTS, no threshold selects a nonempty set with at most 5% empirical error for any model, even when chosen retrospectively. This also holds for four models evaluated on a large-lesion, distant-negative subset. At a 10% error budget, Gemma-4-12B retains 24.4% of slices and every other model at most 2.5%. To investigate this limited selection ability, we separate confidence differences between predicted answers from differences within each answer. For four models with available outcome-level summaries, pooled mean confidence differs more between answers than between correct and incorrect responses within each answer. A covariance decomposition explains how this pattern can produce opposite confidence-correctness associations across populations. At a 20% error budget, accepting every "tumor" answer covers more slices than any confidence threshold for two models. These findings caution against using unvalidated stated confidence as a standalone deferral signal and support population-specific validation before deployment. Code is available at https://github.com/amir-sbg/VLM-Answer-or-Defer.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.