Evaluating Medical LLMs by Levels of Autonomy: A Survey
Abstract
Medical Large language models achieve strong scores on standard benchmarks; however, the transfer of those results to safe and reliable performance in clinical workflows remains a challenge. This survey reframes evaluation through a levels-of-autonomy lens (L0-L3), spanning informational tools, information transformation and aggregation, decision support, and supervised agents. We align existing benchmarks and metrics with the actions permitted at each level and their associated risks, making the evaluation targets explicit. This motivates a level-conditioned blueprint for selecting metrics, assembling evidence, and reporting claims, alongside directions that link evaluation to oversight. By centering autonomy, the survey moves the field beyond score-based claims toward credible, risk-aware evidence for real clinical use.
Citation
@misc{ye2025evaluatingmedicalllmslevels,
title={Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications},
author={Xiao Ye and Jacob Dineen and Zhaonan Li and Zhikun Xu and Weiyu Chen and Shijie Lu and Yuxi Huang and Ming Shen and Phu Tran and Ji-Eun Irene Yum and Muhammad Ali Khan and Muhammad Umar Afzal and Irbaz Bin Riaz and Ben Zhou},
year={2025},
eprint={2510.17764},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2510.17764},
}