framework, (2) collecting LLM responses, and then (3) evaluating these responses by comparing between baseline and perturbed
outputs.
Author: Gourabathina, Abinitha; Gerych, Walter; Pan, Eileen; Ghassemi, Marzyeh
Description: The integration of large language models (LLMs) into clinical diagnostics necessitates a careful understanding of how clinically irrelevant aspects of user inputs directly influence generated treatment recommendations and, consequently, clinical outcomes for end-users. Building on prior research that examines the impact of demographic attributes on clinical LLM reasoning, this study explores how non-clinically relevant attributes shape clinical decision-making by LLMs. Through the perturbation of patient messages, we evaluate whether LLM behavior remains consistent, accurate, and unbiased when non-clinical information is altered. These perturbations assess the brittleness of clinical LLM reasoning by replicating structural errors that may occur during electronic data processing patient questions and simulating interactions between patient-AI systems in diverse, vulnerable patient groups. Our findings reveal notable inconsistencies in LLM treatment recommendations and significant degradation of clinical accuracy in ways that reduce care allocation to patients. Additionally, there are significant disparities in treatment recommendations between gender subgroups as well as between model-inferred gender subgroups. We also apply our perturbation framework to a conversational clinical dataset to find that even in conversation, LLM clinical accuracy decreases post-perturbation, and disparities exist in how perturbations impact gender subgroups. By analyzing LLM outputs in response to realistic yet modified clinical contexts, our work deepens understanding of the sensitivity, inaccuracy, and biases inherent in medical LLMs, offering critical insights for the deployment of patient-AI systems.
Subject headings: Large language models; LLM; Medicine; Communication; Treatment recommendations; Clinical decision-making; Accuracy; Disparities; Gender; Patient-AI systems; Artificial intelligence
Publication year: 2025
Journal or book title: FAccT ’25: Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency
Pages: 1805-1828
Find the full text: https://www.strategian.com/fulltext/Gourabathina2025.pdf
Find more like this one (cited by): https://scholar.google.com/scholar?cites=6386966134442450292&as_sdt=1000005&sciodt=0,16&hl=en
Serial number: 4161