Simplified oncologic computed tomography (CT) reports using large language models (LLMs) enabled patients to better understand the results of their restaging CT scans and reduced overall reading burden, according to the results of a study published in Radiology. However, the study also revealed clinically relevant errors introduced by the LLM.
"Ensuring that patients understand their reports, examinations, and treatments is a central pillar of modern medicine. This is the only way to guarantee informed consent and strengthen health literacy," said corresponding study author Felix Busch, MD, Assistant Physician at the Institute for Diagnostic and Interventional Radiology, Technical University of Munich School of Medicine and Health, Munich, Germany.
Study Methods
Researchers conducted a prospective, controlled, open-label, quasi-randomized trial to determine if using LLMs to simplify oncologic CT reports could improve patients' overall understanding of the report and reduce the reading burden in terms of cognitive workload, text comprehension, and reading time.
A total of 200 adult patients with cancer who had undergone routine CT restaging were enrolled in the study. Participants were alternately assigned to receive standard CT reports or reports simplified by LLM, using Llama 3.3 70B (Meta). Radiologist review was mandatory for the LLM-simplified reports.
Outcomes were measured through participant-reported scores and composite scores.
Key Study Findings
Participants who received standard CT reports spent a median of 7 minutes reading the report, which was reduced to 2 minutes with the simplified report using LLMs (P < .001).
Those who received the LLM-simplified reports noted lower cognitive workload (adjusted odds ratio [aOR] = 0.18; 95% confidence interval [CI] = 0.13–0.25; P < .001), better comprehension of the report (aOR = 13.28; 95% CI = 9.31–18.93; P < .001), and better perception of the usefulness of the report (aOR = 5.46; 95% CI = 3.55–8.38; P < .001) than patients who received standard CT reports.
The LLM-simplified reports also improved readability by lowering the mean Flesch-Kincaid Grade Level for understanding (8.89 ± 0.93 vs 13.69 ± 1.13; P < .001).
Radiologist reviews found factual errors in 6% of the LLM-simplified reports and content omissions in 7%—4% of which were severe in both cases—as well as inappropriate additions in 3%.
“Aside from data protection concerns, language models always carry the risk of factual errors,” concluded lead study author Philipp Prucker, MD, Institute for Diagnostic and Interventional Radiology, Technical University of Munich School of Medicine and Health. “Language models are useful tools, but they are no substitute for medical staff. Without trained specialists verifying the findings, patients may, in the worst case, receive incorrect information about their illness.”
The researchers stressed that patients should not turn to chatbots like ChatGPT to simplify their report without the assistance or review of a physician.
Disclosure: For full disclosures of the study authors, visit pubs.rsna.org.
