A study evaluating ChatGPT’s ability to accurately respond to patient inquiries regarding colon cancer by comparing its responses with assessments from expert clinical oncologists found that questions about symptoms, prevention, and screening for the cancer were highly accurate. However, responses related to diagnosis and treatment were significantly different between expert opinions and ChatGPT-generated answers.1 The study, conducted by Sujata Ojha, MD, of Dell Medical School, The University of Texas at Austin, and colleagues, was presented during the 2025 ASCO Gastrointestinal Cancers Symposium.1
Study Methodology
To assess the quality of responses generated by ChatGPT to commonly asked questions about colon cancer, the researchers compiled 10 comprehensive questions from reliable sources, including the American Society of Colon and Rectal Surgeons, Mount Sinai, the National Cancer Institute, Mayo Clinic, and the American Cancer Society. The questions were separated into two categories based on their content: general oncology characteristics (covering symptoms, screening, and prevention) and diagnosis and treatment.
Sujata Ojha, MD
The questions were then entered into ChatGPT with prompts designed to simulate patient inquiries. The artificial intelligence (AI)-generated responses were subsequently evaluated by oncology experts using a 5-point Likert scale to assess their accuracy and relevance, with scores reflecting the experts’ level of agreement with the answers. On the scale, 1 represents “strongly disagree,” and 5 represents “strongly agree.”
Key Results
The average rating of the responses by oncology experts was 4.72. ANOVA (analysis of variance) analysis was performed, and there was no statistically significant difference in the mean score across all raters (P = .221). However, ratings between the two categories were statistically significant (P = .034).
“This study demonstrates that ChatGPT can provide accurate and relevant responses to patient inquiries about colon cancer, as assessed by medical oncology experts. With an average rating of 4.72 on a 5-point Likert scale, ChatGPT’s responses closely align with expert opinion, particularly for [the] general characteristics category, which included symptoms, prevention, and screening. However, responses related to diagnosis and treatment showed a statistically significant difference between expert opinions and AI, indicating that the experts agreed less with this component of AI’s responses. These findings highlight the potential of AI chatbots in supplementing patient education in oncology, though further research is necessary to explore its limitations and expand its clinical utilities,” concluded the study authors.
Clinical Significance
“This research demonstrates the ability of artificial intelligence to enhance patient engagement, support informed decision-making, and potentially address disparities in
Laura B. Vater, MD, MPH
health-care accessibility. However, it also highlights the limitations of AI, especially in nuanced topics like diagnosis and treatments,” said ASCO expert Laura B. Vater, MD, MPH, Assistant Professor of Clinical Medicine at Indiana University Melvin and Bren Simon Comprehensive Cancer Center, in a statement.
DISCLOSURE: The study authors reported no conflicts of interest.
REFERENCE
1. Ojha S, Sessions WS, Goodgame B: Man versus machine: Comparative analysis of ChatGPT’s colon cancer guidance and expert opinion. 2025 ASCO Gastrointestinal Cancers Symposium. Abstract 299. Presented January 23, 2025.
