A study evaluating ChatGPT’s ability to accurately respond to patient inquiries regarding colon cancer by comparing its responses with assessments from expert clinical oncologists found that questions about symptoms, prevention, and screening for the disease were highly accurate; however, responses related to diagnosis and treatment significantly differed between expert opinions and ChatGPT-generated answers. The study by Ojha et al will be presented during the 2025 ASCO Gastrointestinal Cancers Symposium (Abstract 299).

Study Methodology

To assess the quality of responses generated by ChatGPT to commonly asked questions about colon cancer, researchers compiled 10 comprehensive questions from reliable sources, including the American Society of Colon & Rectal Surgeons, Mount Sinai, the National Cancer Institute, Mayo Clinic, and the American Cancer Society. The questions were categorized into two categories based on their content: (1) general oncology characteristics (covering symptoms, screening, and prevention) and (2) diagnosis and treatment.

The questions were then entered into ChatGPT with prompts designed to simulate patient inquiries. The artificial intelligence (AI)-generated responses were subsequently evaluated by oncology experts using a 5-point Likert scale to assess their accuracy and relevance, with scores reflecting the experts' level of agreement with the answers. On the scale, 1 represents “strongly disagree,” and 5 represents “strongly agree.”

Key Results

The average rating of the responses by oncology experts was 4.72. ANOVA analysis was performed, and there was no statistically significant difference in the mean score across all raters (P = .221). However, ratings between the two categories were statistically significant
(P = .034).

“This study demonstrates that ChatGPT can provide accurate and relevant responses to patient inquiries about colon cancer, as assessed by medical oncology experts. With an average rating of 4.72 on a 5-point Likert scale, ChatGPT’s responses closely align with expert opinion, particularly for [the] general characteristics category, which included symptoms, prevention, and screening. However, responses related to diagnosis and treatment showed a statistically significant difference between expert opinions and AI, indicating that the experts agreed less with this component of AI’s responses. These findings highlight the potential of AI chatbots in supplementing patient education in oncology, though further research is necessary to explore its limitations and expand its clinical utilities,” concluded the study authors.

Clinical Significance

“This research demonstrates the ability of [AI] to enhance patient engagement, support informed decision-making, and potentially address disparities in health-care accessibility. However, it also highlights the limitations of AI, especially in nuanced topics like diagnosis and treatments,” said ASCO expert Laura B. Vater, MD, MPH, Assistant Professor of Clinical Medicine at Indiana University Melvin and Bren Simon Comprehensive Cancer Center, in a statement.

Disclosures: The study authors reported no conflicts of interest.