In the first study of its kind, researchers evaluated the reliability and accuracy of ChatGPT’s cancer information, as chatbots and artificial intelligence (AI) have become popular resources. A report published by Johnson et al in JNCI Cancer Spectrum summarized the accuracy of the tool in providing accurate information about common cancer myths and misconceptions.
Using the National Cancer Institute’s (NCI) common myths and misconceptions about cancer webpage as a comparator, the research team found that 97% of the answers provided by ChatGPT were correct. Following expert review, the percentage of overall agreement for accuracy was 100% for NCI answers and 96.9% for ChatGPT outputs (ĸ = ‒0.03, standard error = 0.08).
However, these findings come with some important caveats, including a concern among the team that some of the ChatGPT answers could be interpreted incorrectly. “This could lead to some bad decisions by patients. The team suggested caution when advising patients about whether they should use chatbots for information about cancer,” said first study author Skyler Johnson, MD, a physician-scientist at Huntsman Cancer Institute and Assistant Professor in the Department of Radiation Oncology at the University of Utah.
The study reviewers were blinded, ie, they didn’t know whether the answers came from the chatbot or the NCI. Though the answers were accurate, reviewers found ChatGPT’s language was indirect, vague, and in some cases, unclear.
“I recognize and understand how difficult it can feel for patients [with cancer] and caregivers to access accurate information,” said Dr. Johnson. “These sources need to be studied so that we can help patients navigate the murky waters that exist in the online information environment as they try to seek answers about their diagnoses.”
The researchers' next steps are to evaluate how often patients are using chatbots to seek out information about cancer, what questions they are asking, and whether AI chatbots provide accurate answers to uncommon or unusual questions about cancer.
Disclosure: The study was supported by the National Institutes of Health/National Cancer Institute and the Huntsman Cancer Foundation. For full disclosures of the study authors, visit academic.oup.com/jncics.
