Over the past 2 years, IBM’s Watson for Oncology cognitive computing system, which uses artificial intelligence (AI) algorithms to generate treatment recommendations, has come under fire for allegedly not delivering on expectations to provide state-of-the-art personalized treatment for patients with cancer and for producing advice that is “unsafe and incorrect.”1,2 Since its inception in 2012, Watson for Oncology has undergone intensive medical training by oncologists at Memorial Sloan Kettering Cancer Center (MSK) in New York to learn key data associated with a patient’s cancer, such as results from blood tests; pathology and imaging reports detailing the type, size, and location of the tumor; and the presence of genetic mutations. The program then combs through huge amounts of medical literature to provide evidence-based treatment recommendations for that specific patient. To date, Watson has been trained to support treatment decisions in lung, breast, colorectal, bladder, cervical, endometrial, esophageal, stomach, ovarian, prostate, and thyroid cancers, and training will begin soon in several types of lymphomas.
In addition to Watson for Oncology, IBM has launched several other oncology-related offerings aimed at accelerating personalized patient care. They include Watson for Genomics, Watson Genomics from Quest Diagnostics (a Watson-powered genomic-sequencing service), and IBM Watson for Clinical Trial Matching, which uses natural language processing to increase the efficiency and accuracy of the clinical trial matching process.
Nathan Levitan, MD, MBA
According to Nathan Levitan, MD, MBA, Chief Medical Officer at IBM Watson Health Oncology and Genomics, Watson for Oncology and its oncology-related products are in use in hundreds of hospitals and health-care systems in more than a dozen countries. In a wide-ranging interview with The ASCO Post, Dr. Levitan discussed the criticisms plaguing Watson for Oncology and how they are being addressed, how Watson for Oncology is impacting treatment decision-making, and the potential ethical pitfalls of cognitive computing in health care.
Technical Challenges
There have been news reports of instances of Watson for Oncology delivering inaccurate treatment information to physician-users. Are there training problems with Watson, and have the issues been resolved?
Allow me to begin at a high level by identifying the specific challenges in the delivery of cancer care Watson Health is focusing on and the ways in which we believe technology-based tools can help address them. Cancer is the second leading cause of death globally, and an estimated 9.6 million people died from cancer in 2018.3 The rise in cancer death rates can be approached in two ways: through screening and prevention and by ensuring that patients diagnosed with cancer receive state-of-the-art care.
With respect to the delivery of state-of-the-art care, the medical literature shows that there is considerable variability, both in the United States and globally. One component of this variability is the challenge that physicians face in remaining up-to-date in the multidisciplinary management of all types of cancer. A second is the proliferation of cancer-related publications. Of course, in some countries, resource constraints are a factor as well.
“We have an extensive quality management system that tests every version of every product before it is released and that also responds to any reports from users in the field regarding system performance.”— Nathan Levitan, MD, MBA
Tweet this quote
I am optimistic that technical solutions that utilize cognitive computing systems like Watson for Oncology can mitigate some of these challenges. At Watson Health, we are privileged to collaborate with and learn from our clinical partners at MSK, as well as with the physicians who use Watson for Oncology every day.
Regarding your specific question, a media report from one news outlet indicated Watson output included isolated erroneous treatment recommendations1 among the tens of thousands of correct recommendations. They were identified by our own quality process and were corrected before ever reaching a patient.
We have an extensive quality management system that tests every version of every product before it is released and that also responds to any reports from users in the field regarding system performance. As with any health-care quality system, ours is both proactive and responsive.
Providing Treatment Decision Support
Because Watson is being trained by MSK, using the hospital’s patient and treatment data system, one of the criticisms against Watson is that the information it produces is biased toward MSK’s cancer treatments and may make recommendations that are inappropriate for patients in other countries and even among other cancer institutions in the United States. How do you respond to that?
Knowing how rapidly cancer treatment options change and the importance of keeping the system up-to-date, Watson for Oncology training is not based on an analysis of historical patient records in hospital data systems. Such an approach could indeed introduce treatment bias based on specific patients who happen to be treated at MSK, for instance, and it also might not reflect recent practice changes that occur as a result of emerging medical literature. Accordingly, our approach is to train Watson with “synthetic” patient cases, developed by our partners at MSK, which specifically reflect state-of-the-art oncology care.
In addition to utilizing clinical decision support based on MSK subspecialty physician expertise, Watson integrates literature references recommended by MSK oncologists, relevant literature identified by AI-driven curation, and clinical trial matching. We are continuously adding oncology care guidelines that are used both in the United States and in Asia, including those from the National Comprehensive Cancer Network and the Chinese Society of Clinical Oncology, among others.
When Watson for Oncology is used in countries outside the United States, we localize recommendations to suit that environment, such as drug availability, dosing units, and language translation. We recognize that the needs of physicians around the globe vary, and each physician will use the types of information produced by Watson that are most relevant to his or her patients’ needs. And with each release of Watson for Oncology, we add clinical content and enhance user functionality.
Watson for Oncology is intended to support treating oncologists through the provision of information. We wish to preserve the autonomy of physician decision-making and enable oncologists to make the optimal treatment decisions for—and with—their patients.
“We recognize that the needs of physicians around the globe vary, and each physician will use the types of information produced by Watson that are most relevant to his or her patients’ needs.”— Nathan Levitan, MD, MBA
Tweet this quote
Improving Access to Care in Lower- to Middle-Income Countries
During the 2019 ASCO Annual Meeting, IBM Watson Health presented 22 studies showing progress in how Watson for Oncology and Watson for Genomics are being used globally. In one randomized study evaluating the role of AI in clinical decision-making for 1,000 patients in India diagnosed with breast, lung, and colorectal cancers, members of a multidisciplinary tumor board at Manipal Comprehensive Cancer Center in Bangalore changed their treatment decisions in 13.6% of the cases based on information provided by Watson.4 Please talk about this finding and its potential impact on oncology care.
The findings from this study are important because they show that a decision-support tool not only provides treatment information, it can also influence decision-making. In this case, the tumor board changed its treatment decisions because Watson provided recent evidence for newer treatment (55%), more personalized alternative treatment (30%), and new insights from genotypic and phenotypic data and evolving clinical experiences (15%).
Using AI to Increase Clinical Trial Participation
In 2018, a study assessed the ability of Watson for Clinical Trial Matching to match patients with breast cancer being treated at the Mayo Clinic in Rochester, Minnesota, with an appropriate clinical trial. The study found that over an 18-month period, there was, on average, an 84% increase in trial enrollment.5 Currently, fewer than 5% of patients with cancer enroll in clinical trials.6 How can Watson help increase participation in clinical trials?
As you note, the rates of patient enrollment in clinical trials are low in the United States, and there are disparities in enrollment across demographic subsets of the population. One barrier to clinical trial enrollment is the difficulty busy oncologists have finding an appropriate clinical study for a specific patient. A second barrier is finding the time it takes to enroll a patient. And a third is the variable availability of clinical trials across practice locations.
Watson for Clinical Trial Matching can ingest information about thousands of open clinical trials, extract the eligibility requirements, and then match patients with trials. In the Mayo Clinic study, over the 18 months following implementation, 6.3 patients per month were enrolled in a group of breast cancer trials compared to 3.5 patients per month in the period prior to using Watson.
In another study of patients with breast and lung cancers in a community practice setting, Watson for Clinical Trial Matching was able to reduce the amount of time it took to screen these patients by 78%: 1 hour and 50 minutes by manual curation vs 24 minutes using Watson.7
“There is reason for excitement about leveraging technology to support the delivery of high-quality cancer care.”— Nathan Levitan, MD, MBA
Tweet this quote
Ethical Dimensions of AI in Oncology Care
Although AI technology offers the opportunity to improve the efficiency of health-care delivery and quality of patient care, it also presents ethical risks, including threats to patient privacy, informed consent, and patient autonomy. There is also concern that data used to create algorithms can contain bias, such as patients’ ability to pay for a specific treatment or insurance status, which can impact clinical recommendations. Do you have any ethical concerns about this technology and its role in oncology care?
IBM has been the leader in data security across the industries it serves and takes data responsibility very seriously. In establishing Watson Health in Oncology several years ago, IBM utilized its technology and expertise to address issues such as patient privacy, HIPAA [Health Insurance Portability and Accountability Act] compliance, and data security.
When machine learning is utilized, it often looks at past experience, learns from a data set, and then creates an algorithm with which to analyze new data points. The potential for bias occurs when a system is trained with an existing set of data that was produced with bias. Watson for Oncology is not trained in that way, so the potential risk for introducing this type of bias is mitigated.
Through advances in data science, AI and machine learning offer an opportunity to support the delivery of consistent evidence-based care in a way that preserves the autonomy of the oncologist, leverages the oncologist’s skills to the fullest, and empowers patients to participate in shared decision-making.
Solutions of this kind have the potential to achieve global scale and global impact in oncology care. There is reason for excitement about leveraging technology to support the delivery of high-quality cancer care. ■
DISCLOSURE: Dr. Levitan is Chief Medical Officer at IBM Watson Health Oncology and Genomics.
REFERENCES
1. Ross C, Swetlitz I: IBM’s Watson supercomputer recommended “unsafe and incorrect” cancer treatments, internal documents show. STAT. July 25, 2018. Available at www.statnews.com/wp-content/uploads/2018/09/IBMs-Watson-recommended-unsafe-and-incorrect-cancer-treatments-STAT.pdf. Accessed August 26, 2019.
2. Hernandez D, Greenwald T: IBM has a Watson dilemma. The Wall Street Journal. August 11, 2018. Available at www.wsj.com/articles/ibm-bet-billions-that-watson-could-improve-cancer-treatment-it-hasnt-worked-1533961147. Accessed August 26, 2019.
3. World Health Organization: Cancer: Key facts. Available at www.who.int/news-room/fact-sheets/detail/cancer. Accessed August 26, 2019.
4. Somashekhar SP, Sepúlveda MJ, Shortliffe, et al: A prospective blinded study of 1,000 cases analyzing the role of artificial intelligence: Watson for Oncology and change in decision-making of a multidisciplinary tumor board from a tertiary care cancer center. 2019 ASCO Annual Meeting. Abstract 6533. Presented June 1, 2019.
5. Haddad TC, Helgeson J, et al: Impact of a cognitive computing clinical trial matching system in an ambulatory oncology practice. J Clin Oncol 36(15 suppl):6550, 2018.
6. Unger JM, Vaidya R, Hershman DL, et al: Systematic review and meta-analysis of the magnitude of structural, clinical, and physician and patient barriers to cancer clinical trial participation. J Natl Cancer Inst 111:245-255, 2019.
7. Beck IT, Vinegra M, Dankwa-Mullan I, et al: Cognitive technology addressing optimal cancer clinical trial matching and protocol feasibility in a community cancer practice. J Clin Oncol 35(15 suppl):6501, 2017.
