KENNETH R. CARSON, MD, PhD
THOMAS WILSON, PhD, DrPH
CHARLES L. BENNETT, MD, PhD, MPP, MS (Law), FASCO
Real-world data (RWD) provide raw material from which real-world evidence (RWE) is generated, but the two terms are not synonymous. Over the past decade, RWD and RWE have become increasingly prominent in oncology. A search of the 2016 ASCO Annual Meeting abstracts identified 48 with terms “real world” or “real-world” in the title, compared with 794 in 2026—a 16.5-fold increase in 10 years. Whereas there were likely additional abstracts using real world data that did not contain those words in the title, this comparison illustrates the extraordinary growth of real-world research in oncology.
As physicians and health services researchers who have spent our careers evaluating real-world clinical outcomes in academia and industry, we welcome this growth and are enthusiastic about its potential to improve patient care. Our concern arises not from use of RWD itself, but from how studies using RWD are designed, analyzed, and interpreted, ultimately determining whether resulting evidence can credibly inform patient care.
Although the terms RWD and RWE are relatively new, the fundamental challenges associated with retrospective observational research have been recognized for decades. Most notably, both bias and residual confounding can produce clinical conclusions that are misleading or, in some cases, simply wrong.
Methodologic Rigor
The use of carefully designed RWD analyses in regulatory settings demonstrates that carefully conducted observational studies can generate meaningful evidence. These analyses must be accompanied by extensive assessments of data quality, prespecified analytic plans, sensitivity analyses, validation on independent datasets, and careful statistical reviews. Although it is possible to apply the same rigor to a basic science or clinical research article to a RWD-based article with detailed methods and supplementary materials, this information is not consistently requested, provided, or assessed. Assessing methodologic rigor is even more challenging with meeting abstracts, where the rapid growth of RWD research is most visible and methods are often condensed into a handful of sentences.
A first level of concern is whether underlying data are accurate and sufficiently complete to answer the clinical question. RWD may be collected or derived from a variety of structured fields, claims, manual chart abstraction, natural language processing, or AI-based methods. Each approach has strengths and weaknesses; clinically important data or even entire variables may be missing, incompletely captured, or misclassified. Whereas some types of data missingness can be addressed by modern analytic methods, these approaches cannot fully address misclassification or compensate for information that was simply not collected. The consequences can be substantial; one of us (TW) demonstrated in his dissertation that data patterns can be misclassified and could result in incorrect clinical diagnoses. Transparency regarding how key variables were collected, derived, and validated, including available performance measures, is critical to determine if the data are fit for purpose.
A second level of concern is whether available data adequately capture important potential confounders. In addition to exposures and outcomes, observational studies must account for patient characteristics that could influence treatment selection and prognosis. Comorbid conditions are a prime example. Two of us demonstrated in our dissertation research that comorbidities influenced treatment decisions and outcomes in lymphoma (KRC) and prostate cancer (CLB). Without this information, our studies could have produced different conclusions. No degree of statistical sophistication can fully account for an important confounder that was never measured. This can occur even in pristine randomized controlled trials when results are extrapolated to the real world.
Professional Judgement
The methods used to analyze RWD must also be appropriate for the clinical question being addressed. Increasingly used techniques, including machine learning and AI, can identify complex relationships and make predictions that may not have been observable with traditional statistical methods. Of course, it is impossible for predictions to be 100% accurate, thus there is a need for independent validation in real world settings. However, accurately predicting an at-risk population or an outcome is different from determining if a treatment caused that outcome in that population. The familiar adage remains true: association does not establish causation. Translated into clinical language, sophisticated analytics do not necessarily produce more valid clinical conclusions. Professional judgement is necessary in our healing profession.
Even with an understanding of the limitations of observational studies, a busy clinician may overinterpret a study that addresses an important clinical question in an easily digestible way. Kaplan-Meier curves provide a familiar example. In a randomized controlled trial, treatment assignment helps ensure that differences in outcomes between treatment groups can reasonably be attributed to the treatments being compared. Similar-looking curves generated from RWD may be compelling, but unless the confounders are equivalent (as in randomized controlled trials),the treatment groups may include patients with important underlying differences that also influence their outcomes. Statistical adjustment can reduce these imbalances but cannot fully eliminate them, particularly when important clinical characteristics were not measured. Authors, reviewers, and editors therefore share responsibility for ensuring that the conclusions of observational comparative-effectiveness studies reflect these limitations.
Critical Appraisal
Clinicians are increasingly using large language model (LLM)–based tools to search and synthesize the medical literature, resulting in the final challenge we highlight. For a busy oncologist, these tools can rapidly identify studies relevant to a specific clinical question and summarize the findings in their training data sets (that were not always pristine studies). However, an accurate study summary does not always convey how much confidence should be placed in the conclusions. Evidence-grading approaches may separate higher from lower levels of evidence, but they do not necessarily distinguish the methodologic rigor of one observational study from another. Differences in study design, sensitivity analyses, and validation may only be apparent from a careful review of methods, appendices, or supplementary materials. Thus, accurate retrieval and summarization of a study should not be confused with critical appraisal of the evidence.
After reviewing challenges described herein, one might think we are skeptical of RWD and observational research more broadly. Quite the contrary, we are bullish on the potential of RWD to generate credible RWE and therefore improve patient care. Observational research can address important questions that cannot practicably be answered in randomized trials and provide insights into patient populations, treatments, and outcomes not adequately studied in clinical trials.Researchers, regulators, editors, journal reviewers, and, most importantly, clinicians share responsibility for ensuring that the evidence generated from RWD is sufficiently valid to produce RWE to inform oncology patient care. As our ability to generate and rapidly synthesize RWD continues to grow, the need for these guardrails will only increase. The future of RWE is bright, but the availability of more RWD—and increasingly powerful tools to analyze and summarize it—makes rigor in transforming data into real world evidence more important, not less. ν
DISCLOSURE: Drs. Bennett and Wilson reported no conflicts of interest. Dr. Carson holds an equity interest with Vaalia Health.
Kenneth R. Carson, MD, PhD is Chief Medical Officer and co-founder at Vaalia Health, Inc. a health-care start-up company working to support patients initiating oncology therapies. Previously, he was the Vice-President of Clinical Solutions for then the privately held Tempus Labs, Inc (following the June 2024 Initial Public Offering it is called Tempus AI). Concurrent with the Tempus position, Dr. Carson was a lymphoma specialist at the Rush University Cancer Center in Chicago. Charles L. Bennett, MD, PhD, MPP, MS (Law), FASCO is the Frank P and Josie M Fletcher Chair for Medication Safety and the Director of the SmartState Center for Medication Safety and Efficacy for the University of South Carolina College of Pharmacy. He is also an Affiliated Investigator to the Johns Hopkins/Bloomberg School of Public Health/ Arnold Ventures Drug Access and Affordability Program. Thomas Wilson, PhD, DrPH is an independent, pragmatic epidemiologist specializing in the design, evaluation, and science-based improvement of population health programs and AI-driven predictive health solutions. He is the co-founder of Trajectory® Health AI in Annapolis, Maryland (https://trajectoryhealth.ai/) and currently serves as their Chief Epidemiologist.
Disclaimer: This commentary represents the views of the author and may not necessarily reflect the views of ASCO or The ASCO Post.
