Randomized clinical trials are highly regulated initiatives that must comply with multiple requirements while maintaining high epistemic standards, a balance that becomes increasingly difficult as the research questions surrounding immunotherapy and targeted agents become more complex. To shed light on this important issue, The ASCO Post recently spoke with Jonathan Kimmelman, PhD, of McGill University in Montreal. Among his current research activities, Dr. Kimmelman is investigating risk prediction, validity, and knowledge value across the trajectory of drug development.
Research Ethics
Please tell the readers about your current position and work.
I’m the James McGill Professor in the Biomedical Ethics Unit of McGill University. My research centers on the ethical, social, and policy challenges in testing novel medical technologies in human beings. I run a research group called STREAM—from studies of translation, ethics, and medicine—where we apply a variety of empiric and analytic methods to better understand the ethical as well as scientific dimensions of drug development. The majority of my work is focused on cancer, although I work in other disease areas as well.
The bottom line from our study is that even experts have a really difficult time discriminating between treatments in trials that are likely to show a positive effect and those that won’t.— Jonathan Kimmelman, PhD
Tweet this quote
Medicine is saturated with ethical challenges, and our task as ethicists is to help physicians, policymakers, and patients use rigorous thinking and evidence to navigate difficult ethical challenges. My study area is called research ethics, in particular clinical trial ethics. We try to develop analysis and evidence addressing recurrent ethical challenges encountered in developing novel treatments. Some challenges are obvious, such as those surrounding valid informed consent. Many, however, may not be obvious to the uninitiated. I view my job as discovering latent issues that we may not have noticed but are critical to the ethical implementation of clinical cancer trials.
Predicting Clinical Efficacy
You were part of a study that looked at whether the expert opinion of oncologists could reliably predict the efficacy of cancer treatments tested in randomized controlled trials. Please describe the design of the study and the results.
Trials involve a lot of expert judgment. For instance, in cancer research, there aren’t enough patients and resources to test every drug or hypothesis we might want to. Instead, we need to prioritize. In that context, it’s important to know how good experts are at prioritizing the most promising treatments for testing. Decisions to launch trials are generally made by expert committees that have reviewed prior clinical and preclinical evidence. Peer review panels at granting agencies typically scrutinize supporting and pilot evidence when selecting trials for funding.
In our study, we took a sample of active clinical trials that hadn’t announced results and presented them to a panel of experts specializing in genitourinary, lung, and colorectal cancers. We asked them to predict the probability that the trials would be positive on their primary endpoint.
In a nutshell, we found that experts in the specific malignancies did poorly at predicting the outcomes of randomized clinical trials. In fact, they would have done better if they had simply abstained from offering predictions. To put this in context, think about being told there is a 50% chance of rain tomorrow; it really doesn’t give you enough information about whether to bring an umbrella or not. In effect, it’s maximum uncertainty. In fact, on average, experts we looked at were worse at predicting success than guessing 50% all the time. The bottom line of our study is that even experts have a difficult time discriminating between treatments in trials that are likely to show a positive effect and those that won’t.
However, there are a couple of provisos to the findings. To begin, it’s very tricky to conduct studies of this type because of various confounding elements. For instance, if we’d better incentivized the experts for giving their accurate predictions, that might have changed the results. Like any first study, the results need to be replicated. We have published one study already supporting this finding. But we are currently writing up results of another study where oncologists showed surprisingly high skill in predicting trial outcomes. We need more research in this very complex decision-making setting before using our findings to inform clinical trial policy.
Practical Implications
Are there real-world applications derived from the results of your study?
If, in fact, further studies were to confirm our results, it would suggest that we need to be very wary of policymaking that relies on low-quality evidence and/or expert judgment. For instance, clinical practice guidelines in cancer contain numerous recommendations based either on expert judgment or investigations that are less than rigorous. Many of these recommendations are used to guide reimbursement for Medicare. If our findings are confirmed, it suggests Medicare is paying for a lot of unnecessary, low-value cancer services.
Oncologists are clearly clinical care experts, but we need to find better ways to tap that expertise. The downstream goal of my work is to develop better techniques for harnessing the best possible judgments for maximizing the medical gain per patient enrolled in trials.
Clinical Trial Failure Rate
U.S. Food and Drug Administration (FDA) approval currently has an extremely high failure rate in clinical trials for oncology. Given this reality, how do we ethically encourage trial participation within the informed consent criteria for patients with cancer seeking potential therapeutic benefit?
First off, a lot of people talk about clinical trial failure rates in a totally negative manner, which is a fundamental misunderstanding of the system. Every trial that produces a negative result is saving countless numbers of patients with cancer from receiving an ineffective drug.
The task in enrolling patients on trial is twofold. First we need to be extremely judicious about deciding whether to test an agent in the first place (including testing approved drugs in new malignancies). Once that decision is made, we probably need to raise the bar on trials—at least for some drugs. For example, it’s surprisingly common for companies to launch phase III trials absent any supporting evidence in a phase II trial. It’s also surprisingly common for drugs with uncertain or marginal effectiveness to receive FDA approval. Before a patient is placed on a trial, informed consent discussions must be clear about the chances that the drug being tested will demonstrably improve their cancer outcomes.
Ethics of Randomization
Canadian bioethicist Benjamin Freedman considered it to be entirely ethical for individual clinician-investigators with a preference for one of the alternatives being tested to enroll patients for randomization, even if it places a patient in a treatment arm deemed inferior by that physician. What are your thoughts on that issue?
Back in the 1950s and 1960s, when clinical epidemiologists started to recognize that we needed randomization to develop evidence by which we could make good treatment decisions, random assignment of patients was shunned by physicians, because they felt it wasn’t in their patients’ best interest. Therefore, we needed a good tool to describe when it was ethical to enroll a patient with cancer on a randomized trial.
The original moral solution to this problem was called individual equipoise, which stated that if you’re a doctor and you have no preference between the two drugs on a clinical trial, then it is ethical to enroll your patient. However, the minute a physician sees any evidence of benefit emerging, even though it may not be reliable, a preference emerges, moving the physician out of individual equipoise, necessitating their withdrawal from the trial. So, you could never run a trial long enough to make sound policy decisions.
Under clinical equipoise, a physician is morally justified in randomly assigning patients, even though that physician may feel that the drug being given is inferior to standard care, provided there is a degree of uncertainty among the expert community. This takes the moral and scientific responsibility off the shoulders of the individual physician, and puts it in the community of experts. Think of it this way: all oncologists in a given department probably have very competent colleagues they disagree with on certain clinical issues. Would it be ethical for you as a physician to refer a patient to that colleague down the hall, even though he or she might disagree with you on certain clinical decisions? Essentially, clinical equipoise would say, yes, that’s competent medicine.
The beauty of clinical equipoise is that, in order to run clinical trials, it’s not that you just need to recruit patients. You also need to recruit physicians who take their Hippocratic oath seriously. The concept of clinical equipoise makes trial participation an ethical and valuable proposition for those physicians.
This moral principle requires that randomized controlled trials should not be pursued without uncertainty in the expert community about the clinical merits of a treatment compared with the standard of care. If trial results were completely predictable, there would be no point in running randomized trials. And without clinical equipoise as a guiding ethos, it would be almost impossible to maintain as robust a clinical trial enterprise as the present one.
DISCLOSURE: Dr. Kimmelman has served as a consultant or advisor for Ultragenyx Therapeutics and Praxis Biotech.
