One consequence of my writing this blog is that many of the residents I work with have started reading it -- a risk I clearly hadn't adequately considered. I received a request from one resident to write about some problem that I think residents commonly have with applying evidence to practice. That was quickly followed by my sitting in as discussant at a resident-led journal club, which led to my thinking about the many unfair and fair ways that people attack published randomized trials. I'll have to give additional thought in the future to the more general question of common ways residents misapply evidence in practice.
By way of background, it is standard practice in academic medical centers to present a trial at journal club and then rip it to shreds. After repeating this exercise over and over, it's easy for all involved to conclude that every trial is flawed and nothing can be learned from any of them. It can become almost a game to figure out why not to believe a given trial, rather than discover which trial results should and should not be applied to practice. (I should point out, here, that the resident who led the journal club I mentioned above made fair criticisms of the presented trial and concluded with how the trial would affect her clinical thinking and decisions.)
I think I fully recognized how far out there people could get in criticizing virtually any trial back in 1993 in the midst of the H. pylori debates. At that point there was still a lot of doubt in some quarters as to whether H. pylori caused peptic ulcers. There had been trials of eradicating H. pylori showing reduced ulcer recurrence, but the eradication regimens included bismuth which was believed to have some direct anti-ulcer activity, leaving a hole in the causal argument.
To deal with this a trial was performed looking at recurrence rates after an eradication regimen using amoxicillin plus metronidazole. Ulcer recurrence rates at one year were 8% in the treated arm and 86% in the placebo arm. To me, this seemed to give a final answer to the question of H. pylori causing ulcers, but I listened to a room full of internal medicine residents from one of the top programs in the country come up with outlandish criticisms of the results. Several residents argued, apparently convincingly to others in the group, that amoxicillin and/or metronidazole might have direct anti-ulcer effects and so the reduced recurrence rates might have had nothing to do with H. pylori. There were several problems with this argument, including the lack of any a priori belief that these antibiotics had such an effect, that an anti-ulcer effect of a 12-day regimen would be unlikely to have such an enormous impact a year later, and that when the trial was analyzed by H. pylori eradication rather than treatment arm, the effect size was even larger.
Another common and generally unfair way to attack a randomized trial is to look for unmeasured confounders or confounders that are somewhat unbalanced in the table 1 data.
Confounders are things associated with both the exposure and the outcome of interest in a study, that through these associations confound the assessment of a relationship between exposure and outcome. For instance, we might find in a study that coffee drinking is associated with lung cancer. Coffee drinkers may smoke more than other people, and so, if we fail to control for smoking, we might falsely conclude that coffee drinking causes lung cancer.
In a properly performed randomized trial of a reasonable size, unmeasured confounders will sort nearly equally among the treatment arms. Even if smoking status is unmeasured, if there are 800 patients in each of two arms, we can be pretty sure that the number of smokers in the two arms will be close enough not to disturb the trial results. Table 1 in a published RCT typically gives the baseline data on the groups, and if lots of potential confounders are measured and reported on at the 0.05 significance level, we shouldn't be surprised when 1 in 20 appears unbalanced at such a level. That said, if lots of measures appear unbalanced, with one group clearly sicker than the other, it's important to go back and see that allocation in the trial was properly concealed. It's rare, though, to see problems of unbalanced allocation in a major clinical trial, particularly a placebo-controlled trial of a medication.
As is clear from some of my earlier posts on this blog, I obviously believe that some attacks on randomized trials are fair and appropriate. More fruitful attacks often deal with issues of what the GRADE group calls "indirectness". Results of a randomized trial might be indirect to the question a clinician is interested in because the trial did not study the exact intervention of interest (for example, too high or low a dose of a comparator drug was used, as in a trial comparing esomeprazole with omeprazole), perform the study in the exact population of interest (for example, the study was performed in young healthy patients, and the target population is the sick elderly), or measure the exact outcome of interest (for example, a surrogate outcome like carotid intima medial thickness was measured, rather than a clinical outcome).
I recall an argument years ago on Usenet about whether aspartame really causes headaches in the many people who report headaches when they consume aspartame-flavored foods and drinks. At the time, I pointed out the randomized trial in the NEJM showing no difference in rates of headaches in subjects who believed they were sensitive to aspartame, whether given aspartame or placebo. Another poster on Usenet objected, saying that he did not think that aspartame in a pill caused headaches, but rather that the combination of sweet tasting food and lack of sugar led to headaches, perhaps by inducing reactive hypoglycemia. Testing aspartame without letting the subjects taste the preparation was completely unconvincing to him. This was essentially an argument about indirectness of intervention -- the intervention of interest was aspartame as commonly consumed in foods and drinks, not aspartame in pill form.
To me, on reflection, this seemed like a fair objection, though I didn't think it really held up to scrutiny when looking at the actual trial results. In the trial, which had a crossover design, the 40 subjects developed headaches 35% of the time when they were given aspartame, and 45% of the time when given placebo. To argue that encapsulated aspartame doesn't cause headaches in susceptible individuals is to miss that the trial found that placebo induced headaches in these individuals almost half the time.
I'll come back to the issue of indirectness of outcomes in the near future when I'll write more about surrogate outcomes, since this underlies some of the arguments that came up in the comments responding to my vitamin D posting, and I'll likely write about indirectness with regard to population down the road.
The general point, though, is the importance of being fair, open-minded, and keeping a big-picture view when evaluating a new randomized trial, and then interpreting the results in the context of existing evidence. Lee Goldman talks about dealing with randomized trials when one is performing the role of playwright rather than theater critic. But even if you're just the theater critic, if you hate all plays you see, your reviews won't prove terribly edifying to your readers.
Comments