Evidence In Medicine

Attacking Randomized Trials

One consequence of my writing this blog is that many of the residents I work with have started reading it -- a risk I clearly hadn't adequately considered. I received a request from one resident to write about some problem that I think residents commonly have with applying evidence to practice. That was quickly followed by my sitting in as discussant at a resident-led journal club, which led to my thinking about the many unfair and fair ways that people attack published randomized trials. I'll have to give additional thought in the future to the more general question of common ways residents misapply evidence in practice.

By way of background, it is standard practice in academic medical centers to present a trial at journal club and then rip it to shreds. After repeating this exercise over and over, it's easy for all involved to conclude that every trial is flawed and nothing can be learned from any of them. It can become almost a game to figure out why not to believe a given trial, rather than discover which trial results should and should not be applied to practice. (I should point out, here, that the resident who led the journal club I mentioned above made fair criticisms of the presented trial and concluded with how the trial would affect her clinical thinking and decisions.)

I think I fully recognized how far out there people could get in criticizing virtually any trial back in 1993 in the midst of the H. pylori debates. At that point there was still a lot of doubt in some quarters as to whether H. pylori caused peptic ulcers. There had been trials of eradicating H. pylori showing reduced ulcer recurrence, but the eradication regimens included bismuth which was believed to have some direct anti-ulcer activity, leaving a hole in the causal argument.

To deal with this a trial was performed looking at recurrence rates after an eradication regimen using amoxicillin plus metronidazole. Ulcer recurrence rates at one year were 8% in the treated arm and 86% in the placebo arm. To me, this seemed to give a final answer to the question of H. pylori causing ulcers, but I listened to a room full of internal medicine residents from one of the top programs in the country come up with outlandish criticisms of the results. Several residents argued, apparently convincingly to others in the group, that amoxicillin and/or metronidazole might have direct anti-ulcer effects and so the reduced recurrence rates might have had nothing to do with H. pylori. There were several problems with this argument, including the lack of any a priori belief that these antibiotics had such an effect, that an anti-ulcer effect of a 12-day regimen would be unlikely to have such an enormous impact a year later, and that when the trial was analyzed by H. pylori eradication rather than treatment arm, the effect size was even larger.

Another common and generally unfair way to attack a randomized trial is to look for unmeasured confounders or confounders that are somewhat unbalanced in the table 1 data.

Confounders are things associated with both the exposure and the outcome of interest in a study, that through these associations confound the assessment of a relationship between exposure and outcome. For instance, we might find in a study that coffee drinking is associated with lung cancer. Coffee drinkers may smoke more than other people, and so, if we fail to control for smoking, we might falsely conclude that coffee drinking causes lung cancer.

In a properly performed randomized trial of a reasonable size, unmeasured confounders will sort nearly equally among the treatment arms. Even if smoking status is unmeasured, if there are 800 patients in each of two arms, we can be pretty sure that the number of smokers in the two arms will  be close enough not to disturb the trial results. Table 1 in a published RCT typically gives the baseline data on the groups, and if lots of potential confounders are measured and reported on at the 0.05 significance level, we shouldn't be surprised when 1 in 20 appears unbalanced at such a level. That said, if lots of measures appear unbalanced, with one group clearly sicker than the other, it's important to go back and see that allocation in the trial was properly concealed. It's rare, though, to see problems of unbalanced allocation in a major clinical trial, particularly a placebo-controlled trial of a medication.

As is clear from some of my earlier posts on this blog, I obviously believe that some attacks on randomized trials are fair and appropriate. More fruitful attacks often deal with issues of what the GRADE group calls "indirectness". Results of a randomized trial might be indirect to the question a clinician is interested in because the trial did not study the exact intervention of interest (for example, too high or low a dose of a comparator drug was used, as in a trial comparing esomeprazole with omeprazole), perform the study in the exact population of interest (for example, the study was performed in young healthy patients, and the target population is the sick elderly), or measure the exact outcome of interest (for example, a surrogate outcome like carotid intima medial thickness was measured, rather than a clinical outcome).

I recall an argument years ago on Usenet about whether aspartame really causes headaches in the many people who report headaches when they consume aspartame-flavored foods and drinks. At the time, I pointed out the randomized trial in the NEJM showing no difference in rates of headaches in subjects who believed they were sensitive to aspartame, whether given aspartame or placebo. Another poster on Usenet objected, saying that he did not think that aspartame in a pill caused headaches, but rather that the combination of sweet tasting food and lack of sugar led to headaches, perhaps by inducing reactive hypoglycemia. Testing aspartame without letting the subjects taste the preparation was completely unconvincing to him. This was essentially an argument about indirectness of intervention -- the intervention of interest was aspartame as commonly consumed in foods and drinks, not aspartame in pill form.

To me, on reflection, this seemed like a fair objection, though I didn't think it really held up to scrutiny when looking at the actual trial results. In the trial, which had a crossover design, the 40 subjects developed headaches 35% of the time when they were given aspartame, and 45% of the time when given placebo. To argue that encapsulated aspartame doesn't cause headaches in susceptible individuals is to miss that the trial found that placebo induced headaches in these individuals almost half the time.

I'll come back to the issue of indirectness of outcomes in the near future when I'll write more about surrogate outcomes, since this underlies some of the arguments that came up in the comments responding to my vitamin D posting, and I'll likely write about indirectness with regard to population down the road.

The general point, though, is the importance of being fair, open-minded, and keeping a big-picture view when evaluating a new randomized trial, and then interpreting the results in the context of existing evidence. Lee Goldman talks about dealing with randomized trials when one is performing the role of playwright rather than theater critic. But even if you're just the theater critic, if you hate all plays you see, your reviews won't prove terribly edifying to your readers.

Interpreting Confidence Intervals and Grading Recommendations

My previous post, though not really intended to be focused on p values, led to a long and interesting discussion on the issue by Revere.

Revere commented that it wasn't so clear whether I was writing as a frequentist or a Bayesian. In reality I'm a consumer of biostatistics, not a statistician. For me, the issue is pretty simply to try to know whether a given result I'm seeing is true.

Although I wrote about this in my last post I didn't frame it in exactly this way: the problem with p values is that I want to know how likely it is that the result I'm seeing is correct, while the p value tells me how unlikely it is that I would be seeing a given result if the null hypothesis were true. This is similar to the problem with knowing the sensitivity and specificity of a test, when what you really want to know is the positive predictive value (how likely is it that the patient has the disease now that the test has come back positive?).

The academic medical world started hearing a lot about using confidence intervals in preference to p values 15 or 20 years ago, and I think that led to many doctors concluding that confidence intervals solve this basic p value problem. Confidence intervals have some real benefits compared with p values, but this is not one of them. To look at this, we need to try to understand what a CI means.

Again using the HIV vaccine example from my last post, we can look at the same parameter that had a p value of 0.04, but now examine the point estimate of efficacy (31.2%) and its 95% CI (1.1 to 52.1%). What is it that has a 95% chance of being true, given that CI of 1.1 to 52.1%?

However tricky p value interpretation is for physicians, understanding the meaning of this CI is much worse. I've almost never heard a physician able to correctly interpret the meaning of such a CI when put on the spot with the above question. When trying to teach the interpretation I repeatedly get told there must be a simpler way to communicate the idea. Given my years of failure at this, I have little hope that this post will adequately clarify things for most readers, so if others who teach this have found a way that works to explain it, please write!

All we can really say about that CI is that (excluding any problems with the design or performance of the trial) if we performed the same study 100 times and calculated 95% CIs each time, we would expect that 95 of the 100 CIs formed in this way would include the true point estimate of vaccine efficacy.

Note that just as with p values, this isn't what we want to know: we want to know how likely it is that the true value is inside the particular CI that we are looking at, but that isn't what the CI actually tells us.

However, despite recognizing what the CI really does and does not tell us, consumer of biostatistics that I am, I (and others) approach CIs operationally: we choose to interpret the CI as a range of values with which the data are reasonably compatible, and to interpret values outside the CI as reasonably incompatible with the data. So, other things being equal, I would say that a vaccine efficacy of 5% was compatible with the results of the NEJM study, while a vaccine efficacy of 60% was not. This does not mean that I think the study has excluded the possibility of the vaccine having 60% efficacy, just that this would be unusual under the plays of chance.

This operational definition works as I decide how to write recommendations about whether to administer such a vaccine. If the vaccine truly had an efficacy of 31% I would likely recommend wide use in high risk patients. If the high end of the CI were true (52% efficacy), I might recommend universal vaccination. If the low end were true (1% efficacy) I would probably recommend leaving it on the shelf. Looking at this, I can quickly realize that if this trial were the only information I had about vaccine efficacy then I have inadequately precise data to support whatever recommendation (or set of recommendations) I might want to make about administering HIV vaccine.

If I were using the GRADE scheme for grading such recommendations, I might have started with the assumption that I had high quality evidence from a large randomized trial. But when I realize that I would make different recommendations based on the reasonable values at each end of the CI, I know that I must downgrade the quality of the evidence for such  imprecision. Recommendations for HIV vaccine based on this trial, using the GRADE scheme, would certainly be graded as having no better than moderate quality evidence because of this imprecision.

Instead of some arbitrary definition of whether a trial is large enough or precise enough, using the CI in this way allows me to communicate something important about the quality of the evidence as I grade recommendations. I downgrade for imprecision not because a CI crosses a null effect boundary (like 1.0 for a relative risk) but because the CI crosses a clinical boundary where the appropriate recommendation would change from one side of the boundary to the other.

In this way, the CI is far more useful than a p value. I keep in the back of my mind, though, that the CI doesn't really mean what I'm trying to use it to mean -- it's just that I usually don't have anything better.

I'll write more in the future about how GRADE looks at other types of limitations on the quality of evidence from randomized trials.

HIV Vaccines, p values, and Proof

As mentioned in my last post, I want to continue the discussion of plausibility with the recent evidence suggesting benefit from an AIDS vaccine. This was a randomized, placebo-controlled trial in 16,000 adults, that, at least in the initial press releases, showed a statistically significant 31% decrease in HIV infection in vaccinated individuals. How confident should we be in the results?

The analysis that showed the statistically significant benefit (p = 0.04), was a reasonable one. There were 56 people who developed HIV in the vaccine arm, and 76 in the placebo arm, but seven patients were found to be infected with HIV at the time the vaccine was administered, and this analysis excluded those patients. Before we come back to that decision, we should keep in mind what that p value means -- what it is that has a 4% chance of having happened?

Although this causes untold confusion for physicians at all levels of training, that p value says that (excluding any problems with the design or performance of the trial), if vaccine were no better than placebo we would expect to see a difference as large or larger than the one seen in this trial only 4 in 100 times. This is distinctly different from saying that there is a 96% chance that this result is correct, which is how many people wrongly interpret such a p value.

One reason this is so is that many values of truth closer to the null hypothesis (but still where vaccine is superior to placebo) are possible and consistent with the result.

A more important issue of iterpretation in many cases including this one, though, is that we are not starting from a position where all results are equally likely. We had a lot of reason to be suspicious prior to this trial that the vaccine would fail to protect against HIV. Although this way of looking at results is often given important-sounding terms like "Bayesian reasoning", really all we're saying is that if something was unlikely to begin with, we need a higher level of statistical "proof" than if something were likely; if something is extremely likely to begin with, we need a much lower level of "proof".

Imagine a small randomized trial of statins in a population where they have haven't been clearly studied (say, hypertensive smokers without known CHD). If the trial found an insignificant relative risk of CV events in the treated group of 0.78 (22% relative risk  reduction,p = 0.40), it would not be the case that there is a 40% likelihood that this result was due to chance. We have lots of prior evidence suggesting that just such a RR is likely, and our belief that this reduction in risk was due to chance should be much smaller than suggested by the p value in a single trial.

In the case of an HIV vaccine, the failure out multiple prior HIV vaccines gives us a baseline doubt about any new vaccine. Added to that is the apparent difficulty of even actual HIV infection to prevent subsequent infection: patients with HIV who have a normal CD4 count and are on HAART that is adequately controlling their infection can become superinfected with a strain of HIV that is resistant to the HAART regimen they are receiving. This raises real concerns about how to create a vaccine to prevent infection.

So, the modestly positive result found in the trial must be weighed against our prior belief that such a vaccine would fail. Had the vaccine been dramatically protective, giving us much stronger evidence of efficacy, our prior doubts would be more likely to give way in the face of high quality evidence of benefit.

And then, into the mix must enter the reality of the multiple analyses performed by the researchers. Although I said the analysis that gave this result was reasonable, it was not the analysis they apparently first planned to perform. The study specified an intention to treat analysis (that did not find a statistically significant benefit at the 0.05 level), and a per protocol analysis (that also did not find a statistically significant benefit).

While the actual analysis the investigators decided to make primary would be completely appropriate had it been specified up front, it now suffers under the concern of showing marginal significance after three bites at the statistical apple; these three bites have to adversely affect our belief in the importance of that p value. And, it's not so obvious why they would have reported this result rather than excluding those 7 patients from the per protocol analysis and making that the primary analysis; there might have been yet a fourth analysis that could have been reported had it shown that all important p value below 0.05.

The authors of the study tell us that the choice of analysis was made prior to unblinding of the results, but that probably isn't adequate. It had been used in interim analyses (and the final analysis) on the blinded data, but likely so had the intention to treat and per protocol analyses. If this analysis were the only one that achieved a p value below 0.05, it could have been chosen as the preferred analysis for that reason alone. The investigators wouldn't have known prior to unblinding whether the trial would show benefit or harm with vaccine, but either result would be more interesting than a statistically non-significant outcome. The vaccine manufacturer would be far better off with a statistical benefit, but probably no worse off if the vaccine showed harm than if it showed no effect.

I've been struck since the beginning of my medical training that biostatistics is an area of mathematics where people can have remarkably different opinions about what is or is not not resonable. I don't usually think of "opinion" and "mathematics" falling in the same sentence, but perhaps that just shows my lack of expertise in theoretical math.

In this case, while the investigators may have felt they were reasonable picking the most "appropriate" analysis, we who interpret the study have to worry about those three statistical bites. Combined with the prior likelihood that the vaccine would fail, I think we need to remain very wary that despite an apparent 31% vaccine efficacy in a trial involving 16,000 patients, the vaccine may in reality be no better than placebo.