Evidence In Medicine

HIV Vaccines, p values, and Proof

As mentioned in my last post, I want to continue the discussion of plausibility with the recent evidence suggesting benefit from an AIDS vaccine. This was a randomized, placebo-controlled trial in 16,000 adults, that, at least in the initial press releases, showed a statistically significant 31% decrease in HIV infection in vaccinated individuals. How confident should we be in the results?

The analysis that showed the statistically significant benefit (p = 0.04), was a reasonable one. There were 56 people who developed HIV in the vaccine arm, and 76 in the placebo arm, but seven patients were found to be infected with HIV at the time the vaccine was administered, and this analysis excluded those patients. Before we come back to that decision, we should keep in mind what that p value means -- what it is that has a 4% chance of having happened?

Although this causes untold confusion for physicians at all levels of training, that p value says that (excluding any problems with the design or performance of the trial), if vaccine were no better than placebo we would expect to see a difference as large or larger than the one seen in this trial only 4 in 100 times. This is distinctly different from saying that there is a 96% chance that this result is correct, which is how many people wrongly interpret such a p value.

One reason this is so is that many values of truth closer to the null hypothesis (but still where vaccine is superior to placebo) are possible and consistent with the result.

A more important issue of iterpretation in many cases including this one, though, is that we are not starting from a position where all results are equally likely. We had a lot of reason to be suspicious prior to this trial that the vaccine would fail to protect against HIV. Although this way of looking at results is often given important-sounding terms like "Bayesian reasoning", really all we're saying is that if something was unlikely to begin with, we need a higher level of statistical "proof" than if something were likely; if something is extremely likely to begin with, we need a much lower level of "proof".

Imagine a small randomized trial of statins in a population where they have haven't been clearly studied (say, hypertensive smokers without known CHD). If the trial found an insignificant relative risk of CV events in the treated group of 0.78 (22% relative risk  reduction,p = 0.40), it would not be the case that there is a 40% likelihood that this result was due to chance. We have lots of prior evidence suggesting that just such a RR is likely, and our belief that this reduction in risk was due to chance should be much smaller than suggested by the p value in a single trial.

In the case of an HIV vaccine, the failure out multiple prior HIV vaccines gives us a baseline doubt about any new vaccine. Added to that is the apparent difficulty of even actual HIV infection to prevent subsequent infection: patients with HIV who have a normal CD4 count and are on HAART that is adequately controlling their infection can become superinfected with a strain of HIV that is resistant to the HAART regimen they are receiving. This raises real concerns about how to create a vaccine to prevent infection.

So, the modestly positive result found in the trial must be weighed against our prior belief that such a vaccine would fail. Had the vaccine been dramatically protective, giving us much stronger evidence of efficacy, our prior doubts would be more likely to give way in the face of high quality evidence of benefit.

And then, into the mix must enter the reality of the multiple analyses performed by the researchers. Although I said the analysis that gave this result was reasonable, it was not the analysis they apparently first planned to perform. The study specified an intention to treat analysis (that did not find a statistically significant benefit at the 0.05 level), and a per protocol analysis (that also did not find a statistically significant benefit).

While the actual analysis the investigators decided to make primary would be completely appropriate had it been specified up front, it now suffers under the concern of showing marginal significance after three bites at the statistical apple; these three bites have to adversely affect our belief in the importance of that p value. And, it's not so obvious why they would have reported this result rather than excluding those 7 patients from the per protocol analysis and making that the primary analysis; there might have been yet a fourth analysis that could have been reported had it shown that all important p value below 0.05.

The authors of the study tell us that the choice of analysis was made prior to unblinding of the results, but that probably isn't adequate. It had been used in interim analyses (and the final analysis) on the blinded data, but likely so had the intention to treat and per protocol analyses. If this analysis were the only one that achieved a p value below 0.05, it could have been chosen as the preferred analysis for that reason alone. The investigators wouldn't have known prior to unblinding whether the trial would show benefit or harm with vaccine, but either result would be more interesting than a statistically non-significant outcome. The vaccine manufacturer would be far better off with a statistical benefit, but probably no worse off if the vaccine showed harm than if it showed no effect.

I've been struck since the beginning of my medical training that biostatistics is an area of mathematics where people can have remarkably different opinions about what is or is not not resonable. I don't usually think of "opinion" and "mathematics" falling in the same sentence, but perhaps that just shows my lack of expertise in theoretical math.

In this case, while the investigators may have felt they were reasonable picking the most "appropriate" analysis, we who interpret the study have to worry about those three statistical bites. Combined with the prior likelihood that the vaccine would fail, I think we need to remain very wary that despite an apparent 31% vaccine efficacy in a trial involving 16,000 patients, the vaccine may in reality be no better than placebo.

Pharma, Evidence, and Trust

A number of the other blogs that have noted the existence of this blog seem to have "Pharma" in their titles in one way or another, and the implication is not that the blogger likes much about Big Pharma.

In the late 1990s, when I was spending much of my clinical time providing HIV care, I was a defender of Pharma on Usenet, at least as it related to HIV. We'd discovered the HIV virus in 1985, and by late 1995 we had a spectacularly successful treatment, due, at least in part, to the efforts of the drug manufacturers. HAART changed the lives of people with AIDS, and the lives of their caregivers, and costs of $10K to $15K per year to keep a young person alive and functional seemed perfectly reasonable to me. I thought the carping about the high prices of these medications was unfair -- it was a remarkable accomplishment to find an effective treatment for a viral infection in so short a time, and the billions spent on it deserved some reward.

By a few years later, though, I was finding myself as distrustful and angry at Pharma as anyone out there. I spend a lot of my time reading research papers and interpreting evidence, and realized one day that Pharma was employing people at least as smart and knowledgeable as I to come up with ways to trick us all about what studies showed. One example of this is running parallel studies of an agent and then only publishing the positive studies while burying the negative studies, as was done with trials of SSRIs in adolescents. The concept of a p value becomes essentially meaningless if researchers perform an unknown N of parallel trials and then publish the ones that show a positive result.

I'll write about other ways of manipulating results in future posts, but for today I want to comment on one pernicious effect of this behavior -- I have come to distrust and then dislike the companies that provide the (often useful) drugs I must use on patients. When the new HDL-raising drug torcetrapib went down in flames a few years ago, I was actually pleased: I was furious with Pfizer for having tried to get torcetrapib approved in a way that would have only marketed it in a combo pill with atorvastatin. Objectively, it makes no sense that I should be happy at the failure of a drug that would have helped my patients had it worked.

Pharma not only tries to manipulate the interpretation of results of trials, in the design, analysis, and publication phases, they apparently also try to sow doubt about existing competitors. Adriane Fugh-Berman wrote in 2005 about how she was recruited to be the author on a ghost-written manuscript intended to highlight the importance of herbal interactions in patients on warfarin (interactions that Dr. Fugh-Berman felt were overstated). The company recruiting her was presumably working for AstraZeneca, the manufacturer of ximelgatran. Ximelgatran, a direct thrombin inhibitor that aimed to replace warfarin, was eventually withdrawn from the market in Europe (never approved in the US) because of hepatotoxicity.

In the face of this comes the RE-LY trial comparing warfarin with dabigatran, another direct thrombin inhibitor. I didn't start blogging until well after the publication of RE-LY, but hopefully others noted that this 18,000 person trial may reflect one of the biggest advances in medical therapy of the decade. Dabigatran appears to be a superior drug to warfarin -- at lower doses it is safer than wafarin and as effective, and at higher doses it is equally safe to warfarin and more effective. Unlike warfarin it does not require a careful diet or routine monitoring of INRs.

By all rights, dabigatran should take over the oral anticoagulation market, at least for atrial fibrillation (we need additional trials in other clinical settings). The natural skeptic in me would always want to wait a bit on using a new drug. Warfarin has been around for decades and has been used in tens of millions of people and we have a good sense of its benefits and (myriad) risks and burdens, while dabigatran has not been used nearly so widely nor for so long. New drugs have a habit of showing up rare side effects once they are out of clinical trials and on the market.

But my greater fear is that somehow Pharma is pulling something over on me, the editors of the NEJM that published RE-LY, and everyone else, and that it will somehow turn out that dabigatran not only isn't as big an advance as it appears, but that its manufacturers already know this and aren't telling.

Ultimately, if I had to bet, I don't think this will be the case with dabigatran. Five to ten years from now, I'm guessing that the residents I teach will have no clue that we used to routinely treat patients with rat poison and monitor bleeding parameters every couple of weeks or so in order to prevent clots and emboli. But I wish I didn't have to fear that the manufacturers of the drugs I use might not actually be working toward the same goals that I am, and instead might be working to fool me yet again.