Evidence In Medicine

Meta-analysis and New Knowledge

When hierarchies of evidence are listed for the EBM world, meta-analyses of randomized trials generally sit at the pinnacle.

And yet, the actual meta-analyses that you encounter when researching a clinical question can be far less enlightening. Even if we grant a pass to the many systematic reviews at The Cochrane Collaboration that conclude with the a priori obvious fact that no high quality RCTs addressing a question have been performed, and another pass to the reviews that find a single RCT and publish its results as the results of the systematic review, we are still left with the innumerable meta-analyses that seem to provide less of a window on truth than the underlying trials.

Frequently such meta-analyses are either driven by the single large RCT that everyone would have cited anyway or, worse, a number of small, poorly-performed RCTs are combined with a moderate-sized, well-performed RCT and alter the results away from what was likely the best estimate of reality: the results of the well-performed RCT.

Meta-analysts often seem to either be too removed from their subject area and thus lack the expertise to really understand what went clinically right and wrong in the underlying RCTs (or be unwilling to use that knowledge to discriminate among the trials), or be too cozy with a single trial (typically as an author) and thus too willing to ding trials that found conflicting results.

Ultimately, meta-analysis only rarely seems to importantly advance our knowledge of an issue beyond where we would have found ourselves by just reading through the RCTs.

So with that background it is always interesting to me when a meta-analysis comes along that really seems to shed new light on a subject such that we seem to know something that we somehow didn't know when we just had the underlying trials.

An example came along in The Lancet last week.

Despite the enormous number of patients participating in randomized trials of statins, it has been uncertain what effect statins have on the development of diabetes. Some biochemical and animal studies suggested that statins might prevent diabetes. Clinical trials have been conflicting with some showing protection and other showing increased risk. In reviewing the underlying trials, it has been hard to figure out what is going on:

• Are some statins protective while others are harmful?
• Are hydrophilic statins having different effects than lipophilic statins?
• Was the observation of increased diabetes risk in the JUPITER Trial just a random event that became noticeable because of reporting bias (where positive or interesting secondary outcomes are more likely to show up in a paper than negative results).
• Are the varying results of the statin trials due to random variation around a single truth, or do the results suggest that the underlying trials differed from each other in some important way (perhaps because of the population studied, the way the statin was administered, or the way diabetes was assesses?

A month ago, anyone simply looking at the collection of trials would have had a hard time giving a coherent answer to the above questions. Now, after a nicely done meta-analysis by Prof. Naveed Sattar et al., there are reasonable answers to all these questions. And thinking about these questions also sheds light on how to read and judge a meta-analysis.

The new analysis found that patients treated with statins had about a 9% higher risk of diabetes than those treated with placebo or other agents. When I started reading the analysis, I had the questions in the list above already in mind and so was prepared to challenge the meta-analysis on several fronts. The authors of the analysis had appropriately anticipated my concerns and, to the extent the data allowed, answered them:

1) Was this really a chance finding driven by JUPITER? Before JUPITER found an increased risk of diabetes, there had been little discussion of statins and diabetes risk. JUPITER's findings could have been due to chance, but the publicity around the result could have triggered the meta-analysis. JUPITER was large enough to sway the results in the meta-analysis and perhaps lead to a self-fulfilling conclusion based in random variation. The meta-analysis, though, did a secondary analysis that excluded JUPITER, and found that the results were essentially the same.

2) Were the varying results in the trials due to random variation or true differences? The meta-analysis found little need to invoke anything more than randomness (as measured by a statistic called the I²⁾. What had seemed to be conflicting results was likely nearly entirely due to random variation around a likely single true effect of slightly increased risk of diabetes.

3) Are some statins protective while others cause diabetes? The finding of little heterogeneity suggests the answer is no, but ultimately this is a hard question to answer definitively because of the more limited data about each individual statin. The meta-analysis found that the confidence intervals of the effects for individual statins overlapped such that it seemed unlikely that there were important differences among the statins, but it's hard to be certain. Additionally, lipophilic and hydrophilic statins showed the same effects on diabetes. And beyond that, the meta-analysis found that one of the main trials that had suggested a protective effect of pravastatin on diabetes had used an unusual definition of diabetes, and the effect was not seen when they substituted a standard definition.

While no new trials were published, as a result of this meta-analysis we have a much better feel for the effect of statins on diabetes than we had a few weeks ago. So, if after hours of trying to answer clinical questions by reading Cochrane you find yourself wondering whether meta-analyses are ever worth the effort that seems to go into them, remember this one and how much we learned about diabetes and statins from a new analysis of existing data.

Evidence, Medicine, and EBM

Lay of the Land

There have been some recent blog posts around yet again debating the concept of evidence-based medicine (EBM).

This blog is titled "Evidence in Medicine" not because "Evidence Based Medicine" was already taken as a URL and blog title, but rather because I didn't want some of the baggage and implications of the EBM label.

EBM is frequently defined as "the conscientious, explicit and judicious use of current best evidence in making decisions about the care or individual patients", and you would think it would be hard to object to that. And yet....

One blogger, Laika Spoetnik (Jacqueline), recently wrote a piece complaining about the attacks on EBM, and mentioned the humorous parachute example that I had referred to in an earlier post. She wrote about the parachute piece::

I found the article only mildly amusing. It is so unrealistic, that it becomes absurd. Not that I don’t enjoy absurdities at times, but  absurdities should not assume a live of their own.  In this way it doesn’t evoke a true discussion, but only worsens the prejudice some people already have.

(As I'd suggested in that same earlier post, EBM folks are not necessarily known for their great senses of humor.)

In response to Jacqueline's post, Kimball Atwood wrote a thoughtful piece explaining his problems with EBM, and linked to an earlier piece he'd written on the same subject. I would encourage people to read Dr. Atwood's posts.

Personal Systematic Reviews

So, how could any reasonable person possibly object to the use of best evidence in making medical decisions?

The answer is that they could not, but that EBM is not necessarily synonymous with using best evidence, despite the definition above. If I'd been writing a few years ago, I would have agreed with a number of the posters attacking EBM, since many of the people I have seen promulgate EBM do so in a way that misses some basic truths.

EBM,at least as it was promoted in academic internal medicine circles in the 1990s, was very focused on a specific hierarchy of evidence (RCTs > cohort studies > case control studies > case series) and on the belief that we should train primary care internists that the right way to answer a clinical question they encountered was to perform a literature search along with their own, on the fly, systematic review. I'll come back to the hierarchy of evidence in a bit.

The idea of teaching people to perform personal systematic reviews was being pushed at a time when I was co-directing the resident journal club at my hospital. This was couched as "teaching EBM", and the various residency course directors around the country seemed to be spending a lot of time telling each other what an important activity this would be for academic general internists to engage in. I and the other director of journal club pointed out repeatedly that real systematic reviews take months, and that it was a bizarre notion to think that the right way to answer a question that arose in a clinic would be to try to do this on the cheap. A quick review of randomized trials is fraught with the likelihood of missing important pieces of evidence. When residents try to do  this, they often end up focusing on the most recently published trial rather than the entirety of the available knowledge.

We argued, instead, that for most such questions residents would be far more likely to find the right answer in a textbook than by trying to interpret a body of literature in between seeing patients. (These days I would make the same argument but replace "textbooks" with "electronic resources", however that raises a conflict of interest with one of my day jobs, so I won't push that particular argument any further.)

As a result of our resistance, this curriculum never quite got implemented for our journal club, but my sense was that put our hospital in the minority. I think many programs in the late 90s and early 2000s tried to convince residents that it was sensible, practical, and necessary to perform their own systematic reviews whenever confronted with a clinical question.

Hierarchies of Evidence

Coming back to the hierarchy of evidence issue, it's clear that a system that focuses too much on the hierarchy mentioned above will reach absurd conclusions. For instance, one might decide that we have higher quality evidence that an HIV vaccine is protective than that people who wish to avoid lung cancer should not smoke. We have essentially no evidence from RCTs in humans supporting that latter hypothesis and yet we should clearly believe it to be true as surely as we believe anything in medicine, while it remains quite uncertain that an HIV vaccine is of any benefit despite the published RCT with a "statistically significant" p value.

And when effect sizes are large enough, even a couple of clinical events may provide higher quality evidence than some RCTs that try to examine questions of minor benefits of therapies. How many people have to survive meningococcal meningitis after treatment with penicillin before you can be quite sure that penicillin is lifesaving?

Dr. Atwood goes to the specific issue of biologic plausibility in the hierarchy of evidence as it applies to homeopathy. I actually agree with those who are suspicious of biologic plausibility arguments, since smart people can make almost anything sound biologically plausible. Dr. Atwood focuses more on biologic implausibility and I tend to find these arguments more compelling on average. In the case of homeopathy, he argues, it would be pointless and a waste of resources to perform RCTs examining homeopathic remedies. This is a fair point, since if homeopathy is true than our understanding of chemistry and physics is incorrect; thus it would require extreme levels of proof before we should begin to entertain the notion that a homeopathic preparation could be of benefit.

(I plan to write a future post about the burdens placed on the scientific medicine community by people who promote unproven remedies; valuable resources must then often be spent to prove the lack of benefit of the now widely popular yet worthless preparations and interventions.)

EBM and GRADE

Given what I've written you might conclude that I lean toward the anti-EBM camp. As mentioned, that's likely where I would have placed myself a few years ago. In the interim, though, I've started working with the GRADE group and in particular with Gordon Guyatt.

Through these interactions, I've come to realize that when a really smart and thoughtful clinician like Dr. Guyatt discusses EBM, the above problems and concerns mostly go away. For instance, GRADE recognizes the importance of randomized trials, but also that high quality evidence can be found in other places. The group works to formalize what might lead rational clinicians to conclude they have high quality evidence, and while I don't agree with GRADE 100% of the time, my disagreements mostly involve some edge cases that really don't arise very often.

And since Dr. Guyatt is generally credited with inventing the term EBM, it's hard to dismiss his interpretation as being unrepresentative of true EBM. Instead, I've come to believe that most of the people who have interpreted and disseminated "EBM" lacked either the clinical understanding or the epidemiologic knowledge to communicate a fair representation of EBM to the clinical world.

I'd encourage those who find themselves troubled by aspects of EBM to spend some time looking at what the GRADE Working Group is doing. Even better, if you have the chance to interact with Gordon Guyatt at one of his courses or lectures, jump at the opportunity. You likely will feel differently about EBM for having done so.

Revere's Thought Experiment and Observational Studies

Over at Effect Measure, Revere has posed a thought experiment (or, perhaps, a disguised real experiment) that provides a valuable jumping off point for discussing the worth of observational data.

In brief, Revere describes an uncontrolled study for a new application of an antiepileptic drug as a treatment for refractory hypertension. A study is conducted by administering the drug to 29 patients and checking their blood pressures at baseline and after a month of therapy.

I glanced at the comments on the blog, and interestingly a lot of people are spending time trying to apply a name to what exactly this study is (i.e. "case series", "observational", and the like). I often don't find naming things all that helpful unless everyone has a clear understanding of the naming convention, but this would most commonly either be called an observational study (if we view that the therapy was going to be administered independent of the interest in collecting data about what happened), or an uncontrolled clinical trial (if we view that the primary goal was to find out if the AED works for refractory hypertension).

In the thought experiment, the blood pressure readings a month later are substantially lower.

Revere poses the following question:

"...is this a sufficiently reliable study that a reasonable practitioner, committed to the use of scientific evidence in her practice, would consider?"

I'm going to move on quickly from that question, because I think the correct answer isn't that helpful: the study contains data about a potential new therapy, and a reasonable practitioner should always consider such evidence if she or he has the time and ability to do so.

To understand, instead, whether we should believe that the results in this study can be ascribed to the drug working, we might look first at how we would feel about the result if it had been obtained from a randomized trial performed in the same manner described as the uncontrolled trial, except now we have 29 patients randomly assigned to receive the AED, and 29 to not receive it. Notice that this RCT is still unblinded and, as such, has no placebo arm.

We'd likely note a few problems with this RCT. First, it has a relatively small number of patients, though if the drug is highly effective this might not be a problem. We'll come back to this point a bit later.

Second, the outcome is fairly indirect (in GRADE terminology) to an outcome of clinical importance. Revere points out that BP reduction is not a clinical outcome: it's a commonly used surrogate, but what we really care about is a reduction in cardiovascular events and this is not being measured. BP lowering is such a standard surrogate that it's easy to lose sight that it is a surrogate at all, but not everything that lowers BP is good for you (consider endotoxin, for instance). However the actual measurement has an additional level of indirectness: we are measuring BP reduction at one month as a surrogate for long-term blood pressure reduction. Long-term BP reduction is the marker that we actually typically use as a surrogate for a reduction in CV events.

Third, as noted above, the study is unblinded. Does this matter? We're told that the BP both at baseline and at one month is being taken with an automated cuff and all the clinical assistant has to do is write down the measurement. This sounds like a simple solution, but it is fraught with problems. The patients are unblinded, and so the ones taking the medication may believe in its efficacy, relax more, and so have lower BP readings in the office. The clinical assistants believe in the drug and want the study to succeed, and so may take several readings in patients on the AED ("why don't you sit here and relax, and I'll recheck your BP in a few minutes -- this drug looks like it's working in almost everyone, so I'm sure your BP will come down if we just wait"), but record the first BP reading in the control group patients. And, of course, those analyzing the results know who is and is not receiving the AED, and this might affect all sorts of things (not that we like to admit this).

So this small, unblinded, short-term study of a surrogate outcome might not be all that convincing, even as an RCT.

When we push it a step further and make it uncontrolled, we add an additional major problem of regression to the mean (as well as secular trends in BP over time). In the RCT, we had a control group, and so we at least knew what happened to the BP readings in patients not treated with the AED. in the uncontrolled study we have no such knowledge. We're not told exactly what the entry criteria are, but presumably having a well-controlled BP would prevent a patient from getting the AED as an additional anti-hypertensive. Thus, patients who are oscillating around a mean BP value will likely be excluded if they are at a BP minimum and enrolled if they are at a BP maximum. One month later, just due to the usual biologic variation plus this entry criterion, the BP can be expected to be lower on average.

Adding this problem to the ones we noted in the RCT version, we are unlikely to be terribly sanguine about believing the results of this small trial.

Are there any circumstances, though, where despite all this we might trust the results anyway? One of the real insights of the GRADE group, to my mind, is a focus on magnitude of effect in thinking about observational data, rather than focusing on minor differences in study design (such as cohort versus case control). With large enough effects, we might accept some pretty minimalist study designs.

For instance, rabies is historically a 100% fatal illness once clinical symptoms appear in someone who has never been vaccinated. In 2005, a 15-year-old girl survived rabies after treatment with a novel experimental regimen. You could imagine that anything can happen once, and it may have been coincidence that this girl survived and received this novel regimen. However, if one more person with rabies were to receive the regimen and survive, it would seem spectacularly unlikely that the explanation would be anything but that the regimen works. A rational clinician would treat any new patient with rabies with that regimen from that moment until a superior regimen was found. (As far as I know, no other patient has survived rabies on this regimen.)

Similarly, in response to the humorous attack on EBM, asking what the RCT data are supporting parachutes when skyjumping, the GRADE group would respond that the magnitude of effect of parachutes is sufficient to constitute high quality evidence for their use (okay, clinical epidemiologists can be somewhat humor-challenged). That is, we have all sorts of historical evidence about what happens when people fall from great heights (and that we might consider only slightly indirect to falling from an airplane), as well as lots of observational data about what happens when people fall from airplanes wearing parachutes. Not everyone who falls from a great height without a parachute dies, and not everyone wearing a parachute lives, but the effect size is so large that we have high quality evidence for parachutes in the absence of a clinical trial.

To give one example that might feel more real, I was doing a lot of AIDS care in the 1990s. In 1995 an abstract was published from one of the ID meetings about the effects of ritonavir in about 50 people with late AIDS. (I've tried in vain in the recent past to find this abstract -- if anyone can point me to it, I would be grateful.) The results were like nothing we had seen before -- patients' CD4 counts rose dramatically, opportunistic infections improved, and many patients who would have been expected to die improved instead. We did not know what would happen long-term, but it was obvious, without any RCT, that ritonavir was effective therapy for AIDS, at least in the short term. By 1996, we were treating people with triple therapy "cocktails" for HIV, again without any RCTs with clinical endpoints, and watching people who had been dying walk out of hospitals and hospice care as their OIs resolved. The magnitude of effect was such that we had high quality evidence for these cocktails based on observational data alone. (Not that this actually prevented researchers from proceeding with an RCT of triple therapy, but that's a post for another day.)

In the refractory hypertension thought experiment, the problem is that we do not have enough evidence about how unusual the BP lowering seen was to conclude that this was a very large (and thus very unexpected) result. The result could have been due to measurement biases and regression to the mean, and so have had nothing to do with the actual drug being studied.

Revere started his/her post talking about Cochrane and the focus on RCTs. I agree both with concerns about Cochrane reviews (which can be of quite variable quality), and about the general issue of ignoring high quality observational data (most often high quality because of very large magnitudes of effect) to focus exclusively on RCTs. However, I don't find the evidence from the study in Revere's thought experiment to be sufficiently high quality that I would be generally willing to administer the AED for refractory hypertension without further study. Having read such a study, I might take it into consideration sufficiently to administer the AED as a last resort -- a situation in which I might be willing to grasp at any straw of evidence.

Attacking Randomized Trials

One consequence of my writing this blog is that many of the residents I work with have started reading it -- a risk I clearly hadn't adequately considered. I received a request from one resident to write about some problem that I think residents commonly have with applying evidence to practice. That was quickly followed by my sitting in as discussant at a resident-led journal club, which led to my thinking about the many unfair and fair ways that people attack published randomized trials. I'll have to give additional thought in the future to the more general question of common ways residents misapply evidence in practice.

By way of background, it is standard practice in academic medical centers to present a trial at journal club and then rip it to shreds. After repeating this exercise over and over, it's easy for all involved to conclude that every trial is flawed and nothing can be learned from any of them. It can become almost a game to figure out why not to believe a given trial, rather than discover which trial results should and should not be applied to practice. (I should point out, here, that the resident who led the journal club I mentioned above made fair criticisms of the presented trial and concluded with how the trial would affect her clinical thinking and decisions.)

I think I fully recognized how far out there people could get in criticizing virtually any trial back in 1993 in the midst of the H. pylori debates. At that point there was still a lot of doubt in some quarters as to whether H. pylori caused peptic ulcers. There had been trials of eradicating H. pylori showing reduced ulcer recurrence, but the eradication regimens included bismuth which was believed to have some direct anti-ulcer activity, leaving a hole in the causal argument.

To deal with this a trial was performed looking at recurrence rates after an eradication regimen using amoxicillin plus metronidazole. Ulcer recurrence rates at one year were 8% in the treated arm and 86% in the placebo arm. To me, this seemed to give a final answer to the question of H. pylori causing ulcers, but I listened to a room full of internal medicine residents from one of the top programs in the country come up with outlandish criticisms of the results. Several residents argued, apparently convincingly to others in the group, that amoxicillin and/or metronidazole might have direct anti-ulcer effects and so the reduced recurrence rates might have had nothing to do with H. pylori. There were several problems with this argument, including the lack of any a priori belief that these antibiotics had such an effect, that an anti-ulcer effect of a 12-day regimen would be unlikely to have such an enormous impact a year later, and that when the trial was analyzed by H. pylori eradication rather than treatment arm, the effect size was even larger.

Another common and generally unfair way to attack a randomized trial is to look for unmeasured confounders or confounders that are somewhat unbalanced in the table 1 data.

Confounders are things associated with both the exposure and the outcome of interest in a study, that through these associations confound the assessment of a relationship between exposure and outcome. For instance, we might find in a study that coffee drinking is associated with lung cancer. Coffee drinkers may smoke more than other people, and so, if we fail to control for smoking, we might falsely conclude that coffee drinking causes lung cancer.

In a properly performed randomized trial of a reasonable size, unmeasured confounders will sort nearly equally among the treatment arms. Even if smoking status is unmeasured, if there are 800 patients in each of two arms, we can be pretty sure that the number of smokers in the two arms will  be close enough not to disturb the trial results. Table 1 in a published RCT typically gives the baseline data on the groups, and if lots of potential confounders are measured and reported on at the 0.05 significance level, we shouldn't be surprised when 1 in 20 appears unbalanced at such a level. That said, if lots of measures appear unbalanced, with one group clearly sicker than the other, it's important to go back and see that allocation in the trial was properly concealed. It's rare, though, to see problems of unbalanced allocation in a major clinical trial, particularly a placebo-controlled trial of a medication.

As is clear from some of my earlier posts on this blog, I obviously believe that some attacks on randomized trials are fair and appropriate. More fruitful attacks often deal with issues of what the GRADE group calls "indirectness". Results of a randomized trial might be indirect to the question a clinician is interested in because the trial did not study the exact intervention of interest (for example, too high or low a dose of a comparator drug was used, as in a trial comparing esomeprazole with omeprazole), perform the study in the exact population of interest (for example, the study was performed in young healthy patients, and the target population is the sick elderly), or measure the exact outcome of interest (for example, a surrogate outcome like carotid intima medial thickness was measured, rather than a clinical outcome).

I recall an argument years ago on Usenet about whether aspartame really causes headaches in the many people who report headaches when they consume aspartame-flavored foods and drinks. At the time, I pointed out the randomized trial in the NEJM showing no difference in rates of headaches in subjects who believed they were sensitive to aspartame, whether given aspartame or placebo. Another poster on Usenet objected, saying that he did not think that aspartame in a pill caused headaches, but rather that the combination of sweet tasting food and lack of sugar led to headaches, perhaps by inducing reactive hypoglycemia. Testing aspartame without letting the subjects taste the preparation was completely unconvincing to him. This was essentially an argument about indirectness of intervention -- the intervention of interest was aspartame as commonly consumed in foods and drinks, not aspartame in pill form.

To me, on reflection, this seemed like a fair objection, though I didn't think it really held up to scrutiny when looking at the actual trial results. In the trial, which had a crossover design, the 40 subjects developed headaches 35% of the time when they were given aspartame, and 45% of the time when given placebo. To argue that encapsulated aspartame doesn't cause headaches in susceptible individuals is to miss that the trial found that placebo induced headaches in these individuals almost half the time.

I'll come back to the issue of indirectness of outcomes in the near future when I'll write more about surrogate outcomes, since this underlies some of the arguments that came up in the comments responding to my vitamin D posting, and I'll likely write about indirectness with regard to population down the road.

The general point, though, is the importance of being fair, open-minded, and keeping a big-picture view when evaluating a new randomized trial, and then interpreting the results in the context of existing evidence. Lee Goldman talks about dealing with randomized trials when one is performing the role of playwright rather than theater critic. But even if you're just the theater critic, if you hate all plays you see, your reviews won't prove terribly edifying to your readers.

Interpreting Confidence Intervals and Grading Recommendations

My previous post, though not really intended to be focused on p values, led to a long and interesting discussion on the issue by Revere.

Revere commented that it wasn't so clear whether I was writing as a frequentist or a Bayesian. In reality I'm a consumer of biostatistics, not a statistician. For me, the issue is pretty simply to try to know whether a given result I'm seeing is true.

Although I wrote about this in my last post I didn't frame it in exactly this way: the problem with p values is that I want to know how likely it is that the result I'm seeing is correct, while the p value tells me how unlikely it is that I would be seeing a given result if the null hypothesis were true. This is similar to the problem with knowing the sensitivity and specificity of a test, when what you really want to know is the positive predictive value (how likely is it that the patient has the disease now that the test has come back positive?).

The academic medical world started hearing a lot about using confidence intervals in preference to p values 15 or 20 years ago, and I think that led to many doctors concluding that confidence intervals solve this basic p value problem. Confidence intervals have some real benefits compared with p values, but this is not one of them. To look at this, we need to try to understand what a CI means.

Again using the HIV vaccine example from my last post, we can look at the same parameter that had a p value of 0.04, but now examine the point estimate of efficacy (31.2%) and its 95% CI (1.1 to 52.1%). What is it that has a 95% chance of being true, given that CI of 1.1 to 52.1%?

However tricky p value interpretation is for physicians, understanding the meaning of this CI is much worse. I've almost never heard a physician able to correctly interpret the meaning of such a CI when put on the spot with the above question. When trying to teach the interpretation I repeatedly get told there must be a simpler way to communicate the idea. Given my years of failure at this, I have little hope that this post will adequately clarify things for most readers, so if others who teach this have found a way that works to explain it, please write!

All we can really say about that CI is that (excluding any problems with the design or performance of the trial) if we performed the same study 100 times and calculated 95% CIs each time, we would expect that 95 of the 100 CIs formed in this way would include the true point estimate of vaccine efficacy.

Note that just as with p values, this isn't what we want to know: we want to know how likely it is that the true value is inside the particular CI that we are looking at, but that isn't what the CI actually tells us.

However, despite recognizing what the CI really does and does not tell us, consumer of biostatistics that I am, I (and others) approach CIs operationally: we choose to interpret the CI as a range of values with which the data are reasonably compatible, and to interpret values outside the CI as reasonably incompatible with the data. So, other things being equal, I would say that a vaccine efficacy of 5% was compatible with the results of the NEJM study, while a vaccine efficacy of 60% was not. This does not mean that I think the study has excluded the possibility of the vaccine having 60% efficacy, just that this would be unusual under the plays of chance.

This operational definition works as I decide how to write recommendations about whether to administer such a vaccine. If the vaccine truly had an efficacy of 31% I would likely recommend wide use in high risk patients. If the high end of the CI were true (52% efficacy), I might recommend universal vaccination. If the low end were true (1% efficacy) I would probably recommend leaving it on the shelf. Looking at this, I can quickly realize that if this trial were the only information I had about vaccine efficacy then I have inadequately precise data to support whatever recommendation (or set of recommendations) I might want to make about administering HIV vaccine.

If I were using the GRADE scheme for grading such recommendations, I might have started with the assumption that I had high quality evidence from a large randomized trial. But when I realize that I would make different recommendations based on the reasonable values at each end of the CI, I know that I must downgrade the quality of the evidence for such  imprecision. Recommendations for HIV vaccine based on this trial, using the GRADE scheme, would certainly be graded as having no better than moderate quality evidence because of this imprecision.

Instead of some arbitrary definition of whether a trial is large enough or precise enough, using the CI in this way allows me to communicate something important about the quality of the evidence as I grade recommendations. I downgrade for imprecision not because a CI crosses a null effect boundary (like 1.0 for a relative risk) but because the CI crosses a clinical boundary where the appropriate recommendation would change from one side of the boundary to the other.

In this way, the CI is far more useful than a p value. I keep in the back of my mind, though, that the CI doesn't really mean what I'm trying to use it to mean -- it's just that I usually don't have anything better.

I'll write more in the future about how GRADE looks at other types of limitations on the quality of evidence from randomized trials.

HIV Vaccines, p values, and Proof

As mentioned in my last post, I want to continue the discussion of plausibility with the recent evidence suggesting benefit from an AIDS vaccine. This was a randomized, placebo-controlled trial in 16,000 adults, that, at least in the initial press releases, showed a statistically significant 31% decrease in HIV infection in vaccinated individuals. How confident should we be in the results?

The analysis that showed the statistically significant benefit (p = 0.04), was a reasonable one. There were 56 people who developed HIV in the vaccine arm, and 76 in the placebo arm, but seven patients were found to be infected with HIV at the time the vaccine was administered, and this analysis excluded those patients. Before we come back to that decision, we should keep in mind what that p value means -- what it is that has a 4% chance of having happened?

Although this causes untold confusion for physicians at all levels of training, that p value says that (excluding any problems with the design or performance of the trial), if vaccine were no better than placebo we would expect to see a difference as large or larger than the one seen in this trial only 4 in 100 times. This is distinctly different from saying that there is a 96% chance that this result is correct, which is how many people wrongly interpret such a p value.

One reason this is so is that many values of truth closer to the null hypothesis (but still where vaccine is superior to placebo) are possible and consistent with the result.

A more important issue of iterpretation in many cases including this one, though, is that we are not starting from a position where all results are equally likely. We had a lot of reason to be suspicious prior to this trial that the vaccine would fail to protect against HIV. Although this way of looking at results is often given important-sounding terms like "Bayesian reasoning", really all we're saying is that if something was unlikely to begin with, we need a higher level of statistical "proof" than if something were likely; if something is extremely likely to begin with, we need a much lower level of "proof".

Imagine a small randomized trial of statins in a population where they have haven't been clearly studied (say, hypertensive smokers without known CHD). If the trial found an insignificant relative risk of CV events in the treated group of 0.78 (22% relative risk  reduction,p = 0.40), it would not be the case that there is a 40% likelihood that this result was due to chance. We have lots of prior evidence suggesting that just such a RR is likely, and our belief that this reduction in risk was due to chance should be much smaller than suggested by the p value in a single trial.

In the case of an HIV vaccine, the failure out multiple prior HIV vaccines gives us a baseline doubt about any new vaccine. Added to that is the apparent difficulty of even actual HIV infection to prevent subsequent infection: patients with HIV who have a normal CD4 count and are on HAART that is adequately controlling their infection can become superinfected with a strain of HIV that is resistant to the HAART regimen they are receiving. This raises real concerns about how to create a vaccine to prevent infection.

So, the modestly positive result found in the trial must be weighed against our prior belief that such a vaccine would fail. Had the vaccine been dramatically protective, giving us much stronger evidence of efficacy, our prior doubts would be more likely to give way in the face of high quality evidence of benefit.

And then, into the mix must enter the reality of the multiple analyses performed by the researchers. Although I said the analysis that gave this result was reasonable, it was not the analysis they apparently first planned to perform. The study specified an intention to treat analysis (that did not find a statistically significant benefit at the 0.05 level), and a per protocol analysis (that also did not find a statistically significant benefit).

While the actual analysis the investigators decided to make primary would be completely appropriate had it been specified up front, it now suffers under the concern of showing marginal significance after three bites at the statistical apple; these three bites have to adversely affect our belief in the importance of that p value. And, it's not so obvious why they would have reported this result rather than excluding those 7 patients from the per protocol analysis and making that the primary analysis; there might have been yet a fourth analysis that could have been reported had it shown that all important p value below 0.05.

The authors of the study tell us that the choice of analysis was made prior to unblinding of the results, but that probably isn't adequate. It had been used in interim analyses (and the final analysis) on the blinded data, but likely so had the intention to treat and per protocol analyses. If this analysis were the only one that achieved a p value below 0.05, it could have been chosen as the preferred analysis for that reason alone. The investigators wouldn't have known prior to unblinding whether the trial would show benefit or harm with vaccine, but either result would be more interesting than a statistically non-significant outcome. The vaccine manufacturer would be far better off with a statistical benefit, but probably no worse off if the vaccine showed harm than if it showed no effect.

I've been struck since the beginning of my medical training that biostatistics is an area of mathematics where people can have remarkably different opinions about what is or is not not resonable. I don't usually think of "opinion" and "mathematics" falling in the same sentence, but perhaps that just shows my lack of expertise in theoretical math.

In this case, while the investigators may have felt they were reasonable picking the most "appropriate" analysis, we who interpret the study have to worry about those three statistical bites. Combined with the prior likelihood that the vaccine would fail, I think we need to remain very wary that despite an apparent 31% vaccine efficacy in a trial involving 16,000 patients, the vaccine may in reality be no better than placebo.

Proof, Plausibility, and a Tale of Two Viruses

Listening to NPR

There was a third incitement to this blog that I have not yet mentioned, as I am not currently in possession of an asbestos suit, but I feel the time has come to discuss it. I was driving to work listening to NPR's Morning Edition last month when I heard, "We have news, now, of a scientific breakthrough about a disease that affects over a million people in the U.S."

My ears perked up, wondering what major breakthrough I had missed. I learned from the interview that an article had been published in Science linking the retrovirus XMRV with chronic fatigue syndrome. On the usually reserved and balanced NPR, I was startled to hear the interview proceed in such a way as to assume that the study was accurate and that it proved wrong all those who had questioned CFS as a single disease entity. I had the sense listening that either the interviewer or the producer of the piece had a personal stake in the outcome of the story.

The article in Science reported that 67% of patients with CFS and 3.7% of healthy controls had XMRV DNA. A news story reported that, after the submission to Science, the authors of the study improved their assay for XMRV antibodies and found them in 95% of patients with CFS. Is it plausible that 95% (or 67%) of patients diagnosed with CFS really have a single disease caused by the retrovirus XMRV?

Plausibility

Plausibility in medicine is a tricky concept. Almost any result can be argued to be biologically plausible by someone with a modicum of creativity. And yet, biologic plausibility often does not pan out so well when it meets real life. When I was in training, beta blockers were considered absolutely contraindicated in patients with heart failure and the incipient notion that peptic ulcers were an infectious disease was considered absurd.

There's a different kind of plausibility, though, for which I have greater fondness. It has to do with prior probabilities developed from a fair-minded look at the epidemiology of diseases or from prior trials that test some similar notion to the current idea under scrutiny. For instance, while JUPITER found a 44% reduction in its composite primary endpoint, it seems highly implausible to me that rosuvastatin really lowers risk by this much. Nearly every statin seems to lower risk in primary and secondary prevention by 20-25%; why should rosuvstatin have such a marked effect in a fairly low risk population? It seems more likely that the effect was exaggerated when the trial was stopped early for benefit than that we can really expect these sorts of results from rosuvastatin in patients with low LDLs and mildly elevated CRPs.

Evaluating the Evidence

The article in Science is an odd read for a clinician. The same article in a medical journal would spend a lot of time describing who the patients and controls were, and how the patients were diagnosed with CFS, but none of that is in the article. Like most articles in Science, it spends its time on bench methodology (how the retrovirus was detected) rather than on the methods by which CFS was diagnosed and the blinding used when the samples were analyzed.

That raises an initial question of "are these CFS patients similar to those who walk into a primary care physician's office carrying a diagnosis of CFS?" That is, can the results be generalized to the broad group of people who are said to have CFS? Without a more detailed description in the article, I have no way to be sure.

But beyond that, what has happened with other CFS "breakthroughs"? Back in the 1990s, after an article showing much higher rates of neurally mediated hypotension in patients with CFS (22 of 23 patients) than in healthy controls (4 of 14 controls), I made the same argument on Usenet that I'm in the process of making here. That article led to patients with CFS being treated with fludrocortisone and similar agents. Eventually people concluded that the neurally mediated hypotension story didn't really make sense and clinicians and patients moved on.

From my point of view, given the patients I have seen carrying the label CFS, it seems extremely unlikely that any single etiology will explain 95% of cases. Patients carrying the label CFS in the real world (rather than in research studies) seem to be a mixed bag. Many seem to have atypical depression or another psychological cause of fatigue (thus my need for the asbestos suit). Others, though, likely have a missed diagnosis or have some condition that is, as yet, unknown to medical science (retroviral infection with XMRV would have previously fit in this category).

Interpretation and Proof

Why does it matter? From the clinical perspective, many people with depression (typical or atypical) can be helped by standard therapies. Avoiding the label of depression for those who are depressed is no kindness (though it is also of no help for doctors to insist that symptoms are due to depression when they are not; I do not believe that all patients with CFS really suffer from depression). And getting therapies for purported etiologies that don't pan out (such as fludrocortisone for neurally mediated hypotension) simply exposes patients to side effects, harms, and costs, without resulting benefits.

Beyond that, though, this issue is at the intersection between evidence, experience, and opinion. I have no expertise in retroviruses, know nothing about XMRV or the methods used to detect it, and am lacking detailed evidence about how the study in Science was performed. But evidence about diseases still must be interpreted through the lens of clinical experience.

Ultimately, many of the arguments between clinicians about CFS (let alone the arguments joined by patients) have to do with differing clinical interpretations of whether CFS feels like a believable single disease process. This is an area where a discipline like EBM doesn't provide much of a road map. Yet expert clinicians must make these sorts of judgments all the time when they decide whether to consider a cluster of signs and symptoms as an individual disease. Before HIV was discovered, I listened to expert clinicians reject the arguments for multiple immune system insults or inhaled nitrates as causes of AIDS, and instead point out that AIDS would turn out to be caused by a virus that was transmitted similarly to hep B given the similar epidemiology of the two diseases. There is no path from evidence to understanding that does not rely on expert interpretation, and, ultimately, no mechanical measure of sufficient evidence or proof outside of what counts as proof to those patients and providers who must make decisions.

XMRV

XMRV may yet turn out to be the cause of CFS, but given the track record of such causes, I think we need better evidence. Before NPR describes a "scientific breakthrough" and gets the hopes up of so many, I wish they would spend more time thinking and interviewing and looking for some balance. I have to believe they could have found an expert or two who would have urged caution in interpreting these results.

If future studies confirm the XMRV hypothesis, those out there now who are infuriated by my comments can feel free to say "we told you so." But if XMRV turns out not to be a very good etiology for real world cases of CFS, perhaps there might be more patience for those of us who urge not trying to explain all the many chronically fatigued patients with a single disease entity. In either case, it's worth recognizing the role that expert interpretation plays in medical controversies.

This post is long enough, and the second virus from the title of this piece will need to wait for a future post: I'll discuss the likelihood that we've really found an effective vaccine for HIV.

(After I wrote this, Marilyn Mann pointed me to a blog posting with some similar concerns about CFS having a specific viral etiology.)

Pharma, Evidence, and Trust

A number of the other blogs that have noted the existence of this blog seem to have "Pharma" in their titles in one way or another, and the implication is not that the blogger likes much about Big Pharma.

In the late 1990s, when I was spending much of my clinical time providing HIV care, I was a defender of Pharma on Usenet, at least as it related to HIV. We'd discovered the HIV virus in 1985, and by late 1995 we had a spectacularly successful treatment, due, at least in part, to the efforts of the drug manufacturers. HAART changed the lives of people with AIDS, and the lives of their caregivers, and costs of $10K to $15K per year to keep a young person alive and functional seemed perfectly reasonable to me. I thought the carping about the high prices of these medications was unfair -- it was a remarkable accomplishment to find an effective treatment for a viral infection in so short a time, and the billions spent on it deserved some reward.

By a few years later, though, I was finding myself as distrustful and angry at Pharma as anyone out there. I spend a lot of my time reading research papers and interpreting evidence, and realized one day that Pharma was employing people at least as smart and knowledgeable as I to come up with ways to trick us all about what studies showed. One example of this is running parallel studies of an agent and then only publishing the positive studies while burying the negative studies, as was done with trials of SSRIs in adolescents. The concept of a p value becomes essentially meaningless if researchers perform an unknown N of parallel trials and then publish the ones that show a positive result.

I'll write about other ways of manipulating results in future posts, but for today I want to comment on one pernicious effect of this behavior -- I have come to distrust and then dislike the companies that provide the (often useful) drugs I must use on patients. When the new HDL-raising drug torcetrapib went down in flames a few years ago, I was actually pleased: I was furious with Pfizer for having tried to get torcetrapib approved in a way that would have only marketed it in a combo pill with atorvastatin. Objectively, it makes no sense that I should be happy at the failure of a drug that would have helped my patients had it worked.

Pharma not only tries to manipulate the interpretation of results of trials, in the design, analysis, and publication phases, they apparently also try to sow doubt about existing competitors. Adriane Fugh-Berman wrote in 2005 about how she was recruited to be the author on a ghost-written manuscript intended to highlight the importance of herbal interactions in patients on warfarin (interactions that Dr. Fugh-Berman felt were overstated). The company recruiting her was presumably working for AstraZeneca, the manufacturer of ximelgatran. Ximelgatran, a direct thrombin inhibitor that aimed to replace warfarin, was eventually withdrawn from the market in Europe (never approved in the US) because of hepatotoxicity.

In the face of this comes the RE-LY trial comparing warfarin with dabigatran, another direct thrombin inhibitor. I didn't start blogging until well after the publication of RE-LY, but hopefully others noted that this 18,000 person trial may reflect one of the biggest advances in medical therapy of the decade. Dabigatran appears to be a superior drug to warfarin -- at lower doses it is safer than wafarin and as effective, and at higher doses it is equally safe to warfarin and more effective. Unlike warfarin it does not require a careful diet or routine monitoring of INRs.

By all rights, dabigatran should take over the oral anticoagulation market, at least for atrial fibrillation (we need additional trials in other clinical settings). The natural skeptic in me would always want to wait a bit on using a new drug. Warfarin has been around for decades and has been used in tens of millions of people and we have a good sense of its benefits and (myriad) risks and burdens, while dabigatran has not been used nearly so widely nor for so long. New drugs have a habit of showing up rare side effects once they are out of clinical trials and on the market.

But my greater fear is that somehow Pharma is pulling something over on me, the editors of the NEJM that published RE-LY, and everyone else, and that it will somehow turn out that dabigatran not only isn't as big an advance as it appears, but that its manufacturers already know this and aren't telling.

Ultimately, if I had to bet, I don't think this will be the case with dabigatran. Five to ten years from now, I'm guessing that the residents I teach will have no clue that we used to routinely treat patients with rat poison and monitor bleeding parameters every couple of weeks or so in order to prevent clots and emboli. But I wish I didn't have to fear that the manufacturers of the drugs I use might not actually be working toward the same goals that I am, and instead might be working to fool me yet again.

ARBITER 6 and Combining Therapies with Statins

The results from ARBITER 6 were released this week.

The interpretation is less clear.

I've been in a minority who has doubted that most drugs that lower LDL are actually beneficial in primary prevention. Statins clearly show benefit, but really no other agents have shown a reduction in all cause mortality or, in primary prevention, in cardiovascular mortality. As such, I'd been raising concerns about ezetimibe even before ENHANCE was published, and have been unconvinced that anyone should be treated with combination therapy to achieve LDL goals, since there is little to suggest added benefit when other lipid-lowering drugs are combined with a statin.

Recently, the Oxford Niaspan Study found that CIMT was improved when niacin was combined with a statin, but CIMT is a surrogate marker and it's not so clear that changes in CIMT will predict clinical benefits. However, of all the non-statin therapies, niacin has been the one that looks most promising for reducing cardiovascular mortality, at least in secondary prevention.

Now, soon after that study, we have ARBITER 6, showing that niacin plus a statin seemingly reduced cardiovascular events compared with ezetimibe plus a statin. There were 2 events in the niacin arm (1%) and 9 in the ezetimibe arm (5%) and this was statistically significant. So, what can we conclude from this?

One possibility is that niacin actually does add some clinical benefit to a statin in secondary prevention, while ezetimibe does not.

Another possibility is that ezetimibe is actively harmful, while niacin is neutral. There were slightly more events in ENHANCE in the ezetimibe arm, but the numbers were tiny.

A third possibility is that this was due to chance. The actual results are quite "fragile" in the sense that only a few events moving from one arm to the other would make the difference disappear. While, in theory, a statistical test captures this, there's been a sense in the EBM world that very small event numbers create a fragility of results not adequately captured by a p value. It seems extremely unlikely that niacin really reduces events by 80%, since this should have been obvious in prior trials of niacin. As such, either niacin combined with a statin is much superior to niacin alone, or these results are in part due to the play of chance. (It's not likely that ezetimibe raises events by this degree when used with a statin, since we would likely have noticed this in other trials.)

ARBITER 6 was stopped early for benefit, based on CIMT results. This seems like an odd decision to say the least, since change in CIMT is not a clinical outcome requiring an ethical decision to stop a trial. The early stopping was unfortunate since with more events we might be more certain about whether the reduction in events was real.

I'll be interested to see how others interpret ARBITER 6. So far I've communicated with one lipid expert (who has not shared my views about non-statin therapy) and he doesn't believe that ezetimibe is clearly to be avoided, while another expert (who does share my views about non-statin therapy) doesn't feel that ARBITER 6 means that niacin should be used. Apparently neither one found ARBITER 6 substantially reshaped their positions on this.

My best guess is that niacin reduces events a bit, though not as much as ARBITER 6 suggests. If so, for the highest risk patients, such as those who have CV events while on a statin, adding niacin might make sense. Given the side effects with niacin (36% of patients in ARBITER 6 had  flushing, and many more in the niacin arm than the ezetimibe arm discontinued therapy), I'd hesitate to use niacin in lower risk patients even for secondary prevention, out of concern that they might discontinue both niacin and the statin.

I'd be quite hesitant to treat anyone with ezetimibe at this point. While I was a skeptic that ezetimibe did anything beneficial even before ARBITER 6, I'm now substantially more concerned that it could be causing real harm.