Evidence In Medicine

Meta-analysis and New Knowledge

When hierarchies of evidence are listed for the EBM world, meta-analyses of randomized trials generally sit at the pinnacle.

And yet, the actual meta-analyses that you encounter when researching a clinical question can be far less enlightening. Even if we grant a pass to the many systematic reviews at The Cochrane Collaboration that conclude with the a priori obvious fact that no high quality RCTs addressing a question have been performed, and another pass to the reviews that find a single RCT and publish its results as the results of the systematic review, we are still left with the innumerable meta-analyses that seem to provide less of a window on truth than the underlying trials.

Frequently such meta-analyses are either driven by the single large RCT that everyone would have cited anyway or, worse, a number of small, poorly-performed RCTs are combined with a moderate-sized, well-performed RCT and alter the results away from what was likely the best estimate of reality: the results of the well-performed RCT.

Meta-analysts often seem to either be too removed from their subject area and thus lack the expertise to really understand what went clinically right and wrong in the underlying RCTs (or be unwilling to use that knowledge to discriminate among the trials), or be too cozy with a single trial (typically as an author) and thus too willing to ding trials that found conflicting results.

Ultimately, meta-analysis only rarely seems to importantly advance our knowledge of an issue beyond where we would have found ourselves by just reading through the RCTs.

So with that background it is always interesting to me when a meta-analysis comes along that really seems to shed new light on a subject such that we seem to know something that we somehow didn't know when we just had the underlying trials.

An example came along in The Lancet last week.

Despite the enormous number of patients participating in randomized trials of statins, it has been uncertain what effect statins have on the development of diabetes. Some biochemical and animal studies suggested that statins might prevent diabetes. Clinical trials have been conflicting with some showing protection and other showing increased risk. In reviewing the underlying trials, it has been hard to figure out what is going on:

• Are some statins protective while others are harmful?
• Are hydrophilic statins having different effects than lipophilic statins?
• Was the observation of increased diabetes risk in the JUPITER Trial just a random event that became noticeable because of reporting bias (where positive or interesting secondary outcomes are more likely to show up in a paper than negative results).
• Are the varying results of the statin trials due to random variation around a single truth, or do the results suggest that the underlying trials differed from each other in some important way (perhaps because of the population studied, the way the statin was administered, or the way diabetes was assesses?

A month ago, anyone simply looking at the collection of trials would have had a hard time giving a coherent answer to the above questions. Now, after a nicely done meta-analysis by Prof. Naveed Sattar et al., there are reasonable answers to all these questions. And thinking about these questions also sheds light on how to read and judge a meta-analysis.

The new analysis found that patients treated with statins had about a 9% higher risk of diabetes than those treated with placebo or other agents. When I started reading the analysis, I had the questions in the list above already in mind and so was prepared to challenge the meta-analysis on several fronts. The authors of the analysis had appropriately anticipated my concerns and, to the extent the data allowed, answered them:

1) Was this really a chance finding driven by JUPITER? Before JUPITER found an increased risk of diabetes, there had been little discussion of statins and diabetes risk. JUPITER's findings could have been due to chance, but the publicity around the result could have triggered the meta-analysis. JUPITER was large enough to sway the results in the meta-analysis and perhaps lead to a self-fulfilling conclusion based in random variation. The meta-analysis, though, did a secondary analysis that excluded JUPITER, and found that the results were essentially the same.

2) Were the varying results in the trials due to random variation or true differences? The meta-analysis found little need to invoke anything more than randomness (as measured by a statistic called the I²⁾. What had seemed to be conflicting results was likely nearly entirely due to random variation around a likely single true effect of slightly increased risk of diabetes.

3) Are some statins protective while others cause diabetes? The finding of little heterogeneity suggests the answer is no, but ultimately this is a hard question to answer definitively because of the more limited data about each individual statin. The meta-analysis found that the confidence intervals of the effects for individual statins overlapped such that it seemed unlikely that there were important differences among the statins, but it's hard to be certain. Additionally, lipophilic and hydrophilic statins showed the same effects on diabetes. And beyond that, the meta-analysis found that one of the main trials that had suggested a protective effect of pravastatin on diabetes had used an unusual definition of diabetes, and the effect was not seen when they substituted a standard definition.

While no new trials were published, as a result of this meta-analysis we have a much better feel for the effect of statins on diabetes than we had a few weeks ago. So, if after hours of trying to answer clinical questions by reading Cochrane you find yourself wondering whether meta-analyses are ever worth the effort that seems to go into them, remember this one and how much we learned about diabetes and statins from a new analysis of existing data.

Pharma, Evidence, and Trust

A number of the other blogs that have noted the existence of this blog seem to have "Pharma" in their titles in one way or another, and the implication is not that the blogger likes much about Big Pharma.

In the late 1990s, when I was spending much of my clinical time providing HIV care, I was a defender of Pharma on Usenet, at least as it related to HIV. We'd discovered the HIV virus in 1985, and by late 1995 we had a spectacularly successful treatment, due, at least in part, to the efforts of the drug manufacturers. HAART changed the lives of people with AIDS, and the lives of their caregivers, and costs of $10K to $15K per year to keep a young person alive and functional seemed perfectly reasonable to me. I thought the carping about the high prices of these medications was unfair -- it was a remarkable accomplishment to find an effective treatment for a viral infection in so short a time, and the billions spent on it deserved some reward.

By a few years later, though, I was finding myself as distrustful and angry at Pharma as anyone out there. I spend a lot of my time reading research papers and interpreting evidence, and realized one day that Pharma was employing people at least as smart and knowledgeable as I to come up with ways to trick us all about what studies showed. One example of this is running parallel studies of an agent and then only publishing the positive studies while burying the negative studies, as was done with trials of SSRIs in adolescents. The concept of a p value becomes essentially meaningless if researchers perform an unknown N of parallel trials and then publish the ones that show a positive result.

I'll write about other ways of manipulating results in future posts, but for today I want to comment on one pernicious effect of this behavior -- I have come to distrust and then dislike the companies that provide the (often useful) drugs I must use on patients. When the new HDL-raising drug torcetrapib went down in flames a few years ago, I was actually pleased: I was furious with Pfizer for having tried to get torcetrapib approved in a way that would have only marketed it in a combo pill with atorvastatin. Objectively, it makes no sense that I should be happy at the failure of a drug that would have helped my patients had it worked.

Pharma not only tries to manipulate the interpretation of results of trials, in the design, analysis, and publication phases, they apparently also try to sow doubt about existing competitors. Adriane Fugh-Berman wrote in 2005 about how she was recruited to be the author on a ghost-written manuscript intended to highlight the importance of herbal interactions in patients on warfarin (interactions that Dr. Fugh-Berman felt were overstated). The company recruiting her was presumably working for AstraZeneca, the manufacturer of ximelgatran. Ximelgatran, a direct thrombin inhibitor that aimed to replace warfarin, was eventually withdrawn from the market in Europe (never approved in the US) because of hepatotoxicity.

In the face of this comes the RE-LY trial comparing warfarin with dabigatran, another direct thrombin inhibitor. I didn't start blogging until well after the publication of RE-LY, but hopefully others noted that this 18,000 person trial may reflect one of the biggest advances in medical therapy of the decade. Dabigatran appears to be a superior drug to warfarin -- at lower doses it is safer than wafarin and as effective, and at higher doses it is equally safe to warfarin and more effective. Unlike warfarin it does not require a careful diet or routine monitoring of INRs.

By all rights, dabigatran should take over the oral anticoagulation market, at least for atrial fibrillation (we need additional trials in other clinical settings). The natural skeptic in me would always want to wait a bit on using a new drug. Warfarin has been around for decades and has been used in tens of millions of people and we have a good sense of its benefits and (myriad) risks and burdens, while dabigatran has not been used nearly so widely nor for so long. New drugs have a habit of showing up rare side effects once they are out of clinical trials and on the market.

But my greater fear is that somehow Pharma is pulling something over on me, the editors of the NEJM that published RE-LY, and everyone else, and that it will somehow turn out that dabigatran not only isn't as big an advance as it appears, but that its manufacturers already know this and aren't telling.

Ultimately, if I had to bet, I don't think this will be the case with dabigatran. Five to ten years from now, I'm guessing that the residents I teach will have no clue that we used to routinely treat patients with rat poison and monitor bleeding parameters every couple of weeks or so in order to prevent clots and emboli. But I wish I didn't have to fear that the manufacturers of the drugs I use might not actually be working toward the same goals that I am, and instead might be working to fool me yet again.

ARBITER 6 and Combining Therapies with Statins

The results from ARBITER 6 were released this week.

The interpretation is less clear.

I've been in a minority who has doubted that most drugs that lower LDL are actually beneficial in primary prevention. Statins clearly show benefit, but really no other agents have shown a reduction in all cause mortality or, in primary prevention, in cardiovascular mortality. As such, I'd been raising concerns about ezetimibe even before ENHANCE was published, and have been unconvinced that anyone should be treated with combination therapy to achieve LDL goals, since there is little to suggest added benefit when other lipid-lowering drugs are combined with a statin.

Recently, the Oxford Niaspan Study found that CIMT was improved when niacin was combined with a statin, but CIMT is a surrogate marker and it's not so clear that changes in CIMT will predict clinical benefits. However, of all the non-statin therapies, niacin has been the one that looks most promising for reducing cardiovascular mortality, at least in secondary prevention.

Now, soon after that study, we have ARBITER 6, showing that niacin plus a statin seemingly reduced cardiovascular events compared with ezetimibe plus a statin. There were 2 events in the niacin arm (1%) and 9 in the ezetimibe arm (5%) and this was statistically significant. So, what can we conclude from this?

One possibility is that niacin actually does add some clinical benefit to a statin in secondary prevention, while ezetimibe does not.

Another possibility is that ezetimibe is actively harmful, while niacin is neutral. There were slightly more events in ENHANCE in the ezetimibe arm, but the numbers were tiny.

A third possibility is that this was due to chance. The actual results are quite "fragile" in the sense that only a few events moving from one arm to the other would make the difference disappear. While, in theory, a statistical test captures this, there's been a sense in the EBM world that very small event numbers create a fragility of results not adequately captured by a p value. It seems extremely unlikely that niacin really reduces events by 80%, since this should have been obvious in prior trials of niacin. As such, either niacin combined with a statin is much superior to niacin alone, or these results are in part due to the play of chance. (It's not likely that ezetimibe raises events by this degree when used with a statin, since we would likely have noticed this in other trials.)

ARBITER 6 was stopped early for benefit, based on CIMT results. This seems like an odd decision to say the least, since change in CIMT is not a clinical outcome requiring an ethical decision to stop a trial. The early stopping was unfortunate since with more events we might be more certain about whether the reduction in events was real.

I'll be interested to see how others interpret ARBITER 6. So far I've communicated with one lipid expert (who has not shared my views about non-statin therapy) and he doesn't believe that ezetimibe is clearly to be avoided, while another expert (who does share my views about non-statin therapy) doesn't feel that ARBITER 6 means that niacin should be used. Apparently neither one found ARBITER 6 substantially reshaped their positions on this.

My best guess is that niacin reduces events a bit, though not as much as ARBITER 6 suggests. If so, for the highest risk patients, such as those who have CV events while on a statin, adding niacin might make sense. Given the side effects with niacin (36% of patients in ARBITER 6 had  flushing, and many more in the niacin arm than the ezetimibe arm discontinued therapy), I'd hesitate to use niacin in lower risk patients even for secondary prevention, out of concern that they might discontinue both niacin and the statin.

I'd be quite hesitant to treat anyone with ezetimibe at this point. While I was a skeptic that ezetimibe did anything beneficial even before ARBITER 6, I'm now substantially more concerned that it could be causing real harm.

First post and origins

Although an early participant in the world of Usenet newsgroups, I'm a latecomer to the world of blogging. I don't read or participate in blogs often, and as a newbie will undoubtedly make mistakes of etiquette and function much as I did in the very early days of Usenet before I got a feel for the music of the newsgroups. I hope anyone reading this will take this into account and cut me some slack!

I'd been considering for a while now blogging about evidence and medicine, something I spend a fair amount of my time thinking about in any case, but a couple of events pushed me to take the plunge.

The first was when a relative, a woman in her mid-40's, called to tell me that she'd seen her new primary care doctor. The doctor had run tests on her including checking a lipid profile. The lipid profile showed a total cholesterol of 160, an HDL of 44, and an LDL of 105. Based on these results, the doctor informed her that she should be very concerned about her "low" HDL number and told her to start taking fish oil.

This was an interesting interpretation of the results and an unusual suggestion for management, and it gives some sense of my concerns about how evidence is used, ignored, or misused in medicine. This relative had a low normal blood pressure, no smoking history, did not have diabetes, and had no family history of coronary heart disease. As such, there is even  some controversy as to whether she should have had a lipid profile checked at all.

Assuming a lipid profile was worth doing, she should certainly have been told that this was an excellent profile putting her at low risk for a cardiovascular event, and that there was little modern medicine could do to substantially affect her risk. Her 10-year risk of an event (using the 2008 calculator that overestimates vascular risk by including things that probably shouldn't be included -- a blog for another day) works out to about 2.2%. If we pretend that somehow fish oil could increase her HDL to 80 and that she would get all the same benefit as if her HDL were naturally 80, we could drop her calculated risk to about 1.4%. So, if she took medication daily for 10 years, she would reduce her risk of an event (under these outlandishly favorable assumptions) by 0.8%. Not something this relative was interested in doing.

But, do we even know that fish oil raises HDL? The main effect of fish oil on lipids is to lower triglycerides. Most studies suggest that fish oil only raises HDL levels by about 3%, so we might expect an increase from an HDL of 44 to an HDL of 45. It's hard to see how the doctor felt this would fix the woman's "low" HDL.

But, beyond that, we have no solid evidence that raising HDL with  medication improves outcomes. Changing HDL with medication is a surrogate outcome for that medication  -- that is, it is of no direct interest to the patient unless it also predicts better cardiovascular outcomes, and HDL appears to be a poor surrogate.  Some medications that raise HDL appear to cause harm and worse vascular outcomes.

As for whether fish oil supplements are beneficial through some non-HDL mechanism, I think we can safely say that the jury remains out as to whether a woman in her 40s at low cardiac risk could expect any important benefit from fish oil supplementation.

Interestingly, my relative's doctor also checked her vitamin D level and told her that a level of 34 (in the normal range) was concerningly low and that she should start taking 1000 IU of vitamin D per day (she was already taking 400 IU). Vitamin D supplementation is controversial, with lots of people claiming big benefits, but we can safely say for now (again, perhaps a longer blog for another day) that there is no solid evidence that having a woman with a normal vitamin D level start taking additional vitamin D is likely to be of any benefit to her.

My relative asked what I thought these events said about her doctor, but I'm not really sure. I see decisions like this made all the time, where evidence is misunderstood, misapplied, or ignored. I have the sense that this particular doctor may believe she is getting out ahead of the evidence -- suggesting therapies that she believes will be proven down the road to be of benefit. She could, of course, turn out to be right, but we should be very hesitant to give preventive medications without a decent evidentiary basis. Other drugs that raise HDL have increased mortality. Supplementation with other vitamins that were thought beneficial has turned out to cause cancer or increase mortality.

I said there were a couple of events that prompted me to start this blog, but this post seems long enough. I'll talk about other events in future postings.