Meta-analysis and New Knowledge

When hierarchies of evidence are listed for the EBM world, meta-analyses of randomized trials generally sit at the pinnacle.

And yet, the actual meta-analyses that you encounter when researching a clinical question can be far less enlightening. Even if we grant a pass to the many systematic reviews at The Cochrane Collaboration that conclude with the a priori obvious fact that no high quality RCTs addressing a question have been performed, and another pass to the reviews that find a single RCT and publish its results as the results of the systematic review, we are still left with the innumerable meta-analyses that seem to provide less of a window on truth than the underlying trials.

Frequently such meta-analyses are either driven by the single large RCT that everyone would have cited anyway or, worse, a number of small, poorly-performed RCTs are combined with a moderate-sized, well-performed RCT and alter the results away from what was likely the best estimate of reality: the results of the well-performed RCT.

Meta-analysts often seem to either be too removed from their subject area and thus lack the expertise to really understand what went clinically right and wrong in the underlying RCTs (or be unwilling to use that knowledge to discriminate among the trials), or be too cozy with a single trial (typically as an author) and thus too willing to ding trials that found conflicting results.

Ultimately, meta-analysis only rarely seems to importantly advance our knowledge of an issue beyond where we would have found ourselves by just reading through the RCTs.

So with that background it is always interesting to me when a meta-analysis comes along that really seems to shed new light on a subject such that we seem to know something that we somehow didn't know when we just had the underlying trials.

An example came along in The Lancet last week.

Despite the enormous number of patients participating in randomized trials of statins, it has been uncertain what effect statins have on the development of diabetes. Some biochemical and animal studies suggested that statins might prevent diabetes. Clinical trials have been conflicting with some showing protection and other showing increased risk. In reviewing the underlying trials, it has been hard to figure out what is going on:

• Are some statins protective while others are harmful?
• Are hydrophilic statins having different effects than lipophilic statins?
• Was the observation of increased diabetes risk in the JUPITER Trial just a random event that became noticeable because of reporting bias (where positive or interesting secondary outcomes are more likely to show up in a paper than negative results).
• Are the varying results of the statin trials due to random variation around a single truth, or do the results suggest that the underlying trials differed from each other in some important way (perhaps because of the population studied, the way the statin was administered, or the way diabetes was assesses?

A month ago, anyone simply looking at the collection of trials would have had a hard time giving a coherent answer to the above questions. Now, after a nicely done meta-analysis by Prof. Naveed Sattar et al., there are reasonable answers to all these questions. And thinking about these questions also sheds light on how to read and judge a meta-analysis.

The new analysis found that patients treated with statins had about a 9% higher risk of diabetes than those treated with placebo or other agents. When I started reading the analysis, I had the questions in the list above already in mind and so was prepared to challenge the meta-analysis on several fronts. The authors of the analysis had appropriately anticipated my concerns and, to the extent the data allowed, answered them:

1) Was this really a chance finding driven by JUPITER? Before JUPITER found an increased risk of diabetes, there had been little discussion of statins and diabetes risk. JUPITER's findings could have been due to chance, but the publicity around the result could have triggered the meta-analysis. JUPITER was large enough to sway the results in the meta-analysis and perhaps lead to a self-fulfilling conclusion based in random variation. The meta-analysis, though, did a secondary analysis that excluded JUPITER, and found that the results were essentially the same.

2) Were the varying results in the trials due to random variation or true differences? The meta-analysis found little need to invoke anything more than randomness (as measured by a statistic called the I²⁾. What had seemed to be conflicting results was likely nearly entirely due to random variation around a likely single true effect of slightly increased risk of diabetes.

3) Are some statins protective while others cause diabetes? The finding of little heterogeneity suggests the answer is no, but ultimately this is a hard question to answer definitively because of the more limited data about each individual statin. The meta-analysis found that the confidence intervals of the effects for individual statins overlapped such that it seemed unlikely that there were important differences among the statins, but it's hard to be certain. Additionally, lipophilic and hydrophilic statins showed the same effects on diabetes. And beyond that, the meta-analysis found that one of the main trials that had suggested a protective effect of pravastatin on diabetes had used an unusual definition of diabetes, and the effect was not seen when they substituted a standard definition.

While no new trials were published, as a result of this meta-analysis we have a much better feel for the effect of statins on diabetes than we had a few weeks ago. So, if after hours of trying to answer clinical questions by reading Cochrane you find yourself wondering whether meta-analyses are ever worth the effort that seems to go into them, remember this one and how much we learned about diabetes and statins from a new analysis of existing data.

Comments

[Marilyn Mann]

It would be interesting to compare the methodology and results of the Lancet meta-analysis with another meta-analysis that was done recently on the same question.

Current Medical Research and Opinion, 2008, Vol. 24, No. 5, Pages 1359-1362
http://informahealthcare.com/doi/abs/10.1185/030079908X292029
Craig I. Coleman, Kurt Reinhart, Jeffrey Kluger and C. Michael White,

The effect of statins on the development of new-onset type 2 diabetes: a meta-analysis of randomized controlled trials

ABSTRACT

Objective: To determine the ability of statins to prevent the development of new-onset type 2 diabetes mellitus through a meta-analysis of randomized, controlled trials.

Research design and methods: A systematic literature search through November 6, 2007 was conducted to identify randomized, placebo-controlled trials of statins that reported data on the incidence of new-onset diabetes mellitus. Incidence of new-onset type 2 diabetes mellitus was treated as a dichotomous variable. Weighted averages were reported as relative risk (RR) with associ­ated 95% confidence intervals (CI). A random-effects model was used.

Results: Five prospective, randomized controlled trials (n=39791) were identified. Upon meta-analysis, the use of a statin did not significantly alter a patient's risk of developing new-onset type 2 diabetes mellitus (relative risk, 1.03; 95% confidence interval 0.89–1.19). Subgroup and sensitivity analyses did not significantly change the results. There was statistical heterogeneity that stemmed from pravastatin's tendency towards a reduction in risk and the other statins showing an increase in risk. The funnel plot could not rule out publication bias.

Conclusions: Statins, as a class, do not demonstrate a statistically significant positive or negative impact on a patient's risk of developing new-onset type 2 diabetes mellitus.

I say it would be interesting to compare the two meta-analyses, but I wouldn't attempt to do so myself because I don't have the necessary expertise (nor do I have free access to the article). To note a couple of obvious points:

The first meta-analysis was done before the publication of the JUPITER trial

It looks like they did not correct for the weird definition of diabetes in the pravastatin trial

Even with those factors, the first meta-analysis found a small insignificant increased risk (3 percent), CI 0.89 to 1.19. The 9 percent increased risk found in the Lancet meta-analysis is well within the CI for the first meta-analysis.

Also, I suspect that when they refer to publication bias in the abstract, they really mean reporting bias.

[Dan Hackam]

The main difference is that the lancet review is an individual patient data meta-analysis (meaning they had access to data on each patient) and the current medical review is a clinical trial level meta-analysis (meaning they only had access to published results). Therefore, the lancet review is far more exhaustive and allows for much tighter subgroup and sensitivity analyses.

Having said this, the meta-analysis:to:randomized trial ratio in the published literature is far too high - we need more RCTs, not just more meta-analyses.

[Ed Whitney]

When journals first appeared online, we heard a lot about the revolution in medical publishing that this change signified. Sure, it means that now, instead of piling up hard copies of unread journals on your desk, you can download only those articles you will actually read. Also, it means that instead of trudging off through the snow to the library to find a journal article, you can stay home and read it in your stocking feet.

But if there were actually to be a revolution in publishing, it would mean that published studies would have online supplements consisting of data files with the individual subject data, so that the reader could check the analyses and run others of interest. For example, if the published articles compared change scores between groups using a t-test, the reader could run an analysis of covariance on the final scores using the baseline score as a covariate and the treatment group as the main effect. In studies using linear regression to analyze the data, the reader could at least run a scatterplot to see if the linearity assumption is halfway reasonable. And the reader could look for homogeneity or the lack thereof between different studies without having to wait for the meta-analyses to be published.

Is there any support for this kind of revolution to be implemented? The ASCII files for an entire journal issue would be no larger than the average video clip of your grandson’s kindergarten graduation. And it would amount to a true revolution in publishing, one that would (pardon the expression) empower the average consumer of the literature.

[impotenta]

@Ed Whitney: Wow. Your comment just made my day. thanks