Statins, Diabetes, and Attacking a Meta-Analysis

I'm a little late reading the June 19th Lancet, but was intrigued to find letters in response to the meta-analysis by Sattar et al. looking at whether statin therapy increases the risk of diabetes.

I had previously written about this well-performed meta-analysis, and also written about some unfair ways that people use to try to attack randomized trials, and these letters provide an interesting (at least to me) intersection between these posts.

Letters in academic scientific journals are sociologically revealing. There's typically a polite veneer on even the most vicious attacks. Letters written to European medical journals have a somewhat different feel from those to American medical journals, and letters to the Lancet often seem to have a sneering tone that would be unusual to find in the NEJM or JAMA.

One letter about the meta-analysis objects that the results cease to be statistically significant when diabetes diagnosed only by physician report are excluded, and secondly that the results involved a post-hoc analysis of the data, with the warning that we might fall victim to the logical fallacy, "Post hoc ergo propter hoc".

Are these fair objections?

Diagnosing diabetes by physician report rather than blood glucose measurement is likely to lead to misclassification: some patients will be classified as having diabetes who don't, and some who have diabetes will be missed. In an RCT, though, misclassification like this will almost certainly be random as well, leading to random misclassification bias. Bias of this sort is toward the null hypothesis (no difference between the groups), as you can convince yourself of if you imagine that the classification is perfectly random such that there is no relation between the classification and diabetes. Under such perfect misclassification, the two groups would have equal numbers of patients classified as having diabetes and there would be no difference between treatment and control. In a meta-analysis that found higher rates of diabetes in patients receiving statins, misclassification bias can be expected to have somewhat reduced the true effect, not to have created an effect out of thin air.

The second objection might be called the "post-hoc-ergo-propter-hoc-fallacy fallacy". The actual fallacy is, of course, a way of saying that just because B follows A, you should not conclude that A caused B. This question of causality is central to epidemiologic research and one of the primary reasons for performing randomized trials, which have particular strengths when arguing for causality. The fallacy has nothing to do with performing post hoc analyses of trials. (To be fair, it's possible the letter writer understood this and was being humorous when writing of this fallacy.) The main problem with a post hoc analysis of a randomized trial is that it often involves multiple comparisons/data dredging, where statistical blips are likely to confuse the issue of what is a true effect. As discussed in my earlier post, a prime issue preceding this meta-analysis was whether JUPITER had found just such a random blip or detected a real problem. The meta-analysis' reason for being performed was primarily to answer this question, and in such a setting there is nothing at all concerning about going back to previously conducted RCTs and performing post hoc analyses looking for diabetes effects. No data dredging was involved, and the analysis should not be looked at askance simply for being post hoc. Revealingly, the meta-analysis found an increased risk of diabetes even when data from JUPITER were excluded.

A second letter complained that the analysis would have been better had it been carried out using hazard ratios rather than odds ratios. While this would likely be true, such an analysis was not possible given the information available to the authors, and it is hard to imagine why an OR analysis would have shown statins to be causing diabetes if it were not true. The same letter also re-raised the possibility that statins appeared to be causing people to have more diabetes by keeping them alive longer to develop diabetes. However, the authors had already addressed this in their meta-analysis and reiterated in their response to the letters that differences in survival were much too small to produce such an effect.

A third letter mis-states the definition of a type I error on its way to arguing that the meta-analysis should have used 99% confidence intervals (p-value cutoff of 0.01) for some reason that was not made terribly clear, but seemed related to concerns that a very large meta-analysis would be more likely to detect a spurious result. It is true that given the enormous N in the analysis, it was possible to find a statistically significant difference in diabetes rates that is likely of little clinical significance, but this has nothing to do with the truth or falsehood of the result itself. The letter also argues that the result is biologically implausible, though it does not seem implausible that a medication could increase diabetes rates during the time of a randomized trial, if only by raising blood sugars in patients near the margin between insulin resistance and diabetes.

A fourth letter suggests that the "diabetes" found in the study might be different in terms of patient-important outcomes than the clinical condition we think of as diabetes. That is, statins might be raising blood sugars in a way that is harmless. While this is possible, it's interesting that when a drug class raises blood sugar people are willing to argue it might be harmless, but when a drug class lowers  blood sugar there's a tendency (at least for the manufacturer) to argue that blood sugar control is an excellent surrogate for clinical outcomes. The author of the letter suggests an analysis that might have been done to sort out this issue, which the authors of the meta-analysis correctly point out would not have answered the question.

There were a few other replies to the article, which I have not detailed. Overall, though, this is a fairly typical picture of what happens when someone publishes a trial that conflicts with conventional beliefs, such as "statins are good". This occurs even when the conflict is quite minor -- the meta-analysis merely shows a small increase in diabetes that would be heavily outweighed by cardiovascular benefit in anyone who would be appropriately treated with a statin.

There is no guarantee that the meta-analysis by Sattar et al. is correct about statins and diabetes, but none of the letters published by Lancet raise a sensible reason to think that the post-analysis state of knowledge should change: it is now far more likely than not that statins cause a small increase in diabetes risk. Our response to a meta-analysis like this should be to congratulate the authors on a job well done, while recognizing the possibilities for errors and chance to disrupt the conclusions. It should not be to search high and low for far-fetched flaws that would allow us to discard the inconvenient likelihood that a new statin side-effect has been detected.

Comments

[sanjiva86]

What do you make of the statin articles in this week's Archives of Internal Medicine?

[David Rind]

I read through them today and need to spend a little more time on them.

First impressions:

The attack on JUPITER seems over the top. I have a number of issues with JUPITER, but the suggestion that it was the only study of its kind to suggest benefits in primary prevention seems like a willful misreading of the literature. It makes me suspicious of all the authors' analyses of "flaws" in JUPITER.

The meta-analysis is surprising for how small the point estimate of a mortality benefit is. I've thought that the real number was closer to 15% in primary prevention but this analysis concluded about 8% (barely missing statistical significance, but that's not really relevant). I'm wondering how much ALLHAT (which had significant flaws) dragged down the number, but overall it may be the case that the relative mortality benefit in primary prevention really is smaller than in secondary prevention. (The absolute benefit is, of course, much smaller than in secondary prevention.) The paper doesn't present estimates of reduction in MI and stroke, which is too bad -- it really does seem to be the first meta-analysis to really sort out patients without prior CVD. Presumably they are planning to publish a follow-up paper with those numbers. I'm uncertain why this paper didn't make it into NEJM, JAMA, or Annals and thus if it has a problem that the reviewers picked up but that I'm not noticing so far. (Anyone care to weigh in on this?)

[Brad F]

David
Any return to this post with a more informed analysis of Archives publications would be greatly appreciated. I too wondered why these were not published in one of the big three. I find this topic fascinating, and beyond the methodology and motivations of authors, the implications for population health, the dogma of "statins = good, put them in the water," and the public's awareness of this class of rx and the desire to take them make this kind of appraisal so important.

I found the analysis reasonable and did not walk away with any skepticism. However, I did sense an ax grinding away in these submissions--clearly they have opinions.

Thanks
Brad
NY, NY

[Dan Hackam]

David,
The failure of primary prevention trials to provide much in the way of a mortality benefit (versus the clear benefit in secondary prevention trials) means that trials - and we in clinical practice - are doing a very poor job of selecting the right types of patients who will benefit from statins.

We should be sorting out who is at higher risk in primary prevention - who in fact actually is a secondary prevention patient among the primary prevention "masses" by virtue of the fact that they have subclinical cardiovascular disease. In my case, that means carotid atherosclerosis imaging. I find many so-called primary prevention patients who actually have quite advanced disease. Since none of the primary prevention trials actually selected out for these types of patients, it is no wonder that they have failed to demonstrate a true mortality benefit. Therefore, we should not be selecting based on risk factors alone - that much seems clear from the meta-analysis, and from the individual trials themselves.

If you don't think this approach is evidence-based, you have to answer the counterfactual question, which is why the large subset of primary prevention patients with demonstrated atherosclerosis would NOT benefit proportionately from statin therapy (in relation to their baseline risk).

[Matt Hitron]

It would seem wise to keep the main points of your thoughtful post in mind when reviewing the maeltrom of editorial comments unleashed by the recent meta-analysis published in Lancet-Oncology on Angiotensin Receptor Blockade and risk of cancer. An unpopular result does not make a study "junk". The authors are careful to state that the clinical implications are unknown. (All the "he said-she said-I asked-they didn't reply" aside, the exclusion of the VALUE trial does appear to be an important limitation).