Evidence In Medicine

Pharma behaving well?

We have known for many years now that patients having an acute coronary syndrome (ACS) do better if they are immediately treated with aspirin, and that they do even better if in addition to that single antiplatelet agent they are also treated with clopidogrel.

In 2009, the PLATO trial was published in the NEJM. It was a randomized trial in more than 18,000 people having an ACS, comparing the benefits of adding either clopidogrel or the newer agent ticagrelor to aspirin. Ticagrelor appeared to cause some increased serious bleeding, but overall it reduced a composite endpoint of vascular death, MI, and stroke (hazard ratio 0.84). AstraZeneca, the manufacturer of ticagrelor, has to have been pleased.

Many subgroups were examined in PLATO, and in a few of them there was a suggestion of differences in effects across the subgroups; that is the benefit seen with ticagrelor was not as great or non-existent in three subgroups: those who weighed less, those not taking statins at enrollment, and those living in North America.

It's common when a trial examines multiple subgroups to find spurious positive results due to chance alone, given the multiple statistical tests being performed. For this reason, there has to be a good reason to bother to pay attention to a subgroup, and overly trusting subgroup analyses is one of the most common errors in thinking about what a study has shown. Given this, and the positive result in PLATO, it would have been typical for the authors to make short shrift of the subgroup analysis and write the results off to chance.

However, one of those subgroups seemed to worry the investigators. They wrote in the discussion section:

The difference in results between patients enrolled in North America and those enrolled elsewhere raises the questions of whether geographic differences between populations of patients or practice patterns influenced the effects of the randomized treatments, although no apparent explanations have been found.

They were likely concerned because the point estimate of effect in North America was that clopidogrel was substantially superior to ticagrelor, the opposite result from that seen elsewhere. However, ticagrelor was apparently approved by the US FDA without concern that the benefits seen in the trial as a whole would not apply to patients in North America, and that might well have been where things were left.

Instead, apparently working closely with authors/investigators at Duke, AstraZeneca carried out an extensive investigation to figure out whether the subgroup effect was real and, if so, what might have caused it. The article detailing this investigation was published in Circulation in 2011.

If you consider the problem, it is unlikely that genetic differences between North American patients and European patients could explain the differences. That suggests that if the finding was not caused by chance it was either due to some way in which patients were managed differently in North America (most likely) or some way in which patients enrolled in North America were consistently different (less likely), such as being sicker or older. Once you start looking for such possibilities, though, you are faced with the high likelihood of even more spurious results caused by chance. All the exploratory analyses are now post-hoc subgroup analyses -- those that you had not thought of before the primary analysis -- and so highly susceptible to findings due to the play of chance.

As they began looking into the issues, the investigators first found that the real difference wasn't between North America and the rest of the world, but really between the US and the rest of the world, so they refocused on the results when patients in Canada were included with the rest of the world. Then they checked in with major study sites to see if the trial was conducted differently in different locations -- it wasn't. Additional checks found that US sites seemed to have gotten the correct study drugs.

They looked at the play of chance, and concluded that results like those seen in North America would have been expected 32% of the time, and results like those seen in the US would have been expected in at least one country 10% of the time, so chance was certainly a possible explanation. Still, because US practices can be so different from those in many other countries, the investigators must have remained concerned.

The investigators then looked into many baseline and management variables to see if they could explain results and started focusing on aspirin dose as their investigation unfolded. At the US sites, patients were typically treated with  more than 300 mg of aspirin (likely 325 mg) daily, while in other locations patients typically received less than 100 mg daily. In statistical analyses, including aspirin dose appeared to explain 80-100% of the regional variation in outcomes.

Additional analyses that stratified by aspirin dose found a dose response effect that seemed similar across regions. Patients in the US who received low dose aspirin saw similar benefits with ticagrelor to patients in the rest of the world, while patients in the rest of the world who received higher dose aspirin saw similar benefits with clopidogrel to patients in the US; patients in the rest of the world who received 100 to 300 mg of aspirin had similar outcomes with either clopidogrel or ticagrelor:

Overall, though, patients receiving low dose aspirin plus ticagrelor had the best outcomes:

Still, this is a post-hoc subgroup analysis, and the results may have all been due to chance. The authors of the paper acknowledge this repeatedly while still suggesting that patients treated with ticagrelor only be given low dose aspirin. This seems like a completely appropriate conclusion. We can't know for sure whether aspirin or chance or some other difference in management in the US explains the results, and it is unlikely that another RCT will be conducted to answer the question, but the analysis from this paper really informs appropriate clinical decision making.

I am not routinely a fan of Pharma in general or of AstraZeneca in particular. One of my favorite posts from the old medical humor website Q Fever does a wonderful job poking fun at the way in which AstraZeneca promoted Nexium over omeprazole (Prilosec) once the latter drug went off patent. The trials used to claim superiority for Nexium are great examples of how Pharma can manipulate results by manipulating study design and interpretation.

But as far as I can tell, AstraZeneca deserves real credit for pursuing the subgroup analyses from the PLATO trial. I can't picture how they could have seen a marketing upside to this, since they had already found an overall benefit with ticagrelor. If the issue were never pursued, the regional variation in effect would likely have been ascribed to chance. Furthermore, it was possible that the in-depth investigation they participated in would have turned something up showing that ticagrelor was no better than clopidogrel or was even inferior.

Instead of resting on the positive result in PLATO, AstraZeneca seems to have moved forward on this and supported the authors/investigators from Duke who clearly also wanted to pursue the issue. In so doing, they found what is likely the right explanation for what really happened in the trial, which allows patients throughout the world who receive ticagrelor to be treated with the appropriate dose of aspirin to maximize the chances of a favorable outcome.

The study's authors and investigators deserve our thanks for pursuing this issue with hard work and logical thinking, but it seems likely that thanks are due to AstraZeneca and its employees as well.

Startling Results

I've written previously about how the plausibility of results might affect the interpretation of new evidence from a trial. In particular, I argued that results that cannot fit in with a web of prior evidence must be viewed with skepticism.

While being overly credulous of results that are unlikely to be true is a common problem in medicine, a less common but more severe problem is how unwilling doctors tend to be to believe a new result that upsets past dogma even when that dogma was built on low quality evidence.

Historical disbelief in startling results

I discussed in the past the unreasonable attacks that I watched academic physicians make on trials showing that H. pylori was causing ulcers. The classic example of this, though, in my medical lifetime is probably what occurred with hormone replacement therapy (HRT). Large, well-analyzed observational studies consistently found that in post-menopausal women, the use of HRT was associated with a reduced risk of coronary disease. Based on this, HRT became a central facet of recommended primary prevention in such women, despite the clear increased risk of breast cancer with HRT. The data showing about a 30% reduction in coronary events and reduced fractures with HRT were felt to more than offset the increased breast cancer risk.

Through the 1990s, more and more post-menopausal women were treated with estrogen. In 1998, the HERS trial was published in JAMA. It was a randomized trial of HRT, but in women who already had coronary disease. In HERS, HRT showed no benefit on coronary events and produced an increase in thromboembolic events and a nonsignificant trend toward higher overall mortality.

HERS was a startling result. If HRT was good for preventing coronary disease, you would have expected that it would be beneficial in preventing the progression of coronary disease. We knew that all the evidence we had for HRT was from observational studies that were at risk of confounding caused by healthier women choosing to take HRT. And yet the nearly universal reaction to HERS was to try to either figure out what was wrong with it or, more commonly , to explain how the observed effects of HRT for secondary prevention did not apply at all to HRT when used for primary prevention. These arguments tended to lose track of how much the opposite was predicted before the trial was conducted. Yet I watched as nearly all the academic primary care internists I knew continued to prescribe and recommend HRT.

When HERS was published, I stopped recommending HRT and started suggesting to women taking it that they might want to stop. I worked with one other physician who did the same and many who argued that HERS simply did not apply to their primary prevention patients. Then, in 2002, the results of the WHI trial were published after the trial was stopped for harm. This was a primary prevention trial of HRT and showed an increased risk of coronary disease. Word spread quickly, and HRT for primary prevention became a thing of the past, though for years after people still tried to explain away the results.

The present day

HRT and disbelief in an infectious cause of ulcer disease are really ancient history at this point, but the tendency of doctors to try to explain away randomized trial results that they should find startling and disturbing remains.

In May of 2011, the NEJM published a trial of giving IV fluids (either saline or albumin) to children in resource limited settings in Africa who appeared to have septic shock. This trial included 3100 very ill children, and I have to think that the IRB approving this trial must have worried about the ethics of having a control arm where such sick children with impaired perfusion were not given fluid boluses, since I would guess that very few clinicians would have thought there was equipoise between the likelihood of clinical benefit and harm to the intervention. The trial must have been intended as a prod to get local governments and international agencies to expend some resources to provide this minimal level of care for septic children.

Stunningly, the trial showed around a 45% increase in mortality in the children given fluid boluses. The researchers stopped the trial for harm and wrote a paper clear on the effects seen in resource limited settings, while pointing out that this raises some concerns for other settings. An accompanying editorial allowed for much greater concern about the generalizability of these results:

It seems clear that the results of this trial indicate that bolus-fluid resuscitation with either crystalloids or colloids in patients with compensated shock who do not have a clinical fluid deficit must be practiced with much greater caution than is now the case and with increased vigilance.

The editorialist goes on to argue for the urgent need for additional study around the questions raised by this trial.

The evidence we have for fluid resuscitation in children with compensated septic shock in resource rich settings is entirely observational. Mortality rates are fairly low (in the 1% to 2% range) and so an individual clinician would be unlikely to notice harm from a therapy that increased mortality by 45%.

And yet...

I have heard and read multiple experts try to explain away the generalizability of the trial. A common aspect of the arguments seems to be a belief that the burden of proof lies with those who want to generalize from these results. They seem to think that because giving fluids is standard practice that this means the practice is supported by high quality evidence, but this just is not the case.

I expect that nearly everyone who I have heard or read attack this trial would have never thought the intervention could cause a 45% increase in mortality in Africa. That is because they thought the benefits they attributed to fluid resuscitation were generalizable. As such, they should both have been completely shocked by this result, and have been extremely worried that the results generalize back from resource limited settings to resource rich settings. They should have examined the basic premises and evidence around fluid resuscitation, in just the way the editorialist did.

Instead, predictably, people tried to figure out how to defeat the trial and prove it could not possibly apply to resource rich settings. The main arguments made seem to be either that many of these children had malaria, or that many of them had severe anemia, and that in both settings there is a suspicion that fluids can be harmful.

Letters in response to the article make both these arguments. This is odd, since the data presented in the paper show them clearly to be unreasonable attacks. The increase in mortality with fluid boluses in children with malaria was 59% and in children without malaria was 43%, and these results were not significantly different from each other. The increase in mortality with fluid boluses in children with severe anemia was 71% and in children without severe anemia was 31% and again these were not significantly different from each other (and note that even if the real increase in mortality in children without severe anemia is lower than in those with severe anemia and the trial did not have the power to show this, the point estimate is still a one-third increase in mortality). One of the letter writers points out that the increase in mortality in those without severe anemia was not statistically significant, but this is a completely unfair argument. Once an increase in mortality is seen in the group as a whole, you can always find subgroups that by virtue of having fewer patients will have a similar mortality rate that does not pass a given level of statistical significance.

All who were startled by the results of this trial should be much less convinced of the benefits of fluid resuscitation in compensated sepsis and should be wary of its use. They should be pushing for trials in resource rich settings, and while awaiting those results should keep the possibility that administering fluids may increase mortality in the front of their minds however they decide to treat septic shock. Reasonable people may still choose to cautiously administer such fluids for compensated shock in resource rich settings, but they should be anxious every time they do so.

Personal notes

The authors of this trial wrote in the paper of how surprised they were by the results and at one point say:

Although fluid boluses adversely affected the outcome, important survival gains, across all groups, may have resulted from training and implementation of triage, basic life-support measures, and regular observation.

When I was reading the paper, it wasn't until I read those words that I realized the impact of this trial not just on the children but on the researchers. There were around 70 excess deaths in the fluid bolus arms of the trial; 70 children who likely died because of the intervention. We can imagine the affect on parents and families, but this must also be a researcher's worst nightmare. This was an incredibly difficult study to carry out, and those who did so must have thought they were heading down a path that would ultimately save innumerable children's lives in resource limited settings. Instead, perversely, the intervention killed around 70 children and those same researchers are likely having trouble sleeping and comforting themselves with the knowledge that the training from the study may have helped save some children overall. They deserve our honor for a trial well designed and executed and our sympathy for what must be the emotional toll.

For me, I remember a man in his 30s who was admitted to my service in septic shock near the end of my internship. We gave him innumerable fluid boluses over the course of 12 hours as he inexorably moved towards death. When I think of my hardest moments as a doctor, telling his 13-year-old daughter in the middle of those 12 hours that her father was not doing alright has pretty much been at the top ever since that day. Now, after reading this trial, I can add to that the worry that those repeated fluid boluses didn't merely swell him with edema past the point of recognizability, but may actually have contributed to his death.

 

Not so SHARP

I've written a bit previously in this blog about the evidence around lipid-lowering therapies, and in particular about the lack of evidence for combining other lipid-lowering therapies, including ezetimibe, with statins.

Tomorrow, the FDA will apparently be taking up a request to approve Vytorin (simvastatin/ezetimibe) for the treatment of patients with chronic kidney disease (CKD). As a result of this request for approval, we also have access to the FDA review of data from the SHARP trial.

SHARP was published in Lancet in June of 2011. SHARP was a trial comparing simvastatin/ezetimibe with placebo in nearly 9300 patients with chronic kidney disease. By way of background, it is important to know that two prior trials of statins in patients on dialysis, 4D and AURORA, had failed to find a statistically significant reduction in events with statin therapy. The non-significant point estimates were about an 8% reduction in events in 4D and a 4% reduction in AURORA.

That makes end-stage renal disease (ESRD; essentially, patients requiring chronic dialysis) an unusual condition when it comes to statin therapy. People with ESRD are at very high risk for cardiovascular events, but get little benefit from statins, unlike almost every other high risk group.

Into this comes simvastatin/ezetimibe, a drug without a clear use. Ezetimibe has never been shown to add clinical benefit to treatment with simvastatin, but based on surrogate outcomes the combination drug became widely prescribed until other surrogate outcomes made  people concerned that ezetimibe might really be worthless or worse.

Perhaps that made CKD seem like fertile ground for exploring benefits with simvastatin/ezetimibe. The SHARP trial when it was published was promoted as showing unique benefits in CKD including, for the first time, in patients with ESRD.

The abstract says things like:

After weighting for subgroup-specific reductions in LDL cholesterol, there was no good evidence that the proportional effects on major atherosclerotic events differed from the summary rate ratio in any subgroup examined, and, in particular, they were similar in patients on dialysis and those who were not.

and

Reduction of LDL cholesterol with simvastatin 20 mg plus ezetimibe 10 mg daily safely reduced the incidence of major atherosclerotic events in a wide range of patients with advanced chronic kidney disease.

So, based on SHARP, the claim is being made that simvastatin/ezetimibe, unlike statin monotherapies before it, works in patients on dialysis because it reduced events in all patients with CKD, and the effect in the subgroup of patients on dialysis was not statistically significantly different from the effect in those not on dialysis.

Is this a fair argument?

In general, we look askance at claiming that effects in subgroups differ from the overall effect in a randomized trial. While people are always asking "does drug X really work in women? the elderly? blacks?", in reality most interventions work similarly across a broad patient population. So why object to such a standard, and seemingly reasonable argument for simvastatin plus ezetimibe?

There are two interrelated main problems:

The first is that we are not starting from a position of believing that the effect of a statin-based intervention should be the same in patients with and without ESRD. Based on what happened in prior trials of statins that included less severe CKD, where statins seemed to have their usual benefits, and the 4D and AURORA trials where they had much smaller benefits if any, we already knew that the ESRD subgroup was likely important and special.

The second is that the actual results are in no way reassuring that the subgroups were similar, even if they were not statistically significantly different. Among all patients in SHARP, simvastatin/ezetimibe reduced events by 17%. However, in the 6247 patients not on dialysis it reduced events by 22%, and in those on dialysis it reduced events by a non-significant 10%. Those numbers look rather far apart given our a priori concerns, and even if not statistically significantly different from each other, 10% seems reminiscent of 4D and AURORA.

Still, 10% is a larger reduction than the 8% in 4D or the 4% in AURORA, right? Well, only if you look at all the dialysis patients in SHARP. The 4D and AURORA trials only enrolled patients on hemodialysis, while SHARP enrolled patients both on peritoneal dialysis and hemodialysis. In the small number of peritoneal dialysis patients in SHARP there was a non-significant 30% reduction in events, which is what brought down that average number for all dialysis patients. If you looked just at the 2500 hemodialysis patients, the population studied in the earlier trials, the reduction in events was just a non-significant 5%, similar to AURORA and not even as good as 4D.

Given, this, the argument is completely unpersuasive. Failing to show a statistical difference between effects in the dialysis subgroup and the non-dialysis subgroup is just a way of hiding the reality that simvastatin/ezetimibe did not produce clinically significant or statistically significant benefits in patients on hemodialysis.

So what are we left with? I think we can confidently say after SHARP that simvastatin/ezetimibe reduces cardiovascular events in patients with CKD who do not have ESRD, and that this reduction is right in the ballpark of what we would have expected with simvastatin alone. Further, we can note that yet again a statin-based treatment showed little or no benefit in patients on hemodialysis and that there is absolutely nothing in the results from SHARP to suggest that ezetimibe provides some added clinical benefit to simvastatin alone in either patients with earlier stage CKD or patients with ESRD.

As was the case before SHARP, we have no convincing evidence that any lipid-lowering therapy added to a statin improves clinical outcomes in any patient population.

Activated Protein C and Conflicting Trials

I won a bottle of wine this week. At least in theory. I haven't posted to this blog in a long while, but the opportunity to brag about this has overcome inertia.

A few years ago I was at a cookout with some friends and colleagues, and got into a discussion with one of them, an emergency physician and sepsis expert, about activated protein C.

Many modifiers of the immune response to sepsis have been studied in an attempt to lower the large mortality rate of severe sepsis, but almost none have shown any benefit. Then, in 2001, a study of activated protein C in severe sepsis was published in the NEJM, and showed a substantial reduction in mortality.

I was suspicious from the beginning, and commented that there had been enough trials in sepsis that we were bound to get a positive result by chance alone. My skepticism, not surprisingly, had little effect on the rapid rise of activated protein C and its manufacturer Lilly. It became a central part of the management of sepsis.

In the original trial there was some suggestion that activated protein C worked better in patients at high risk for death than in those at lower risk. This was somewhat counter-intuitive, though certainly possible. To answer the question and expand the indications for the drug, a study was conducted to look at patients at lower risk of death. This study, published again in the NEJM in 2005, found no suggestion of benefit in patients with severe sepsis at lower risk of death and was stopped early for futility. Additionally, patients treated with activated protein C were at increased risk for serious bleeding. My skepticism about the drug grew.

Then, in 2007, a study was published in the Lancet, showing no benefit of activated protein C in children with severe sepsis.This study, too, was stopped early for futility and an increase in CNS bleeding.

And so, in the setting of these trials, I found myself debating the merits of activated protein C with a sepsis expert. He took what was, up until last week, the pretty standard position of critical care physicians that what this sequence of studies showed was that activated protein C was only effective in adults with severe sepsis who were at high risk for mortality; it was ineffective or harmful in adults at lower risk for mortality from sepsis and for all children with sepsis.

In contrast, I took the position that the first trial had been a statistical blip and that, in reality, activated protein C was useless or harmful in treating sepsis. At the time of our debate, a very large international trial was getting under way to answer this question, and so we wagered a bottle of wine on the results.

These two ways of looking at trial results come up in all sorts of contexts. One option, when trials seem to disagree with each other, is to try to find the thread that resolves the conflict: the specific patient population that benefited in one trial, or the specific dose that made all the difference, or the specific type of location that explains how a drug could show benefits in one trial and be useless in another. As a web of trials is put together, people will often argue that the "evidence-based" approach is to look to the trials as trees and find, for instance, those medium height oaks that are sure to benefit from a particular therapy that is of no benefit for shorter or taller oaks, or for maples of any height.

Occasionally this approach does make sense, but more often, looking at the forest, it becomes clear that a single positive trial result amongst a network of negative trials (or vice-versa) may be better explained by chance or errors in the conduct of the study than by looking to the individual trees.

Researchers get things wrong, by being unlucky, by making errors, by being foolish, by fabricating data, and for any number of other reasons. When a study result feels unbelievable for good reason, it often should not be believed. In an earlier post I argued that XMRV would not turn out to be the cause of chronic fatigue syndrome. It now appears that lab error was responsible for the early results suggesting such a connection, but I certainly had no training or special knowledge to have predicted what might have gone wrong; I had only the clinical knowledge that I thought finding any single pathogenic cause of CFS was extremely unlikely. When neutrinos were found moving faster than the speed of light, scientists were appropriately even more skeptical. (The medical equivalent of this, to my mind, are the occasional studies suggesting benefits from homeopathy.)

With activated protein C, the best way to explain the data seemed to be to assume that the one positive trial was incorrect while awaiting the results of the large international trial. Those results have not been published in final form, but the trial was stopped early for futility this week and activated protein C was withdrawn from the market. My friend the sepsis expert owes me a bottle of wine. I'd certainly trade in that bottle of wine, though, if I could figure out a way to get expert clinicians and clinical epidemiologists to step back from apparently conflicting clinical trials and think about what the results as a whole are suggesting, rather than trying to believe that being fair to the evidence involves taking each individual trial result as true. This latter approach does not sufficiently consider the probability that when trials seem to conflict, some of them are likely to simply be wrong.

 

Independent Events

I was speaking with a patient recently who was explaining why he thought that colon cancer screening with fecal occult blood testing (FOBT) was equivalent to colonoscopy. He wished to avoid colonoscopy for all the usual reasons, but also believed logically that it was not a superior test.

I won't get into the actual evidence behind the two methods of screening; that is not an issue for this post. Instead, I want to discuss his argument. He assumed that colonoscopy had a sensitivity (probability of the test being positive in someone with a significant colon lesion) of 98%, and this figure is in the ballpark of believable numbers. Let's assume that the sensitivity of FOBT is around 35% (also a reasonable ballpark figure). At that rate, he said, since colonoscopy is performed once every 10 years, and FOBT is performed yearly, after 10 years the performance of the two tests is similar with FOBT having a cumulative sensitivity of 99%.

There are two important problems with this argument.  The first, which I will just mention in passing, is also the more obvious. If a test has to be repeated for 10 years to detect a cancer, that cancer may grow and become incurable during the testing period. Had it been found at year 1, it might have had a better chance of being cured.

The second problem is the focus of this post, and to examine it, we need to understand the calculation the patient was performing to get to a cumulative sensitivity of 99%. This is the same type of calculation that is often presented when thinking about the probability of an abnormal lab test due to chance alone in a battery of lab tests. The argument goes as follows:

1. We define "normal" in a lab test that has a continuous result (like serum sodium) as the range of values that captures 95% of healthy patients.
2. That means that 5% of healthy patients (or 1 in 20) will have an "abnormal" result on the test.
3. If we run a battery of different tests on a patient, each of which has a similarly defined normal range, the probability of a single abnormal result due to "chance" goes up.

The actual calculation of the likelihood of an abnormal test result typically confuses medical students and early residents until they've heard it presented repeatedly. A common assumption is that if there is a 1 in 20 chance of an abnormal result on each test, then if 20 tests are run there will definitely be an abnormal result. This is not the correct calculation. Under the usual assumptions that people make in thinking about this, the calculation would be that the probability of all the tests being normal is the probability of a single test being normal raised to the power of the number of tests.

Thus, for 20 tests it would be 0.95^20, which is 0.36. The probability that at least one such test will be abnormal due to "chance" is 1-0.36 or 0.64. Or, about 2/3 of normal patients would be expected to have at least one abnormal test on a battery of 20 tests under these assumptions.

So the patient utilizing FOBT for colon cancer screening was saying that, with a sensitivity of about 35%, he could expect a false negative rate of 0.65 per test but that 0.65^10 (for the ten years of testing) yielded a miss rate of 1%. This cumulative sensitivity would then be similar to that of a single colonoscopy, so why should he get the invasive procedure?

And so we come to the second problem with the patient's argument: each round of testing is not an independent event.

The calculations I described above for cumulative probabilities make the assumption that the individual events are independent from each other. That is, the result of one test has no influence on the others.

If you flip a fair penny three times and get three heads, there is still a 1 in 2 chance that it will come up heads on the fourth flip. But if you perform FOBT once for colon cancer and it is negative it might be in the setting of your particular precancerous lesion that doesn't tend to bleed. If so, it's less likely to be bleeding on subsequent FOBT than an "average" lesion and so the cumulative sensitivity cannot assume independence of events. This is not just a theoretical issue: based on some research, repeated testing is thought to actually have a cumulative sensitivity of around 85%, not 99%.

So the patient was miscalculating in his decision about how to be screened for colon cancer. I actually briefly discussed this with him during the appointment, but since the real issue was that he did not want a colonoscopy he was singularly uninfluenced by the math.

This plays out in other areas as well, though:

What about that standard example above regarding batteries of lab tests that all medical students and residents are taught? The tests in the battery, too, are clearly not independent events. Normal and abnormal tests tend to cluster and so it is likely that the probability of the 20th test being abnormal in a healthy patient is affected by whether the prior 19 tests included any abnormal results.

To contrast with the assumption of probabilities in the face of independent events, here we are talking about "conditional" probabilities where we want to know the likelihood of an event given some other set of events. However, in the real world we have very few data about these situations. If, for instance, I wanted to know how likely, due to random variation, a patient with an abnormal serum sodium and chloride is to have a high serum potassium, it is extremely unlikely that I could get a high quality answer without doing my own primary research.

This problem of not knowing conditional probabilities when faced with non-independent events has an important effect on how diagnostic strategies might be misinterpreted if clinicians really started utilizing a test parameter that is a favorite in the EBM community but has not really permeated the clinical world. I'll address this in a future post.

Statins, Diabetes, and Attacking a Meta-Analysis

I'm a little late reading the June 19th Lancet, but was intrigued to find letters in response to the meta-analysis by Sattar et al. looking at whether statin therapy increases the risk of diabetes.

I had previously written about this well-performed meta-analysis, and also written about some unfair ways that people use to try to attack randomized trials, and these letters provide an interesting (at least to me) intersection between these posts.

Letters in academic scientific journals are sociologically revealing. There's typically a polite veneer on even the most vicious attacks. Letters written to European medical journals have a somewhat different feel from those to American medical journals, and letters to the Lancet often seem to have a sneering tone that would be unusual to find in the NEJM or JAMA.

One letter about the meta-analysis objects that the results cease to be statistically significant when diabetes diagnosed only by physician report are excluded, and secondly that the results involved a post-hoc analysis of the data, with the warning that we might fall victim to the logical fallacy, "Post hoc ergo propter hoc".

Are these fair objections?

Diagnosing diabetes by physician report rather than blood glucose measurement is likely to lead to misclassification: some patients will be classified as having diabetes who don't, and some who have diabetes will be missed. In an RCT, though, misclassification like this will almost certainly be random as well, leading to random misclassification bias. Bias of this sort is toward the null hypothesis (no difference between the groups), as you can convince yourself of if you imagine that the classification is perfectly random such that there is no relation between the classification and diabetes. Under such perfect misclassification, the two groups would have equal numbers of patients classified as having diabetes and there would be no difference between treatment and control. In a meta-analysis that found higher rates of diabetes in patients receiving statins, misclassification bias can be expected to have somewhat reduced the true effect, not to have created an effect out of thin air.

The second objection might be called the "post-hoc-ergo-propter-hoc-fallacy fallacy". The actual fallacy is, of course, a way of saying that just because B follows A, you should not conclude that A caused B. This question of causality is central to epidemiologic research and one of the primary reasons for performing randomized trials, which have particular strengths when arguing for causality. The fallacy has nothing to do with performing post hoc analyses of trials. (To be fair, it's possible the letter writer understood this and was being humorous when writing of this fallacy.) The main problem with a post hoc analysis of a randomized trial is that it often involves multiple comparisons/data dredging, where statistical blips are likely to confuse the issue of what is a true effect. As discussed in my earlier post, a prime issue preceding this meta-analysis was whether JUPITER had found just such a random blip or detected a real problem. The meta-analysis' reason for being performed was primarily to answer this question, and in such a setting there is nothing at all concerning about going back to previously conducted RCTs and performing post hoc analyses looking for diabetes effects. No data dredging was involved, and the analysis should not be looked at askance simply for being post hoc. Revealingly, the meta-analysis found an increased risk of diabetes even when data from JUPITER were excluded.

A second letter complained that the analysis would have been better had it been carried out using hazard ratios rather than odds ratios. While this would likely be true, such an analysis was not possible given the information available to the authors, and it is hard to imagine why an OR analysis would have shown statins to be causing diabetes if it were not true. The same letter also re-raised the possibility that statins appeared to be causing people to have more diabetes by keeping them alive longer to develop diabetes. However, the authors had already addressed this in their meta-analysis and reiterated in their response to the letters that differences in survival were much too small to produce such an effect.

A third letter mis-states the definition of a type I error on its way to arguing that the meta-analysis should have used 99% confidence intervals (p-value cutoff of 0.01) for some reason that was not made terribly clear, but seemed related to concerns that a very large meta-analysis would be more likely to detect a spurious result. It is true that given the enormous N in the analysis, it was possible to find a statistically significant difference in diabetes rates that is likely of little clinical significance, but this has nothing to do with the truth or falsehood of the result itself. The letter also argues that the result is biologically implausible, though it does not seem implausible that a medication could increase diabetes rates during the time of a randomized trial, if only by raising blood sugars in patients near the margin between insulin resistance and diabetes.

A fourth letter suggests that the "diabetes" found in the study might be different in terms of patient-important outcomes than the clinical condition we think of as diabetes. That is, statins might be raising blood sugars in a way that is harmless. While this is possible, it's interesting that when a drug class raises blood sugar people are willing to argue it might be harmless, but when a drug class lowers  blood sugar there's a tendency (at least for the manufacturer) to argue that blood sugar control is an excellent surrogate for clinical outcomes. The author of the letter suggests an analysis that might have been done to sort out this issue, which the authors of the meta-analysis correctly point out would not have answered the question.

There were a few other replies to the article, which I have not detailed. Overall, though, this is a fairly typical picture of what happens when someone publishes a trial that conflicts with conventional beliefs, such as "statins are good". This occurs even when the conflict is quite minor -- the meta-analysis merely shows a small increase in diabetes that would be heavily outweighed by cardiovascular benefit in anyone who would be appropriately treated with a statin.

There is no guarantee that the meta-analysis by Sattar et al. is correct about statins and diabetes, but none of the letters published by Lancet raise a sensible reason to think that the post-analysis state of knowledge should change: it is now far more likely than not that statins cause a small increase in diabetes risk. Our response to a meta-analysis like this should be to congratulate the authors on a job well done, while recognizing the possibilities for errors and chance to disrupt the conclusions. It should not be to search high and low for far-fetched flaws that would allow us to discard the inconvenient likelihood that a new statin side-effect has been detected.

Composite Endpoints and the CREST Trial

In recent years, a common practice in clinical trials is to make the primary endpoint a composite endpoint. That is, the primary endpoint is a combination of various clinical events that might happen, such as heart attack or death or stroke, where any one of those events would count as part of the composite endpoint.

There can be good reasons for composite endpoints. If a single therapy is likely to have similar effects on a number of different related endpoints (such as cardiovascular events) creating a composite can help achieve adequate power. For instance, if statins decrease nonfatal strokes, nonfatal MIs, and cardiac deaths by about 25%, then creating these endpoints as a composite may make it possible to run an adequately powered trial with fewer patients. Additionally, there may be situations were a therapy might decrease an outcome by killing patients off before they can reach that outcome, and so a composite may be a way of evaluating safety.

Composites can quickly get you into trouble, though, if you combine events of very different importance to patients. Sometimes this appears to have been done with the intention of obscuring the real outcome of a trial or to make a therapy look far better than it really is. The primary outcome of DREAM, for instance, was "the development of diabetes or death". DREAM was a randomized trial that looked at the effects of rosiglitazone in patients at high risk for developing diabetes. The article states in the Discussion:

This large, prospective, blinded international clinical trial shows that 8 mg of rosiglitazone daily, together with lifestyle recommendations, substantially reduces the risk of diabetes or death by 60% in individuals at high risk for diabetes.

A reasonable person reading this might conclude that rosiglitazone had had a rather dramatic and important effect on the risk of death in patients in the DREAM trial. In reality, however, the study found no effect of rosiglitazone on mortality (and was always much too small to find any difference), and the primary outcome was driven entirely by development of diabetes. Additionally, since development of diabetes was assessed while patients were taking rosiglitazone (a drug that reduces blood sugar), it was virtually certain that the arm taking rosiglitazone would appear to have a lower rate of "diabetes and death" simply because development of diabetes is based on blood sugar and A1c measurements. Even if rosiglitazone had no effect on the natural history of diabetes, it could be counted on to reduce blood sugar and A1c during the time patients were taking it, and thus appear to delay the onset of laboratory-diagnosed diabetes. Worse, rosiglitzaone has harmful cardiac effects that could be minimized by focusing on a dramatic reduction on a primary endpoint that included the word "death".

It hardly seems fair or true to conclude that rosiglitazone decreased "diabetes and death", but by making that the primary composite endpoint, the trial can claim that this is the only statistically fair way to report the results.

With this in mind, we are now faced with the CREST trial, published online in the New England Journal. CREST was a randomized trial that compared two methods of dealing with carotid vascular disease: carotid endarterectomy and carotid stenting. The primary endpoint was a composite of periprocedural stroke, myocardial infarction, or death, or ipsilateral stroke in the following four years.

CREST found no difference between the procedures in the primary endpoint, but this obscures what really happened. Unsurprisingly, patients treated with stenting, a less invasive procedure, had lower rates or periprocedural MI (1.1 versus 2.3 percent). But they also had higher rates of periprocedural stroke ( 4.1 versus 2.3 percent), as well as a trend toward higher mortality.

I suspect most people undergoing a procedure to avoid having a stroke (the rationale behind both endarterectomy and stenting) would hesitate to accept a 1.2 percent gain in the form of a lower risk of MI in return for a 1.8 percent loss in the form of a higher rate of stroke. As such, the apparent equivalence on the primary endpoint obscures the importance of the underlying components of the composite.

The article was much more upfront about these individual outcomes than the DREAM article, but this did not alter a general buzz in the press (presumably traceable back, in some fashion, to the way the results were promoted) that CREST had shown equivalence between stenting and endarterectomy, and that stenting had now proven itself in a randomized trial.

Hopefully, though, when doctors talk to patients about the choice between carotid endarterectomy and carotid stenting, they will focus on the comparative risks of stroke and MI, and not on the composite outcome, since I suspect very few patients would consider a stroke and an MI to be outcomes that could be fairly equated and thus lumped into a single basket.

Rooting for Results

A few weeks ago, coinciding with its presentation at the ACC meeting, someone sent me the NEJM online publication of the ACCORD Lipid trial.

ACCORD Lipid, performed in patients with type 2 diabetes being treated with simvastatin, compared adding fenofibrate or placebo. I realized as I read the title and saw the methods in the abstract that I had a rooting interest in the results.

In the past few years, I have written about lipid management, argued about it, and had my judgments vociferously called into question. I've posited that whether or not LDL is important in the pathogenesis of coronary heart disease (CHD), what we really know is that statins reduce adverse cardiovascular outcomes and mortality, and further that we have little evidence that any other lipid-lowering agent (with the possible exception of niacin) actually reduces mortality.

While I'm certainly not alone in questioning the efficacy of non-statin lipid agents, my position has been outside the mainstream. I'd made this point publicly and specifically about ezetimibe at a time when there were no data on clinical endpoints. That position became more mainstream after the ENHANCE trial was published (though it too looked only at a surrogate endpoint), but most experts seemed to feel that ENHANCE only cast doubt on ezetimibe and not on the larger group of non-statin agents. Three days before ACCORD Lipid was published, I was making this argument yet again in a resident journal club meeting, where the article being discussed was analyzing a statin-centric approach to lipids that the article credited, at least in part, to me.

So, within that context of being publicly identified with one position in the argument, and a minority position at that, I read the abstract of ACCORD hoping that it would show no benefit from adding a fibrate to statin therapy, since, if adding a fibrate worked, much of what I had been arguing was likely wrong.

It probably goes without saying -- but I'll say it anyway -- that this is no way to approach a scientific result. The best way to approach a result in a paper and fit it into prior evidence is from a position of disinterest. It allows you to realize that peptic ulcer disease is an infectious disease and estrogen supplementation causes heart attacks, even if, prior to the pivotal publications, you had been treating PUD as a psychological disorder or recommending that female friends and family members take estrogen supplementation.

Yet, when I got to the results of ACCORD Lipid, even though it might have been better for my patients had fenofibrate been of benefit, I was clearly pleased to see that the conclusion was negative:

The combination of fenofibrate and simvastatin did not reduce the rate of fatal cardiovascular events, nonfatal myocardial infarction, or nonfatal stroke, as compared with simvastatin alone. These results do not support the routine use of combination therapy with fenofibrate and simvastatin to reduce cardiovascular risk in the majority of high-risk patients with type 2 diabetes.

But, of course, nothing is ever that clear cut. The authors suggested that fenofibrate might have been of benefit in the subgroup of patients with both high triglycerides and a low HDL-C. In these patients, mortality was 5% lower with fenofibrate with an interaction p value that was borderline significant at 0.06. They even were allowed to make this point within the abstract.

To me, that this seemed like a stretch would be putting it mildly. The paper had looked at at least 10 subgroups, suggesting that even a statistically significant p value in a subgroup would be dubious. Furthermore, fenofibrate had shown no hint of additional benefit in patients with high triglycerides or low HDL-C, but only in patients with both.

Yet, the authors argued that this subgroup result was consistent with post-hoc analyses of a number of prior trials, and the editorial accompanying the paper took the argument quite seriously and stated that the value of fenofibrate in such patients remained unsettled.

So, was I right dismissing this subgroup analysis as likely no more than a statistical blip, or were these others right in thinking it pointed toward an important subgroup of patients who could benefit from fibrate therapy?

I'd like to know that in thinking about this, I had approached the data from a position of objective neutrality, but as I discussed above, I already knew I hadn't. I also knew that the authors of the paper and editorial all had their own intellectual agendas as well.

Conflicts of interest come in many shapes and sizes. But even without such conflicts influencing the interpretation of results, the desire to have been right in the past, and for their current research to have been of value, causes important, and perhaps overriding, biases in experts as they confront and interpret new evidence.

Meta-analysis and New Knowledge

When hierarchies of evidence are listed for the EBM world, meta-analyses of randomized trials generally sit at the pinnacle.

And yet, the actual meta-analyses that you encounter when researching a clinical question can be far less enlightening. Even if we grant a pass to the many systematic reviews at The Cochrane Collaboration that conclude with the a priori obvious fact that no high quality RCTs addressing a question have been performed, and another pass to the reviews that find a single RCT and publish its results as the results of the systematic review, we are still left with the innumerable meta-analyses that seem to provide less of a window on truth than the underlying trials.

Frequently such meta-analyses are either driven by the single large RCT that everyone would have cited anyway or, worse, a number of small, poorly-performed RCTs are combined with a moderate-sized, well-performed RCT and alter the results away from what was likely the best estimate of reality: the results of the well-performed RCT.

Meta-analysts often seem to either be too removed from their subject area and thus lack the expertise to really understand what went clinically right and wrong in the underlying RCTs (or be unwilling to use that knowledge to discriminate among the trials), or be too cozy with a single trial (typically as an author) and thus too willing to ding trials that found conflicting results.

Ultimately, meta-analysis only rarely seems to importantly advance our knowledge of an issue beyond where we would have found ourselves by just reading through the RCTs.

So with that background it is always interesting to me when a meta-analysis comes along that really seems to shed new light on a subject such that we seem to know something that we somehow didn't know when we just had the underlying trials.

An example came along in The Lancet last week.

Despite the enormous number of patients participating in randomized trials of statins, it has been uncertain what effect statins have on the development of diabetes. Some biochemical and animal studies suggested that statins might prevent diabetes. Clinical trials have been conflicting with some showing protection and other showing increased risk. In reviewing the underlying trials, it has been hard to figure out what is going on:

• Are some statins protective while others are harmful?
• Are hydrophilic statins having different effects than lipophilic statins?
• Was the observation of increased diabetes risk in the JUPITER Trial just a random event that became noticeable because of reporting bias (where positive or interesting secondary outcomes are more likely to show up in a paper than negative results).
• Are the varying results of the statin trials due to random variation around a single truth, or do the results suggest that the underlying trials differed from each other in some important way (perhaps because of the population studied, the way the statin was administered, or the way diabetes was assesses?

A month ago, anyone simply looking at the collection of trials would have had a hard time giving a coherent answer to the above questions. Now, after a nicely done meta-analysis by Prof. Naveed Sattar et al., there are reasonable answers to all these questions. And thinking about these questions also sheds light on how to read and judge a meta-analysis.

The new analysis found that patients treated with statins had about a 9% higher risk of diabetes than those treated with placebo or other agents. When I started reading the analysis, I had the questions in the list above already in mind and so was prepared to challenge the meta-analysis on several fronts. The authors of the analysis had appropriately anticipated my concerns and, to the extent the data allowed, answered them:

1) Was this really a chance finding driven by JUPITER? Before JUPITER found an increased risk of diabetes, there had been little discussion of statins and diabetes risk. JUPITER's findings could have been due to chance, but the publicity around the result could have triggered the meta-analysis. JUPITER was large enough to sway the results in the meta-analysis and perhaps lead to a self-fulfilling conclusion based in random variation. The meta-analysis, though, did a secondary analysis that excluded JUPITER, and found that the results were essentially the same.

2) Were the varying results in the trials due to random variation or true differences? The meta-analysis found little need to invoke anything more than randomness (as measured by a statistic called the I²⁾. What had seemed to be conflicting results was likely nearly entirely due to random variation around a likely single true effect of slightly increased risk of diabetes.

3) Are some statins protective while others cause diabetes? The finding of little heterogeneity suggests the answer is no, but ultimately this is a hard question to answer definitively because of the more limited data about each individual statin. The meta-analysis found that the confidence intervals of the effects for individual statins overlapped such that it seemed unlikely that there were important differences among the statins, but it's hard to be certain. Additionally, lipophilic and hydrophilic statins showed the same effects on diabetes. And beyond that, the meta-analysis found that one of the main trials that had suggested a protective effect of pravastatin on diabetes had used an unusual definition of diabetes, and the effect was not seen when they substituted a standard definition.

While no new trials were published, as a result of this meta-analysis we have a much better feel for the effect of statins on diabetes than we had a few weeks ago. So, if after hours of trying to answer clinical questions by reading Cochrane you find yourself wondering whether meta-analyses are ever worth the effort that seems to go into them, remember this one and how much we learned about diabetes and statins from a new analysis of existing data.

Evidence and Snake Oil

Recently a friend wrote to me and asked what I thought of a presentation she had heard about a product containing beta-glucan: 

Presentation part 1

Presentation part 2

Presentation part 3

I'm not suggesting anyone spend the time to watch all of this, but it is a 30 minute talk at a meeting (of salespeople, I think) of a company planning to market a version of an existing product with a preparation of beta-glucan added to it.

The speaker is excellent, and his audience is clearly impressed to have such a knowledgeable scientist speaking to them, and excited at the chance to market a new healthful product that the speaker promises will be one of the major revolutions in health in this half century or even century.

A claim like that is a bit astounding, and would obviously require high quality evidence to back it up. As Carl Sagan used to say, "extraordinary claims require extraordinary evidence". That someone would make such claims at a sales meeting, such that the attendees were hearing about this revolution in health there first, rather than from the mainstream media reporting on a health breakthrough, might cause the incredulous to suspect they were about to learn about snake oil.

The credulous, on the other hand, might believe that meetings like this are exactly where health breakthroughs are presented because the mainstream media and mainstream doctors would never allow such information to be presented in a medium that they control.

The speaker talks of the astounding immune stimulating virtues of beta-glucan to repair the ravages that modern society has wreaked on our immune systems and points to a number of clinical trials. He goes on to describe the benefits shown in the clinical trials (mainly for upper respiratory tract infections -- URIs) in glowing terms.

Since I'd been asked by a friend for my opinion, I decide to see what I could learn. I searched for clinical trials of the preparation for URIs and found a small randomized trial showing no apparent benefit of a beta-glucan product on URI in athletes.

Not surprisingly, this RCT had not been highlighted in the video. However, a number of other trials were mentioned and were linked on the product's website. Two of the trials listed were apparently only presented in meetings, one was reported as published and I was unable to find the full text (perhaps others will have more luck), but one trial was published in a form I could access.

On the product website, the accessible study was described as showing:

While there were no significant differences in the incidence of symptomatic respiratory infections among the study groups, the duration and severity of symptoms were alleviated in subjects receiving Wellmune WGP.

In the study results, the Wellmune WGP group reported:

• No missed work or school due to colds, compared with 1.38 days of work/school missed for the placebo group. (p = 0.026)

• No incidence of fever, compared with 3.50 incidence in the placebo group. (p = 0.042)

• An increase in quality of life, including physical energy and emotional well-being, as measured by a clinically validated health survey questionnaire (SF-36v-2). (p = 0.042)

• No adverse events were detected and no safety concerns were present.

This actually tracks fairly closely with what the published study states in its abstract. However, if you actually read the paper it becomes quickly apparent that an enormous number of comparisons were made, nearly all of which showed no benefit to beta-glucan (the missed work or school due to colds was one isolated positive finding). The reported effect on fever does not match the data in the text of the article and, even there, no effect on fever was seen in the intention to treat analysis or in the per protocol analysis, though the latter got close to achieving that all important p<0.05 though it didn't make it.

I communicated back to my friend my doubts that anything very interesting had been shown about the product and inquired about possible financial incentives influencing the interpretations of those who were reporting the results to her.

Why is any of this even worth thinking about? After all, the claims made in the video had raised my doubts as soon as I'd heard them, and there are many such claims for many products that are at least as dubious.

The reality, though, is that this beta-glucan product will be marketed and may gain traction. The friend who asked me about it actually currently sells an acai berry product in which she is a true believer. Acai berry products are being marketed all over the Web and doctors are now fielding questions from patients about their value. In my role as a medical editor, I was asked recently by a reader (a skeptical physician) to have our resource include an article discussing the evidence for and against clinical benefits from acai berry supplements.

If we wrote such an article about acai berry, what should it contain? There is no high quality evidence about acai berry and clinical outcomes. Should an article take an agnostic view about its benefits? Should it point out that lots of antioxidant supplements have failed to improve health or have even caused harm? Should it be written at all, or is there some threshold of believability or widespread use that must first be overcome.

The scientific medicine world generally claims it expects some reasonable evidence before administering a new preparation or trying a new procedure (though is not always as good in practice as in theory), and the pharmaceutical industry, for all its problems, can't release a drug without impressive clinical trials.

But when random claims of snake oil salesmen selling some new product spring up, eventually, if the product really does become widespread in the marketplace, the expectation of many in the media and the public seems to be that the burden has shifted to scientific medicine to disprove the benefit or else not take a position about whether a product is good or bad.

At the extreme, you have the most ludicrous ideas of alternative medicine like homeopathy. But even more plausible-sounding ideas when untested do not deserve much credence. That someone who is selling a product can weave a good story about why it is beneficial should not require the resources of the rest of us to test those claims.

We'd all be better off starting out as skeptics, particularly when those promoting an untested product have a financial stake in its success. Snake oil salesmen have been a bane of appropriate medical care for centuries, and there's no reason to think this will change. Those who care about science and evidence, though, must remember that we need not approach entirely untested claims from a position of neutrality. Most things you could imagine doing to improve health do not work, and many are harmful. Though absence of evidence is not evidence of absence, we need not spend time and thought disproving the claims of every snake oil salesman who comes along. It already takes enough resources testing just those claims that have some shred of plausibility as well as the products that have become so widespread that letting them stay untested endangers health.

Attacking Randomized Trials

One consequence of my writing this blog is that many of the residents I work with have started reading it -- a risk I clearly hadn't adequately considered. I received a request from one resident to write about some problem that I think residents commonly have with applying evidence to practice. That was quickly followed by my sitting in as discussant at a resident-led journal club, which led to my thinking about the many unfair and fair ways that people attack published randomized trials. I'll have to give additional thought in the future to the more general question of common ways residents misapply evidence in practice.

By way of background, it is standard practice in academic medical centers to present a trial at journal club and then rip it to shreds. After repeating this exercise over and over, it's easy for all involved to conclude that every trial is flawed and nothing can be learned from any of them. It can become almost a game to figure out why not to believe a given trial, rather than discover which trial results should and should not be applied to practice. (I should point out, here, that the resident who led the journal club I mentioned above made fair criticisms of the presented trial and concluded with how the trial would affect her clinical thinking and decisions.)

I think I fully recognized how far out there people could get in criticizing virtually any trial back in 1993 in the midst of the H. pylori debates. At that point there was still a lot of doubt in some quarters as to whether H. pylori caused peptic ulcers. There had been trials of eradicating H. pylori showing reduced ulcer recurrence, but the eradication regimens included bismuth which was believed to have some direct anti-ulcer activity, leaving a hole in the causal argument.

To deal with this a trial was performed looking at recurrence rates after an eradication regimen using amoxicillin plus metronidazole. Ulcer recurrence rates at one year were 8% in the treated arm and 86% in the placebo arm. To me, this seemed to give a final answer to the question of H. pylori causing ulcers, but I listened to a room full of internal medicine residents from one of the top programs in the country come up with outlandish criticisms of the results. Several residents argued, apparently convincingly to others in the group, that amoxicillin and/or metronidazole might have direct anti-ulcer effects and so the reduced recurrence rates might have had nothing to do with H. pylori. There were several problems with this argument, including the lack of any a priori belief that these antibiotics had such an effect, that an anti-ulcer effect of a 12-day regimen would be unlikely to have such an enormous impact a year later, and that when the trial was analyzed by H. pylori eradication rather than treatment arm, the effect size was even larger.

Another common and generally unfair way to attack a randomized trial is to look for unmeasured confounders or confounders that are somewhat unbalanced in the table 1 data.

Confounders are things associated with both the exposure and the outcome of interest in a study, that through these associations confound the assessment of a relationship between exposure and outcome. For instance, we might find in a study that coffee drinking is associated with lung cancer. Coffee drinkers may smoke more than other people, and so, if we fail to control for smoking, we might falsely conclude that coffee drinking causes lung cancer.

In a properly performed randomized trial of a reasonable size, unmeasured confounders will sort nearly equally among the treatment arms. Even if smoking status is unmeasured, if there are 800 patients in each of two arms, we can be pretty sure that the number of smokers in the two arms will  be close enough not to disturb the trial results. Table 1 in a published RCT typically gives the baseline data on the groups, and if lots of potential confounders are measured and reported on at the 0.05 significance level, we shouldn't be surprised when 1 in 20 appears unbalanced at such a level. That said, if lots of measures appear unbalanced, with one group clearly sicker than the other, it's important to go back and see that allocation in the trial was properly concealed. It's rare, though, to see problems of unbalanced allocation in a major clinical trial, particularly a placebo-controlled trial of a medication.

As is clear from some of my earlier posts on this blog, I obviously believe that some attacks on randomized trials are fair and appropriate. More fruitful attacks often deal with issues of what the GRADE group calls "indirectness". Results of a randomized trial might be indirect to the question a clinician is interested in because the trial did not study the exact intervention of interest (for example, too high or low a dose of a comparator drug was used, as in a trial comparing esomeprazole with omeprazole), perform the study in the exact population of interest (for example, the study was performed in young healthy patients, and the target population is the sick elderly), or measure the exact outcome of interest (for example, a surrogate outcome like carotid intima medial thickness was measured, rather than a clinical outcome).

I recall an argument years ago on Usenet about whether aspartame really causes headaches in the many people who report headaches when they consume aspartame-flavored foods and drinks. At the time, I pointed out the randomized trial in the NEJM showing no difference in rates of headaches in subjects who believed they were sensitive to aspartame, whether given aspartame or placebo. Another poster on Usenet objected, saying that he did not think that aspartame in a pill caused headaches, but rather that the combination of sweet tasting food and lack of sugar led to headaches, perhaps by inducing reactive hypoglycemia. Testing aspartame without letting the subjects taste the preparation was completely unconvincing to him. This was essentially an argument about indirectness of intervention -- the intervention of interest was aspartame as commonly consumed in foods and drinks, not aspartame in pill form.

To me, on reflection, this seemed like a fair objection, though I didn't think it really held up to scrutiny when looking at the actual trial results. In the trial, which had a crossover design, the 40 subjects developed headaches 35% of the time when they were given aspartame, and 45% of the time when given placebo. To argue that encapsulated aspartame doesn't cause headaches in susceptible individuals is to miss that the trial found that placebo induced headaches in these individuals almost half the time.

I'll come back to the issue of indirectness of outcomes in the near future when I'll write more about surrogate outcomes, since this underlies some of the arguments that came up in the comments responding to my vitamin D posting, and I'll likely write about indirectness with regard to population down the road.

The general point, though, is the importance of being fair, open-minded, and keeping a big-picture view when evaluating a new randomized trial, and then interpreting the results in the context of existing evidence. Lee Goldman talks about dealing with randomized trials when one is performing the role of playwright rather than theater critic. But even if you're just the theater critic, if you hate all plays you see, your reviews won't prove terribly edifying to your readers.

Vitamins and Faith

I'd mentioned in an earlier post that a relative was prescribed fairly high dose vitamin D by her primary care physician despite a blood test showing that her vitamin D level was in the normal range. Her PCP may not be much of an outlier. Among the residents I work with, I've been encountering a lot of desire to prescribe vitamin D to pretty much everyone.

Why the willingness to administer vitamins? Would these same doctors feel that a drug with similar levels of evidence for safety and efficacy should be administered to tens of millions of generally healthy people?

Somehow patients and providers alike seem willing to treat vitamins as articles of faith -- as substances that must be good for humans based on their very name and nature. Yet the actual track record of vitamins, and particularly the fat-soluble vitamins, should not lead us to be so sanguine.

Based on their antioxidant properties, vitamin A/beta carotene and vitamin E were believed to be good for health before any reasonable studies had been performed. Doctors recommended beta carotene and vitamin E at moderate or high doses, and health-conscious patients took even higher doses. Then the results of randomized trials started to come in:

 Two large randomized trials of vitamin A/beta carotene found increased rates of lung cancer in smokers, pretty much putting an end to the notion of using beta carotene to prevent cancer. Then, a meta-analysis of trials of vitamin E found an apparent increase in mortality in patients treated with higher doses of vitamin E, and suddenly vitamin E no longer appeared so benign.

With this as a backdrop, we might expect physicians, at least, to show some skepticism before recommending other vitamins, and particularly other fat-soluble vitamins.

Yet, in their defense, vitamin D has been studied in randomized trials (mainly for prevention of fractures/osteoporosis), and these trials have reported other outcomes including cancer rates and mortality.

Although much larger trials have found no effects of vitamin D on cancer rates, one trial that I've heard people cite did indeed find a reduction in cancer rates in patients supplemented with calcium and vitamin D. The RR was 0.23 -- in other words, the supplementation appeared to reduce cancers by 77%. Since virtually nothing in medicine has an effect that large, and such an effect would be hard to have missed in other trials of calcium and vitamin D, this result seems implausible at best.

With regard to mortality, a meta-analysis of randomized trials found that vitamin D supplementation appeared to decrease mortality by 7%. A concern, though, is that the result was just barely statistically significant, and there could be reporting bias with regard to mortality: trials that found larger differences in mortality (or any other safety endpoint -- these were, after all, not trials designed to look at mortality) would be more likely to report on mortality than trials that found nothing interesting when they looked at their safety data. Given the risk that the results could be due to chance, even a small amount of reporting bias would be worrisome.

So, I've now objected to one trial of vitamin D for having too large an effect to be believable, and a meta-analysis for having too small an effect to convincingly rule out chance plus bias. Am I being fair and true to the evidence?

If we were talking about a therapy for a morbid illness where we were compelled to treat, I'd likely accept the evidence from the meta-analysis as being sufficiently hopeful to make vitamin D worth a try. After all if treatment is required, even if vitamin D eventually turned out to be harmful, we would know that we had needed to do something for the ill patient and so we made our best guess.

But that is not the case here. Instead, we are talking about administering a therapy to enormous numbers of healthy people with the hope of preventing disease, and where causing even small increases in harmful outcomes would have no justification. And remember the history of vitamin E and beta carotene. If that is not sufficient to make you concerned, read what happened to cancer rates and mortality in two trials of folate/B12 supplementation in Norway, published just last week in JAMA.

In such a situation we should wait for (and demand) more convincing evidence before we act. It may turn out that vitamin D really does prevent cancer, decrease mortality, and protect people from everything from influenza to the common cold. But I would rather be late jumping on this bandwagon than jump early and find that I hadn't learned a thing from past mistakes.

USPSTF, Mammography, and Grading Recommendations

It's hard to have a blog about evidence and medicine without commenting on the changes to the USPSTF's breast cancer screening guidelines.

At a first pass, it's worth pointing out that a lot of academic primary care physicians have been wondering what the USPSTF was thinking when they published their 2002 guidelines recommending mammography for women in their 40s. Yes there may be some small benefits (and lots of harms), but this is just the sort of thing that the USPSTF can usually be relied on to not recommend -- look for instance at what they recommend for prostate cancer screening or lipid screening for women. As a result, there were lots of insinuations that the USPSTF had caved to political pressure in making the mammography recommendation. Perhaps others know more about that one way or the other.

Now they've backed away from that recommendation, going from a favorable to a negative recommendation for mammograms. The overall issues aren't actually that complex: if you imagine 1000 women in their 40's getting mammograms yearly for 10 years, the best estimates are that you will cause more than half of them to need repeat mammograms for concerning findings, and will cause nearly 200 to get breast biopsies, while only preventing 2 deaths from breast cancer.

What is remarkable is the firestorm that has been set off. When I present these data to women patients in their 40s (in a way that I think is fair to both sides of the argument), I've found that approximately half want mammography and half want to wait until they turn 50. Individual patient values and preferences heavily influence the decision. The USPSTF takes this into account by recommending against routine mammograms in the 40s, but states that a decision to screen should take consider an individual woman's values. All in all, not that unreasonable a position as far as I'm concerned.

I think, though, that the USPSTF's way of grading recommendations is awkward in general and may even have contributed to the confusion over their 2002 and 2009 recommendations. The evidence base didn't substantially change over that time frame, but the recommendations did, and that opens them to attack.

I think the USPSTF would be better off using a different grading system, such as GRADE, which they explicitly chose not to do recently.

(Here I need to make one of the only conflict of interest disclosures I should ever need to mention on this blog: one part of my work is as a medical editor, and in our publication we use GRADE.)

That out of the way, the USPSTF grading scheme is unusual in the way that it combines strength of recommendation and quality of evidence into a single value (a letter grade). To me, this overloading causes problems. For many of the recommendations that USPSTF gives a grade of I to (insufficient evidence), GRADE and a number of other schemes would note the low or moderate quality of the evidence and still allow a guideline committee or other recommending body to make a suggestion about what a clinician should do.

When there is enough evidence to make a suggestion (a "weak recommendation" in GRADE parlance), GRADE explicitly deals with the need for considering patient values and preferences. If, instead of being used to this single letter grade from USPSTF, we were used to seeing evidence and recommendation divided, I'm not sure the recommendation change would have caused quite so much trouble.

The USPSTF recommendation, cast into GRADE style, would likely have read:

For most women ages 40 to 50, we suggest not performing yearly mammography (Grade 2B). Women who are willing to accept a high risk of false positive mammograms and a high risk of unnecessary biopsies in return for a small mortality benefit may reasonably prefer annual mammography starting at age 40.

Would a recommendation written this way have caused the same firestorm? I'd be interested in what others think about this. Would it have caused a firestorm if the 2002 recommendation had said the opposite?:

For most women ages 40 to 50, we suggest performing yearly mammography (Grade 2B). Women who are unwilling to accept a high risk of false positive mammograms and a high risk of unnecessary biopsies in return for a small mortality benefit may reasonably prefer annual mammography starting at age 50.

You can see how easy it is to split the unchanged evidence base from the change in what you are suggesting for the majority of women (based on a guess about their values and preferences) when using GRADE. I think such an approach is more transparent than that used by the USPSTF, and keeps the focus where it belongs in a situation like this -- on the benefits, risks, and burdens that influence the decisions for individual patients and for society.

I'll write more about the methods of GRADE, including its strengths and weaknesses, down the road. I do want to say that while I don't like the grading scheme used by USPSTF, I think they mostly do a wonderful job of staying true to the evidence as they present their guidelines. We should all hesitate to recommend screening tests in the absence of solid evidence showing benefit, and the USPSTF can almost always be counted on in this regard.

ARBITER 6 and Combining Therapies with Statins

The results from ARBITER 6 were released this week.

The interpretation is less clear.

I've been in a minority who has doubted that most drugs that lower LDL are actually beneficial in primary prevention. Statins clearly show benefit, but really no other agents have shown a reduction in all cause mortality or, in primary prevention, in cardiovascular mortality. As such, I'd been raising concerns about ezetimibe even before ENHANCE was published, and have been unconvinced that anyone should be treated with combination therapy to achieve LDL goals, since there is little to suggest added benefit when other lipid-lowering drugs are combined with a statin.

Recently, the Oxford Niaspan Study found that CIMT was improved when niacin was combined with a statin, but CIMT is a surrogate marker and it's not so clear that changes in CIMT will predict clinical benefits. However, of all the non-statin therapies, niacin has been the one that looks most promising for reducing cardiovascular mortality, at least in secondary prevention.

Now, soon after that study, we have ARBITER 6, showing that niacin plus a statin seemingly reduced cardiovascular events compared with ezetimibe plus a statin. There were 2 events in the niacin arm (1%) and 9 in the ezetimibe arm (5%) and this was statistically significant. So, what can we conclude from this?

One possibility is that niacin actually does add some clinical benefit to a statin in secondary prevention, while ezetimibe does not.

Another possibility is that ezetimibe is actively harmful, while niacin is neutral. There were slightly more events in ENHANCE in the ezetimibe arm, but the numbers were tiny.

A third possibility is that this was due to chance. The actual results are quite "fragile" in the sense that only a few events moving from one arm to the other would make the difference disappear. While, in theory, a statistical test captures this, there's been a sense in the EBM world that very small event numbers create a fragility of results not adequately captured by a p value. It seems extremely unlikely that niacin really reduces events by 80%, since this should have been obvious in prior trials of niacin. As such, either niacin combined with a statin is much superior to niacin alone, or these results are in part due to the play of chance. (It's not likely that ezetimibe raises events by this degree when used with a statin, since we would likely have noticed this in other trials.)

ARBITER 6 was stopped early for benefit, based on CIMT results. This seems like an odd decision to say the least, since change in CIMT is not a clinical outcome requiring an ethical decision to stop a trial. The early stopping was unfortunate since with more events we might be more certain about whether the reduction in events was real.

I'll be interested to see how others interpret ARBITER 6. So far I've communicated with one lipid expert (who has not shared my views about non-statin therapy) and he doesn't believe that ezetimibe is clearly to be avoided, while another expert (who does share my views about non-statin therapy) doesn't feel that ARBITER 6 means that niacin should be used. Apparently neither one found ARBITER 6 substantially reshaped their positions on this.

My best guess is that niacin reduces events a bit, though not as much as ARBITER 6 suggests. If so, for the highest risk patients, such as those who have CV events while on a statin, adding niacin might make sense. Given the side effects with niacin (36% of patients in ARBITER 6 had  flushing, and many more in the niacin arm than the ezetimibe arm discontinued therapy), I'd hesitate to use niacin in lower risk patients even for secondary prevention, out of concern that they might discontinue both niacin and the statin.

I'd be quite hesitant to treat anyone with ezetimibe at this point. While I was a skeptic that ezetimibe did anything beneficial even before ARBITER 6, I'm now substantially more concerned that it could be causing real harm.

Opinion, Evidence, and Consensus Statements

In the future, I'll write more about this subject, since the distinction between expert opinion and clinical experience seems to cause unending confusion to people who write and talk about EBM.

For now, though, I want to write about the second recent event that caused me to decide to blog on evidence and medicine. I caught a snippet on NPR's Science Friday where they were talking about concussions in athletes and heard the following: "And take a very careful step-wise return not just to the playing of the sport, but also to even, you know, text messaging and doing things intellectually that can be harmful to their brains while they're still recovering from a concussion." www.npr.org/templates/story/story.php?storyId=120174333

I was startled to hear that an expert thought that someone should not be texting after a concussion (it turns out this is referred to as "cognitive rest"), and I had a hard time believing there was any evidence for it. And though I have no expertise in concussion management, the idea sounded somewhat contrived or even goofy to me.

It turns out that this mainly comes from consensus guidelines for the management of concussion (PMID 19433429) where, if they have any evidence to back this up, I couldn't find it. I certainly didn't find any published studies showing that cognitive rest improves outcomes after a concussion. If anyone reading this knows of such evidence, I'd be interested to see it.

To make recommendations, experts always need to interpret what evidence there is (expert opinion), and in the absence of high or moderate quality evidence about cognitive rest, the consensus panel presumably had to make their best guess for management. This particular recommendation seems a bit odd in that it has potentially high burdens on the injured athlete (don't read; don't text) in the absence of good evidence, and rest is not always such a good idea after a physical injury -- bed rest for low back pain and immobilization for an injured shoulder are both clearly a bad idea. But ultimately doctors need to either recommend cognitive rest or not recommend it and if some people think it's a good idea an expert recommendation is required.

However, a certain degree of humility would seem to be required when transmitting a recommendation grounded in such low quality evidence. Perhaps this is not such an instance, and I am just missing the better evidence, but it would certainly be common to hear expert panels and individual doctors make forceful pronouncements based on little or no foundation. If there really is nothing much to support this, perhaps the experts describing the recommendation (both in the guideline and on the radio) might have said, "Well, we think cognitive rest is a good idea, but we don't really know. In fact, for all we know, it might do more harm than good."

So, while I'll write more about expert opinion and clinical experience down the road, this episode of seeing yet another consensus guideline treated as gospel was an added push to create this blog.

First post and origins

Although an early participant in the world of Usenet newsgroups, I'm a latecomer to the world of blogging. I don't read or participate in blogs often, and as a newbie will undoubtedly make mistakes of etiquette and function much as I did in the very early days of Usenet before I got a feel for the music of the newsgroups. I hope anyone reading this will take this into account and cut me some slack!

I'd been considering for a while now blogging about evidence and medicine, something I spend a fair amount of my time thinking about in any case, but a couple of events pushed me to take the plunge.

The first was when a relative, a woman in her mid-40's, called to tell me that she'd seen her new primary care doctor. The doctor had run tests on her including checking a lipid profile. The lipid profile showed a total cholesterol of 160, an HDL of 44, and an LDL of 105. Based on these results, the doctor informed her that she should be very concerned about her "low" HDL number and told her to start taking fish oil.

This was an interesting interpretation of the results and an unusual suggestion for management, and it gives some sense of my concerns about how evidence is used, ignored, or misused in medicine. This relative had a low normal blood pressure, no smoking history, did not have diabetes, and had no family history of coronary heart disease. As such, there is even  some controversy as to whether she should have had a lipid profile checked at all.

Assuming a lipid profile was worth doing, she should certainly have been told that this was an excellent profile putting her at low risk for a cardiovascular event, and that there was little modern medicine could do to substantially affect her risk. Her 10-year risk of an event (using the 2008 calculator that overestimates vascular risk by including things that probably shouldn't be included -- a blog for another day) works out to about 2.2%. If we pretend that somehow fish oil could increase her HDL to 80 and that she would get all the same benefit as if her HDL were naturally 80, we could drop her calculated risk to about 1.4%. So, if she took medication daily for 10 years, she would reduce her risk of an event (under these outlandishly favorable assumptions) by 0.8%. Not something this relative was interested in doing.

But, do we even know that fish oil raises HDL? The main effect of fish oil on lipids is to lower triglycerides. Most studies suggest that fish oil only raises HDL levels by about 3%, so we might expect an increase from an HDL of 44 to an HDL of 45. It's hard to see how the doctor felt this would fix the woman's "low" HDL.

But, beyond that, we have no solid evidence that raising HDL with  medication improves outcomes. Changing HDL with medication is a surrogate outcome for that medication  -- that is, it is of no direct interest to the patient unless it also predicts better cardiovascular outcomes, and HDL appears to be a poor surrogate.  Some medications that raise HDL appear to cause harm and worse vascular outcomes.

As for whether fish oil supplements are beneficial through some non-HDL mechanism, I think we can safely say that the jury remains out as to whether a woman in her 40s at low cardiac risk could expect any important benefit from fish oil supplementation.

Interestingly, my relative's doctor also checked her vitamin D level and told her that a level of 34 (in the normal range) was concerningly low and that she should start taking 1000 IU of vitamin D per day (she was already taking 400 IU). Vitamin D supplementation is controversial, with lots of people claiming big benefits, but we can safely say for now (again, perhaps a longer blog for another day) that there is no solid evidence that having a woman with a normal vitamin D level start taking additional vitamin D is likely to be of any benefit to her.

My relative asked what I thought these events said about her doctor, but I'm not really sure. I see decisions like this made all the time, where evidence is misunderstood, misapplied, or ignored. I have the sense that this particular doctor may believe she is getting out ahead of the evidence -- suggesting therapies that she believes will be proven down the road to be of benefit. She could, of course, turn out to be right, but we should be very hesitant to give preventive medications without a decent evidentiary basis. Other drugs that raise HDL have increased mortality. Supplementation with other vitamins that were thought beneficial has turned out to cause cancer or increase mortality.

I said there were a couple of events that prompted me to start this blog, but this post seems long enough. I'll talk about other events in future postings.