Evidence In Medicine

Evidence, Medicine, and EBM

Lay of the Land

There have been some recent blog posts around yet again debating the concept of evidence-based medicine (EBM).

This blog is titled "Evidence in Medicine" not because "Evidence Based Medicine" was already taken as a URL and blog title, but rather because I didn't want some of the baggage and implications of the EBM label.

EBM is frequently defined as "the conscientious, explicit and judicious use of current best evidence in making decisions about the care or individual patients", and you would think it would be hard to object to that. And yet....

One blogger, Laika Spoetnik (Jacqueline), recently wrote a piece complaining about the attacks on EBM, and mentioned the humorous parachute example that I had referred to in an earlier post. She wrote about the parachute piece::

I found the article only mildly amusing. It is so unrealistic, that it becomes absurd. Not that I don’t enjoy absurdities at times, but  absurdities should not assume a live of their own.  In this way it doesn’t evoke a true discussion, but only worsens the prejudice some people already have.

(As I'd suggested in that same earlier post, EBM folks are not necessarily known for their great senses of humor.)

In response to Jacqueline's post, Kimball Atwood wrote a thoughtful piece explaining his problems with EBM, and linked to an earlier piece he'd written on the same subject. I would encourage people to read Dr. Atwood's posts.

Personal Systematic Reviews

So, how could any reasonable person possibly object to the use of best evidence in making medical decisions?

The answer is that they could not, but that EBM is not necessarily synonymous with using best evidence, despite the definition above. If I'd been writing a few years ago, I would have agreed with a number of the posters attacking EBM, since many of the people I have seen promulgate EBM do so in a way that misses some basic truths.

EBM,at least as it was promoted in academic internal medicine circles in the 1990s, was very focused on a specific hierarchy of evidence (RCTs > cohort studies > case control studies > case series) and on the belief that we should train primary care internists that the right way to answer a clinical question they encountered was to perform a literature search along with their own, on the fly, systematic review. I'll come back to the hierarchy of evidence in a bit.

The idea of teaching people to perform personal systematic reviews was being pushed at a time when I was co-directing the resident journal club at my hospital. This was couched as "teaching EBM", and the various residency course directors around the country seemed to be spending a lot of time telling each other what an important activity this would be for academic general internists to engage in. I and the other director of journal club pointed out repeatedly that real systematic reviews take months, and that it was a bizarre notion to think that the right way to answer a question that arose in a clinic would be to try to do this on the cheap. A quick review of randomized trials is fraught with the likelihood of missing important pieces of evidence. When residents try to do  this, they often end up focusing on the most recently published trial rather than the entirety of the available knowledge.

We argued, instead, that for most such questions residents would be far more likely to find the right answer in a textbook than by trying to interpret a body of literature in between seeing patients. (These days I would make the same argument but replace "textbooks" with "electronic resources", however that raises a conflict of interest with one of my day jobs, so I won't push that particular argument any further.)

As a result of our resistance, this curriculum never quite got implemented for our journal club, but my sense was that put our hospital in the minority. I think many programs in the late 90s and early 2000s tried to convince residents that it was sensible, practical, and necessary to perform their own systematic reviews whenever confronted with a clinical question.

Hierarchies of Evidence

Coming back to the hierarchy of evidence issue, it's clear that a system that focuses too much on the hierarchy mentioned above will reach absurd conclusions. For instance, one might decide that we have higher quality evidence that an HIV vaccine is protective than that people who wish to avoid lung cancer should not smoke. We have essentially no evidence from RCTs in humans supporting that latter hypothesis and yet we should clearly believe it to be true as surely as we believe anything in medicine, while it remains quite uncertain that an HIV vaccine is of any benefit despite the published RCT with a "statistically significant" p value.

And when effect sizes are large enough, even a couple of clinical events may provide higher quality evidence than some RCTs that try to examine questions of minor benefits of therapies. How many people have to survive meningococcal meningitis after treatment with penicillin before you can be quite sure that penicillin is lifesaving?

Dr. Atwood goes to the specific issue of biologic plausibility in the hierarchy of evidence as it applies to homeopathy. I actually agree with those who are suspicious of biologic plausibility arguments, since smart people can make almost anything sound biologically plausible. Dr. Atwood focuses more on biologic implausibility and I tend to find these arguments more compelling on average. In the case of homeopathy, he argues, it would be pointless and a waste of resources to perform RCTs examining homeopathic remedies. This is a fair point, since if homeopathy is true than our understanding of chemistry and physics is incorrect; thus it would require extreme levels of proof before we should begin to entertain the notion that a homeopathic preparation could be of benefit.

(I plan to write a future post about the burdens placed on the scientific medicine community by people who promote unproven remedies; valuable resources must then often be spent to prove the lack of benefit of the now widely popular yet worthless preparations and interventions.)

EBM and GRADE

Given what I've written you might conclude that I lean toward the anti-EBM camp. As mentioned, that's likely where I would have placed myself a few years ago. In the interim, though, I've started working with the GRADE group and in particular with Gordon Guyatt.

Through these interactions, I've come to realize that when a really smart and thoughtful clinician like Dr. Guyatt discusses EBM, the above problems and concerns mostly go away. For instance, GRADE recognizes the importance of randomized trials, but also that high quality evidence can be found in other places. The group works to formalize what might lead rational clinicians to conclude they have high quality evidence, and while I don't agree with GRADE 100% of the time, my disagreements mostly involve some edge cases that really don't arise very often.

And since Dr. Guyatt is generally credited with inventing the term EBM, it's hard to dismiss his interpretation as being unrepresentative of true EBM. Instead, I've come to believe that most of the people who have interpreted and disseminated "EBM" lacked either the clinical understanding or the epidemiologic knowledge to communicate a fair representation of EBM to the clinical world.

I'd encourage those who find themselves troubled by aspects of EBM to spend some time looking at what the GRADE Working Group is doing. Even better, if you have the chance to interact with Gordon Guyatt at one of his courses or lectures, jump at the opportunity. You likely will feel differently about EBM for having done so.

Revere's Thought Experiment and Observational Studies

Over at Effect Measure, Revere has posed a thought experiment (or, perhaps, a disguised real experiment) that provides a valuable jumping off point for discussing the worth of observational data.

In brief, Revere describes an uncontrolled study for a new application of an antiepileptic drug as a treatment for refractory hypertension. A study is conducted by administering the drug to 29 patients and checking their blood pressures at baseline and after a month of therapy.

I glanced at the comments on the blog, and interestingly a lot of people are spending time trying to apply a name to what exactly this study is (i.e. "case series", "observational", and the like). I often don't find naming things all that helpful unless everyone has a clear understanding of the naming convention, but this would most commonly either be called an observational study (if we view that the therapy was going to be administered independent of the interest in collecting data about what happened), or an uncontrolled clinical trial (if we view that the primary goal was to find out if the AED works for refractory hypertension).

In the thought experiment, the blood pressure readings a month later are substantially lower.

Revere poses the following question:

"...is this a sufficiently reliable study that a reasonable practitioner, committed to the use of scientific evidence in her practice, would consider?"

I'm going to move on quickly from that question, because I think the correct answer isn't that helpful: the study contains data about a potential new therapy, and a reasonable practitioner should always consider such evidence if she or he has the time and ability to do so.

To understand, instead, whether we should believe that the results in this study can be ascribed to the drug working, we might look first at how we would feel about the result if it had been obtained from a randomized trial performed in the same manner described as the uncontrolled trial, except now we have 29 patients randomly assigned to receive the AED, and 29 to not receive it. Notice that this RCT is still unblinded and, as such, has no placebo arm.

We'd likely note a few problems with this RCT. First, it has a relatively small number of patients, though if the drug is highly effective this might not be a problem. We'll come back to this point a bit later.

Second, the outcome is fairly indirect (in GRADE terminology) to an outcome of clinical importance. Revere points out that BP reduction is not a clinical outcome: it's a commonly used surrogate, but what we really care about is a reduction in cardiovascular events and this is not being measured. BP lowering is such a standard surrogate that it's easy to lose sight that it is a surrogate at all, but not everything that lowers BP is good for you (consider endotoxin, for instance). However the actual measurement has an additional level of indirectness: we are measuring BP reduction at one month as a surrogate for long-term blood pressure reduction. Long-term BP reduction is the marker that we actually typically use as a surrogate for a reduction in CV events.

Third, as noted above, the study is unblinded. Does this matter? We're told that the BP both at baseline and at one month is being taken with an automated cuff and all the clinical assistant has to do is write down the measurement. This sounds like a simple solution, but it is fraught with problems. The patients are unblinded, and so the ones taking the medication may believe in its efficacy, relax more, and so have lower BP readings in the office. The clinical assistants believe in the drug and want the study to succeed, and so may take several readings in patients on the AED ("why don't you sit here and relax, and I'll recheck your BP in a few minutes -- this drug looks like it's working in almost everyone, so I'm sure your BP will come down if we just wait"), but record the first BP reading in the control group patients. And, of course, those analyzing the results know who is and is not receiving the AED, and this might affect all sorts of things (not that we like to admit this).

So this small, unblinded, short-term study of a surrogate outcome might not be all that convincing, even as an RCT.

When we push it a step further and make it uncontrolled, we add an additional major problem of regression to the mean (as well as secular trends in BP over time). In the RCT, we had a control group, and so we at least knew what happened to the BP readings in patients not treated with the AED. in the uncontrolled study we have no such knowledge. We're not told exactly what the entry criteria are, but presumably having a well-controlled BP would prevent a patient from getting the AED as an additional anti-hypertensive. Thus, patients who are oscillating around a mean BP value will likely be excluded if they are at a BP minimum and enrolled if they are at a BP maximum. One month later, just due to the usual biologic variation plus this entry criterion, the BP can be expected to be lower on average.

Adding this problem to the ones we noted in the RCT version, we are unlikely to be terribly sanguine about believing the results of this small trial.

Are there any circumstances, though, where despite all this we might trust the results anyway? One of the real insights of the GRADE group, to my mind, is a focus on magnitude of effect in thinking about observational data, rather than focusing on minor differences in study design (such as cohort versus case control). With large enough effects, we might accept some pretty minimalist study designs.

For instance, rabies is historically a 100% fatal illness once clinical symptoms appear in someone who has never been vaccinated. In 2005, a 15-year-old girl survived rabies after treatment with a novel experimental regimen. You could imagine that anything can happen once, and it may have been coincidence that this girl survived and received this novel regimen. However, if one more person with rabies were to receive the regimen and survive, it would seem spectacularly unlikely that the explanation would be anything but that the regimen works. A rational clinician would treat any new patient with rabies with that regimen from that moment until a superior regimen was found. (As far as I know, no other patient has survived rabies on this regimen.)

Similarly, in response to the humorous attack on EBM, asking what the RCT data are supporting parachutes when skyjumping, the GRADE group would respond that the magnitude of effect of parachutes is sufficient to constitute high quality evidence for their use (okay, clinical epidemiologists can be somewhat humor-challenged). That is, we have all sorts of historical evidence about what happens when people fall from great heights (and that we might consider only slightly indirect to falling from an airplane), as well as lots of observational data about what happens when people fall from airplanes wearing parachutes. Not everyone who falls from a great height without a parachute dies, and not everyone wearing a parachute lives, but the effect size is so large that we have high quality evidence for parachutes in the absence of a clinical trial.

To give one example that might feel more real, I was doing a lot of AIDS care in the 1990s. In 1995 an abstract was published from one of the ID meetings about the effects of ritonavir in about 50 people with late AIDS. (I've tried in vain in the recent past to find this abstract -- if anyone can point me to it, I would be grateful.) The results were like nothing we had seen before -- patients' CD4 counts rose dramatically, opportunistic infections improved, and many patients who would have been expected to die improved instead. We did not know what would happen long-term, but it was obvious, without any RCT, that ritonavir was effective therapy for AIDS, at least in the short term. By 1996, we were treating people with triple therapy "cocktails" for HIV, again without any RCTs with clinical endpoints, and watching people who had been dying walk out of hospitals and hospice care as their OIs resolved. The magnitude of effect was such that we had high quality evidence for these cocktails based on observational data alone. (Not that this actually prevented researchers from proceeding with an RCT of triple therapy, but that's a post for another day.)

In the refractory hypertension thought experiment, the problem is that we do not have enough evidence about how unusual the BP lowering seen was to conclude that this was a very large (and thus very unexpected) result. The result could have been due to measurement biases and regression to the mean, and so have had nothing to do with the actual drug being studied.

Revere started his/her post talking about Cochrane and the focus on RCTs. I agree both with concerns about Cochrane reviews (which can be of quite variable quality), and about the general issue of ignoring high quality observational data (most often high quality because of very large magnitudes of effect) to focus exclusively on RCTs. However, I don't find the evidence from the study in Revere's thought experiment to be sufficiently high quality that I would be generally willing to administer the AED for refractory hypertension without further study. Having read such a study, I might take it into consideration sufficiently to administer the AED as a last resort -- a situation in which I might be willing to grasp at any straw of evidence.

Attacking Randomized Trials

One consequence of my writing this blog is that many of the residents I work with have started reading it -- a risk I clearly hadn't adequately considered. I received a request from one resident to write about some problem that I think residents commonly have with applying evidence to practice. That was quickly followed by my sitting in as discussant at a resident-led journal club, which led to my thinking about the many unfair and fair ways that people attack published randomized trials. I'll have to give additional thought in the future to the more general question of common ways residents misapply evidence in practice.

By way of background, it is standard practice in academic medical centers to present a trial at journal club and then rip it to shreds. After repeating this exercise over and over, it's easy for all involved to conclude that every trial is flawed and nothing can be learned from any of them. It can become almost a game to figure out why not to believe a given trial, rather than discover which trial results should and should not be applied to practice. (I should point out, here, that the resident who led the journal club I mentioned above made fair criticisms of the presented trial and concluded with how the trial would affect her clinical thinking and decisions.)

I think I fully recognized how far out there people could get in criticizing virtually any trial back in 1993 in the midst of the H. pylori debates. At that point there was still a lot of doubt in some quarters as to whether H. pylori caused peptic ulcers. There had been trials of eradicating H. pylori showing reduced ulcer recurrence, but the eradication regimens included bismuth which was believed to have some direct anti-ulcer activity, leaving a hole in the causal argument.

To deal with this a trial was performed looking at recurrence rates after an eradication regimen using amoxicillin plus metronidazole. Ulcer recurrence rates at one year were 8% in the treated arm and 86% in the placebo arm. To me, this seemed to give a final answer to the question of H. pylori causing ulcers, but I listened to a room full of internal medicine residents from one of the top programs in the country come up with outlandish criticisms of the results. Several residents argued, apparently convincingly to others in the group, that amoxicillin and/or metronidazole might have direct anti-ulcer effects and so the reduced recurrence rates might have had nothing to do with H. pylori. There were several problems with this argument, including the lack of any a priori belief that these antibiotics had such an effect, that an anti-ulcer effect of a 12-day regimen would be unlikely to have such an enormous impact a year later, and that when the trial was analyzed by H. pylori eradication rather than treatment arm, the effect size was even larger.

Another common and generally unfair way to attack a randomized trial is to look for unmeasured confounders or confounders that are somewhat unbalanced in the table 1 data.

Confounders are things associated with both the exposure and the outcome of interest in a study, that through these associations confound the assessment of a relationship between exposure and outcome. For instance, we might find in a study that coffee drinking is associated with lung cancer. Coffee drinkers may smoke more than other people, and so, if we fail to control for smoking, we might falsely conclude that coffee drinking causes lung cancer.

In a properly performed randomized trial of a reasonable size, unmeasured confounders will sort nearly equally among the treatment arms. Even if smoking status is unmeasured, if there are 800 patients in each of two arms, we can be pretty sure that the number of smokers in the two arms will  be close enough not to disturb the trial results. Table 1 in a published RCT typically gives the baseline data on the groups, and if lots of potential confounders are measured and reported on at the 0.05 significance level, we shouldn't be surprised when 1 in 20 appears unbalanced at such a level. That said, if lots of measures appear unbalanced, with one group clearly sicker than the other, it's important to go back and see that allocation in the trial was properly concealed. It's rare, though, to see problems of unbalanced allocation in a major clinical trial, particularly a placebo-controlled trial of a medication.

As is clear from some of my earlier posts on this blog, I obviously believe that some attacks on randomized trials are fair and appropriate. More fruitful attacks often deal with issues of what the GRADE group calls "indirectness". Results of a randomized trial might be indirect to the question a clinician is interested in because the trial did not study the exact intervention of interest (for example, too high or low a dose of a comparator drug was used, as in a trial comparing esomeprazole with omeprazole), perform the study in the exact population of interest (for example, the study was performed in young healthy patients, and the target population is the sick elderly), or measure the exact outcome of interest (for example, a surrogate outcome like carotid intima medial thickness was measured, rather than a clinical outcome).

I recall an argument years ago on Usenet about whether aspartame really causes headaches in the many people who report headaches when they consume aspartame-flavored foods and drinks. At the time, I pointed out the randomized trial in the NEJM showing no difference in rates of headaches in subjects who believed they were sensitive to aspartame, whether given aspartame or placebo. Another poster on Usenet objected, saying that he did not think that aspartame in a pill caused headaches, but rather that the combination of sweet tasting food and lack of sugar led to headaches, perhaps by inducing reactive hypoglycemia. Testing aspartame without letting the subjects taste the preparation was completely unconvincing to him. This was essentially an argument about indirectness of intervention -- the intervention of interest was aspartame as commonly consumed in foods and drinks, not aspartame in pill form.

To me, on reflection, this seemed like a fair objection, though I didn't think it really held up to scrutiny when looking at the actual trial results. In the trial, which had a crossover design, the 40 subjects developed headaches 35% of the time when they were given aspartame, and 45% of the time when given placebo. To argue that encapsulated aspartame doesn't cause headaches in susceptible individuals is to miss that the trial found that placebo induced headaches in these individuals almost half the time.

I'll come back to the issue of indirectness of outcomes in the near future when I'll write more about surrogate outcomes, since this underlies some of the arguments that came up in the comments responding to my vitamin D posting, and I'll likely write about indirectness with regard to population down the road.

The general point, though, is the importance of being fair, open-minded, and keeping a big-picture view when evaluating a new randomized trial, and then interpreting the results in the context of existing evidence. Lee Goldman talks about dealing with randomized trials when one is performing the role of playwright rather than theater critic. But even if you're just the theater critic, if you hate all plays you see, your reviews won't prove terribly edifying to your readers.

Interpreting Confidence Intervals and Grading Recommendations

My previous post, though not really intended to be focused on p values, led to a long and interesting discussion on the issue by Revere.

Revere commented that it wasn't so clear whether I was writing as a frequentist or a Bayesian. In reality I'm a consumer of biostatistics, not a statistician. For me, the issue is pretty simply to try to know whether a given result I'm seeing is true.

Although I wrote about this in my last post I didn't frame it in exactly this way: the problem with p values is that I want to know how likely it is that the result I'm seeing is correct, while the p value tells me how unlikely it is that I would be seeing a given result if the null hypothesis were true. This is similar to the problem with knowing the sensitivity and specificity of a test, when what you really want to know is the positive predictive value (how likely is it that the patient has the disease now that the test has come back positive?).

The academic medical world started hearing a lot about using confidence intervals in preference to p values 15 or 20 years ago, and I think that led to many doctors concluding that confidence intervals solve this basic p value problem. Confidence intervals have some real benefits compared with p values, but this is not one of them. To look at this, we need to try to understand what a CI means.

Again using the HIV vaccine example from my last post, we can look at the same parameter that had a p value of 0.04, but now examine the point estimate of efficacy (31.2%) and its 95% CI (1.1 to 52.1%). What is it that has a 95% chance of being true, given that CI of 1.1 to 52.1%?

However tricky p value interpretation is for physicians, understanding the meaning of this CI is much worse. I've almost never heard a physician able to correctly interpret the meaning of such a CI when put on the spot with the above question. When trying to teach the interpretation I repeatedly get told there must be a simpler way to communicate the idea. Given my years of failure at this, I have little hope that this post will adequately clarify things for most readers, so if others who teach this have found a way that works to explain it, please write!

All we can really say about that CI is that (excluding any problems with the design or performance of the trial) if we performed the same study 100 times and calculated 95% CIs each time, we would expect that 95 of the 100 CIs formed in this way would include the true point estimate of vaccine efficacy.

Note that just as with p values, this isn't what we want to know: we want to know how likely it is that the true value is inside the particular CI that we are looking at, but that isn't what the CI actually tells us.

However, despite recognizing what the CI really does and does not tell us, consumer of biostatistics that I am, I (and others) approach CIs operationally: we choose to interpret the CI as a range of values with which the data are reasonably compatible, and to interpret values outside the CI as reasonably incompatible with the data. So, other things being equal, I would say that a vaccine efficacy of 5% was compatible with the results of the NEJM study, while a vaccine efficacy of 60% was not. This does not mean that I think the study has excluded the possibility of the vaccine having 60% efficacy, just that this would be unusual under the plays of chance.

This operational definition works as I decide how to write recommendations about whether to administer such a vaccine. If the vaccine truly had an efficacy of 31% I would likely recommend wide use in high risk patients. If the high end of the CI were true (52% efficacy), I might recommend universal vaccination. If the low end were true (1% efficacy) I would probably recommend leaving it on the shelf. Looking at this, I can quickly realize that if this trial were the only information I had about vaccine efficacy then I have inadequately precise data to support whatever recommendation (or set of recommendations) I might want to make about administering HIV vaccine.

If I were using the GRADE scheme for grading such recommendations, I might have started with the assumption that I had high quality evidence from a large randomized trial. But when I realize that I would make different recommendations based on the reasonable values at each end of the CI, I know that I must downgrade the quality of the evidence for such  imprecision. Recommendations for HIV vaccine based on this trial, using the GRADE scheme, would certainly be graded as having no better than moderate quality evidence because of this imprecision.

Instead of some arbitrary definition of whether a trial is large enough or precise enough, using the CI in this way allows me to communicate something important about the quality of the evidence as I grade recommendations. I downgrade for imprecision not because a CI crosses a null effect boundary (like 1.0 for a relative risk) but because the CI crosses a clinical boundary where the appropriate recommendation would change from one side of the boundary to the other.

In this way, the CI is far more useful than a p value. I keep in the back of my mind, though, that the CI doesn't really mean what I'm trying to use it to mean -- it's just that I usually don't have anything better.

I'll write more in the future about how GRADE looks at other types of limitations on the quality of evidence from randomized trials.

HIV Vaccines, p values, and Proof

As mentioned in my last post, I want to continue the discussion of plausibility with the recent evidence suggesting benefit from an AIDS vaccine. This was a randomized, placebo-controlled trial in 16,000 adults, that, at least in the initial press releases, showed a statistically significant 31% decrease in HIV infection in vaccinated individuals. How confident should we be in the results?

The analysis that showed the statistically significant benefit (p = 0.04), was a reasonable one. There were 56 people who developed HIV in the vaccine arm, and 76 in the placebo arm, but seven patients were found to be infected with HIV at the time the vaccine was administered, and this analysis excluded those patients. Before we come back to that decision, we should keep in mind what that p value means -- what it is that has a 4% chance of having happened?

Although this causes untold confusion for physicians at all levels of training, that p value says that (excluding any problems with the design or performance of the trial), if vaccine were no better than placebo we would expect to see a difference as large or larger than the one seen in this trial only 4 in 100 times. This is distinctly different from saying that there is a 96% chance that this result is correct, which is how many people wrongly interpret such a p value.

One reason this is so is that many values of truth closer to the null hypothesis (but still where vaccine is superior to placebo) are possible and consistent with the result.

A more important issue of iterpretation in many cases including this one, though, is that we are not starting from a position where all results are equally likely. We had a lot of reason to be suspicious prior to this trial that the vaccine would fail to protect against HIV. Although this way of looking at results is often given important-sounding terms like "Bayesian reasoning", really all we're saying is that if something was unlikely to begin with, we need a higher level of statistical "proof" than if something were likely; if something is extremely likely to begin with, we need a much lower level of "proof".

Imagine a small randomized trial of statins in a population where they have haven't been clearly studied (say, hypertensive smokers without known CHD). If the trial found an insignificant relative risk of CV events in the treated group of 0.78 (22% relative risk  reduction,p = 0.40), it would not be the case that there is a 40% likelihood that this result was due to chance. We have lots of prior evidence suggesting that just such a RR is likely, and our belief that this reduction in risk was due to chance should be much smaller than suggested by the p value in a single trial.

In the case of an HIV vaccine, the failure out multiple prior HIV vaccines gives us a baseline doubt about any new vaccine. Added to that is the apparent difficulty of even actual HIV infection to prevent subsequent infection: patients with HIV who have a normal CD4 count and are on HAART that is adequately controlling their infection can become superinfected with a strain of HIV that is resistant to the HAART regimen they are receiving. This raises real concerns about how to create a vaccine to prevent infection.

So, the modestly positive result found in the trial must be weighed against our prior belief that such a vaccine would fail. Had the vaccine been dramatically protective, giving us much stronger evidence of efficacy, our prior doubts would be more likely to give way in the face of high quality evidence of benefit.

And then, into the mix must enter the reality of the multiple analyses performed by the researchers. Although I said the analysis that gave this result was reasonable, it was not the analysis they apparently first planned to perform. The study specified an intention to treat analysis (that did not find a statistically significant benefit at the 0.05 level), and a per protocol analysis (that also did not find a statistically significant benefit).

While the actual analysis the investigators decided to make primary would be completely appropriate had it been specified up front, it now suffers under the concern of showing marginal significance after three bites at the statistical apple; these three bites have to adversely affect our belief in the importance of that p value. And, it's not so obvious why they would have reported this result rather than excluding those 7 patients from the per protocol analysis and making that the primary analysis; there might have been yet a fourth analysis that could have been reported had it shown that all important p value below 0.05.

The authors of the study tell us that the choice of analysis was made prior to unblinding of the results, but that probably isn't adequate. It had been used in interim analyses (and the final analysis) on the blinded data, but likely so had the intention to treat and per protocol analyses. If this analysis were the only one that achieved a p value below 0.05, it could have been chosen as the preferred analysis for that reason alone. The investigators wouldn't have known prior to unblinding whether the trial would show benefit or harm with vaccine, but either result would be more interesting than a statistically non-significant outcome. The vaccine manufacturer would be far better off with a statistical benefit, but probably no worse off if the vaccine showed harm than if it showed no effect.

I've been struck since the beginning of my medical training that biostatistics is an area of mathematics where people can have remarkably different opinions about what is or is not not resonable. I don't usually think of "opinion" and "mathematics" falling in the same sentence, but perhaps that just shows my lack of expertise in theoretical math.

In this case, while the investigators may have felt they were reasonable picking the most "appropriate" analysis, we who interpret the study have to worry about those three statistical bites. Combined with the prior likelihood that the vaccine would fail, I think we need to remain very wary that despite an apparent 31% vaccine efficacy in a trial involving 16,000 patients, the vaccine may in reality be no better than placebo.

Proof, Plausibility, and a Tale of Two Viruses

Listening to NPR

There was a third incitement to this blog that I have not yet mentioned, as I am not currently in possession of an asbestos suit, but I feel the time has come to discuss it. I was driving to work listening to NPR's Morning Edition last month when I heard, "We have news, now, of a scientific breakthrough about a disease that affects over a million people in the U.S."

My ears perked up, wondering what major breakthrough I had missed. I learned from the interview that an article had been published in Science linking the retrovirus XMRV with chronic fatigue syndrome. On the usually reserved and balanced NPR, I was startled to hear the interview proceed in such a way as to assume that the study was accurate and that it proved wrong all those who had questioned CFS as a single disease entity. I had the sense listening that either the interviewer or the producer of the piece had a personal stake in the outcome of the story.

The article in Science reported that 67% of patients with CFS and 3.7% of healthy controls had XMRV DNA. A news story reported that, after the submission to Science, the authors of the study improved their assay for XMRV antibodies and found them in 95% of patients with CFS. Is it plausible that 95% (or 67%) of patients diagnosed with CFS really have a single disease caused by the retrovirus XMRV?

Plausibility

Plausibility in medicine is a tricky concept. Almost any result can be argued to be biologically plausible by someone with a modicum of creativity. And yet, biologic plausibility often does not pan out so well when it meets real life. When I was in training, beta blockers were considered absolutely contraindicated in patients with heart failure and the incipient notion that peptic ulcers were an infectious disease was considered absurd.

There's a different kind of plausibility, though, for which I have greater fondness. It has to do with prior probabilities developed from a fair-minded look at the epidemiology of diseases or from prior trials that test some similar notion to the current idea under scrutiny. For instance, while JUPITER found a 44% reduction in its composite primary endpoint, it seems highly implausible to me that rosuvastatin really lowers risk by this much. Nearly every statin seems to lower risk in primary and secondary prevention by 20-25%; why should rosuvstatin have such a marked effect in a fairly low risk population? It seems more likely that the effect was exaggerated when the trial was stopped early for benefit than that we can really expect these sorts of results from rosuvastatin in patients with low LDLs and mildly elevated CRPs.

Evaluating the Evidence

The article in Science is an odd read for a clinician. The same article in a medical journal would spend a lot of time describing who the patients and controls were, and how the patients were diagnosed with CFS, but none of that is in the article. Like most articles in Science, it spends its time on bench methodology (how the retrovirus was detected) rather than on the methods by which CFS was diagnosed and the blinding used when the samples were analyzed.

That raises an initial question of "are these CFS patients similar to those who walk into a primary care physician's office carrying a diagnosis of CFS?" That is, can the results be generalized to the broad group of people who are said to have CFS? Without a more detailed description in the article, I have no way to be sure.

But beyond that, what has happened with other CFS "breakthroughs"? Back in the 1990s, after an article showing much higher rates of neurally mediated hypotension in patients with CFS (22 of 23 patients) than in healthy controls (4 of 14 controls), I made the same argument on Usenet that I'm in the process of making here. That article led to patients with CFS being treated with fludrocortisone and similar agents. Eventually people concluded that the neurally mediated hypotension story didn't really make sense and clinicians and patients moved on.

From my point of view, given the patients I have seen carrying the label CFS, it seems extremely unlikely that any single etiology will explain 95% of cases. Patients carrying the label CFS in the real world (rather than in research studies) seem to be a mixed bag. Many seem to have atypical depression or another psychological cause of fatigue (thus my need for the asbestos suit). Others, though, likely have a missed diagnosis or have some condition that is, as yet, unknown to medical science (retroviral infection with XMRV would have previously fit in this category).

Interpretation and Proof

Why does it matter? From the clinical perspective, many people with depression (typical or atypical) can be helped by standard therapies. Avoiding the label of depression for those who are depressed is no kindness (though it is also of no help for doctors to insist that symptoms are due to depression when they are not; I do not believe that all patients with CFS really suffer from depression). And getting therapies for purported etiologies that don't pan out (such as fludrocortisone for neurally mediated hypotension) simply exposes patients to side effects, harms, and costs, without resulting benefits.

Beyond that, though, this issue is at the intersection between evidence, experience, and opinion. I have no expertise in retroviruses, know nothing about XMRV or the methods used to detect it, and am lacking detailed evidence about how the study in Science was performed. But evidence about diseases still must be interpreted through the lens of clinical experience.

Ultimately, many of the arguments between clinicians about CFS (let alone the arguments joined by patients) have to do with differing clinical interpretations of whether CFS feels like a believable single disease process. This is an area where a discipline like EBM doesn't provide much of a road map. Yet expert clinicians must make these sorts of judgments all the time when they decide whether to consider a cluster of signs and symptoms as an individual disease. Before HIV was discovered, I listened to expert clinicians reject the arguments for multiple immune system insults or inhaled nitrates as causes of AIDS, and instead point out that AIDS would turn out to be caused by a virus that was transmitted similarly to hep B given the similar epidemiology of the two diseases. There is no path from evidence to understanding that does not rely on expert interpretation, and, ultimately, no mechanical measure of sufficient evidence or proof outside of what counts as proof to those patients and providers who must make decisions.

XMRV

XMRV may yet turn out to be the cause of CFS, but given the track record of such causes, I think we need better evidence. Before NPR describes a "scientific breakthrough" and gets the hopes up of so many, I wish they would spend more time thinking and interviewing and looking for some balance. I have to believe they could have found an expert or two who would have urged caution in interpreting these results.

If future studies confirm the XMRV hypothesis, those out there now who are infuriated by my comments can feel free to say "we told you so." But if XMRV turns out not to be a very good etiology for real world cases of CFS, perhaps there might be more patience for those of us who urge not trying to explain all the many chronically fatigued patients with a single disease entity. In either case, it's worth recognizing the role that expert interpretation plays in medical controversies.

This post is long enough, and the second virus from the title of this piece will need to wait for a future post: I'll discuss the likelihood that we've really found an effective vaccine for HIV.

(After I wrote this, Marilyn Mann pointed me to a blog posting with some similar concerns about CFS having a specific viral etiology.)

Vitamins and Faith

I'd mentioned in an earlier post that a relative was prescribed fairly high dose vitamin D by her primary care physician despite a blood test showing that her vitamin D level was in the normal range. Her PCP may not be much of an outlier. Among the residents I work with, I've been encountering a lot of desire to prescribe vitamin D to pretty much everyone.

Why the willingness to administer vitamins? Would these same doctors feel that a drug with similar levels of evidence for safety and efficacy should be administered to tens of millions of generally healthy people?

Somehow patients and providers alike seem willing to treat vitamins as articles of faith -- as substances that must be good for humans based on their very name and nature. Yet the actual track record of vitamins, and particularly the fat-soluble vitamins, should not lead us to be so sanguine.

Based on their antioxidant properties, vitamin A/beta carotene and vitamin E were believed to be good for health before any reasonable studies had been performed. Doctors recommended beta carotene and vitamin E at moderate or high doses, and health-conscious patients took even higher doses. Then the results of randomized trials started to come in:

 Two large randomized trials of vitamin A/beta carotene found increased rates of lung cancer in smokers, pretty much putting an end to the notion of using beta carotene to prevent cancer. Then, a meta-analysis of trials of vitamin E found an apparent increase in mortality in patients treated with higher doses of vitamin E, and suddenly vitamin E no longer appeared so benign.

With this as a backdrop, we might expect physicians, at least, to show some skepticism before recommending other vitamins, and particularly other fat-soluble vitamins.

Yet, in their defense, vitamin D has been studied in randomized trials (mainly for prevention of fractures/osteoporosis), and these trials have reported other outcomes including cancer rates and mortality.

Although much larger trials have found no effects of vitamin D on cancer rates, one trial that I've heard people cite did indeed find a reduction in cancer rates in patients supplemented with calcium and vitamin D. The RR was 0.23 -- in other words, the supplementation appeared to reduce cancers by 77%. Since virtually nothing in medicine has an effect that large, and such an effect would be hard to have missed in other trials of calcium and vitamin D, this result seems implausible at best.

With regard to mortality, a meta-analysis of randomized trials found that vitamin D supplementation appeared to decrease mortality by 7%. A concern, though, is that the result was just barely statistically significant, and there could be reporting bias with regard to mortality: trials that found larger differences in mortality (or any other safety endpoint -- these were, after all, not trials designed to look at mortality) would be more likely to report on mortality than trials that found nothing interesting when they looked at their safety data. Given the risk that the results could be due to chance, even a small amount of reporting bias would be worrisome.

So, I've now objected to one trial of vitamin D for having too large an effect to be believable, and a meta-analysis for having too small an effect to convincingly rule out chance plus bias. Am I being fair and true to the evidence?

If we were talking about a therapy for a morbid illness where we were compelled to treat, I'd likely accept the evidence from the meta-analysis as being sufficiently hopeful to make vitamin D worth a try. After all if treatment is required, even if vitamin D eventually turned out to be harmful, we would know that we had needed to do something for the ill patient and so we made our best guess.

But that is not the case here. Instead, we are talking about administering a therapy to enormous numbers of healthy people with the hope of preventing disease, and where causing even small increases in harmful outcomes would have no justification. And remember the history of vitamin E and beta carotene. If that is not sufficient to make you concerned, read what happened to cancer rates and mortality in two trials of folate/B12 supplementation in Norway, published just last week in JAMA.

In such a situation we should wait for (and demand) more convincing evidence before we act. It may turn out that vitamin D really does prevent cancer, decrease mortality, and protect people from everything from influenza to the common cold. But I would rather be late jumping on this bandwagon than jump early and find that I hadn't learned a thing from past mistakes.

Pharma, Evidence, and Trust

A number of the other blogs that have noted the existence of this blog seem to have "Pharma" in their titles in one way or another, and the implication is not that the blogger likes much about Big Pharma.

In the late 1990s, when I was spending much of my clinical time providing HIV care, I was a defender of Pharma on Usenet, at least as it related to HIV. We'd discovered the HIV virus in 1985, and by late 1995 we had a spectacularly successful treatment, due, at least in part, to the efforts of the drug manufacturers. HAART changed the lives of people with AIDS, and the lives of their caregivers, and costs of $10K to $15K per year to keep a young person alive and functional seemed perfectly reasonable to me. I thought the carping about the high prices of these medications was unfair -- it was a remarkable accomplishment to find an effective treatment for a viral infection in so short a time, and the billions spent on it deserved some reward.

By a few years later, though, I was finding myself as distrustful and angry at Pharma as anyone out there. I spend a lot of my time reading research papers and interpreting evidence, and realized one day that Pharma was employing people at least as smart and knowledgeable as I to come up with ways to trick us all about what studies showed. One example of this is running parallel studies of an agent and then only publishing the positive studies while burying the negative studies, as was done with trials of SSRIs in adolescents. The concept of a p value becomes essentially meaningless if researchers perform an unknown N of parallel trials and then publish the ones that show a positive result.

I'll write about other ways of manipulating results in future posts, but for today I want to comment on one pernicious effect of this behavior -- I have come to distrust and then dislike the companies that provide the (often useful) drugs I must use on patients. When the new HDL-raising drug torcetrapib went down in flames a few years ago, I was actually pleased: I was furious with Pfizer for having tried to get torcetrapib approved in a way that would have only marketed it in a combo pill with atorvastatin. Objectively, it makes no sense that I should be happy at the failure of a drug that would have helped my patients had it worked.

Pharma not only tries to manipulate the interpretation of results of trials, in the design, analysis, and publication phases, they apparently also try to sow doubt about existing competitors. Adriane Fugh-Berman wrote in 2005 about how she was recruited to be the author on a ghost-written manuscript intended to highlight the importance of herbal interactions in patients on warfarin (interactions that Dr. Fugh-Berman felt were overstated). The company recruiting her was presumably working for AstraZeneca, the manufacturer of ximelgatran. Ximelgatran, a direct thrombin inhibitor that aimed to replace warfarin, was eventually withdrawn from the market in Europe (never approved in the US) because of hepatotoxicity.

In the face of this comes the RE-LY trial comparing warfarin with dabigatran, another direct thrombin inhibitor. I didn't start blogging until well after the publication of RE-LY, but hopefully others noted that this 18,000 person trial may reflect one of the biggest advances in medical therapy of the decade. Dabigatran appears to be a superior drug to warfarin -- at lower doses it is safer than wafarin and as effective, and at higher doses it is equally safe to warfarin and more effective. Unlike warfarin it does not require a careful diet or routine monitoring of INRs.

By all rights, dabigatran should take over the oral anticoagulation market, at least for atrial fibrillation (we need additional trials in other clinical settings). The natural skeptic in me would always want to wait a bit on using a new drug. Warfarin has been around for decades and has been used in tens of millions of people and we have a good sense of its benefits and (myriad) risks and burdens, while dabigatran has not been used nearly so widely nor for so long. New drugs have a habit of showing up rare side effects once they are out of clinical trials and on the market.

But my greater fear is that somehow Pharma is pulling something over on me, the editors of the NEJM that published RE-LY, and everyone else, and that it will somehow turn out that dabigatran not only isn't as big an advance as it appears, but that its manufacturers already know this and aren't telling.

Ultimately, if I had to bet, I don't think this will be the case with dabigatran. Five to ten years from now, I'm guessing that the residents I teach will have no clue that we used to routinely treat patients with rat poison and monitor bleeding parameters every couple of weeks or so in order to prevent clots and emboli. But I wish I didn't have to fear that the manufacturers of the drugs I use might not actually be working toward the same goals that I am, and instead might be working to fool me yet again.

USPSTF, Mammography, and Grading Recommendations

It's hard to have a blog about evidence and medicine without commenting on the changes to the USPSTF's breast cancer screening guidelines.

At a first pass, it's worth pointing out that a lot of academic primary care physicians have been wondering what the USPSTF was thinking when they published their 2002 guidelines recommending mammography for women in their 40s. Yes there may be some small benefits (and lots of harms), but this is just the sort of thing that the USPSTF can usually be relied on to not recommend -- look for instance at what they recommend for prostate cancer screening or lipid screening for women. As a result, there were lots of insinuations that the USPSTF had caved to political pressure in making the mammography recommendation. Perhaps others know more about that one way or the other.

Now they've backed away from that recommendation, going from a favorable to a negative recommendation for mammograms. The overall issues aren't actually that complex: if you imagine 1000 women in their 40's getting mammograms yearly for 10 years, the best estimates are that you will cause more than half of them to need repeat mammograms for concerning findings, and will cause nearly 200 to get breast biopsies, while only preventing 2 deaths from breast cancer.

What is remarkable is the firestorm that has been set off. When I present these data to women patients in their 40s (in a way that I think is fair to both sides of the argument), I've found that approximately half want mammography and half want to wait until they turn 50. Individual patient values and preferences heavily influence the decision. The USPSTF takes this into account by recommending against routine mammograms in the 40s, but states that a decision to screen should take consider an individual woman's values. All in all, not that unreasonable a position as far as I'm concerned.

I think, though, that the USPSTF's way of grading recommendations is awkward in general and may even have contributed to the confusion over their 2002 and 2009 recommendations. The evidence base didn't substantially change over that time frame, but the recommendations did, and that opens them to attack.

I think the USPSTF would be better off using a different grading system, such as GRADE, which they explicitly chose not to do recently.

(Here I need to make one of the only conflict of interest disclosures I should ever need to mention on this blog: one part of my work is as a medical editor, and in our publication we use GRADE.)

That out of the way, the USPSTF grading scheme is unusual in the way that it combines strength of recommendation and quality of evidence into a single value (a letter grade). To me, this overloading causes problems. For many of the recommendations that USPSTF gives a grade of I to (insufficient evidence), GRADE and a number of other schemes would note the low or moderate quality of the evidence and still allow a guideline committee or other recommending body to make a suggestion about what a clinician should do.

When there is enough evidence to make a suggestion (a "weak recommendation" in GRADE parlance), GRADE explicitly deals with the need for considering patient values and preferences. If, instead of being used to this single letter grade from USPSTF, we were used to seeing evidence and recommendation divided, I'm not sure the recommendation change would have caused quite so much trouble.

The USPSTF recommendation, cast into GRADE style, would likely have read:

For most women ages 40 to 50, we suggest not performing yearly mammography (Grade 2B). Women who are willing to accept a high risk of false positive mammograms and a high risk of unnecessary biopsies in return for a small mortality benefit may reasonably prefer annual mammography starting at age 40.

Would a recommendation written this way have caused the same firestorm? I'd be interested in what others think about this. Would it have caused a firestorm if the 2002 recommendation had said the opposite?:

For most women ages 40 to 50, we suggest performing yearly mammography (Grade 2B). Women who are unwilling to accept a high risk of false positive mammograms and a high risk of unnecessary biopsies in return for a small mortality benefit may reasonably prefer annual mammography starting at age 50.

You can see how easy it is to split the unchanged evidence base from the change in what you are suggesting for the majority of women (based on a guess about their values and preferences) when using GRADE. I think such an approach is more transparent than that used by the USPSTF, and keeps the focus where it belongs in a situation like this -- on the benefits, risks, and burdens that influence the decisions for individual patients and for society.

I'll write more about the methods of GRADE, including its strengths and weaknesses, down the road. I do want to say that while I don't like the grading scheme used by USPSTF, I think they mostly do a wonderful job of staying true to the evidence as they present their guidelines. We should all hesitate to recommend screening tests in the absence of solid evidence showing benefit, and the USPSTF can almost always be counted on in this regard.

ARBITER 6 and Combining Therapies with Statins

The results from ARBITER 6 were released this week.

The interpretation is less clear.

I've been in a minority who has doubted that most drugs that lower LDL are actually beneficial in primary prevention. Statins clearly show benefit, but really no other agents have shown a reduction in all cause mortality or, in primary prevention, in cardiovascular mortality. As such, I'd been raising concerns about ezetimibe even before ENHANCE was published, and have been unconvinced that anyone should be treated with combination therapy to achieve LDL goals, since there is little to suggest added benefit when other lipid-lowering drugs are combined with a statin.

Recently, the Oxford Niaspan Study found that CIMT was improved when niacin was combined with a statin, but CIMT is a surrogate marker and it's not so clear that changes in CIMT will predict clinical benefits. However, of all the non-statin therapies, niacin has been the one that looks most promising for reducing cardiovascular mortality, at least in secondary prevention.

Now, soon after that study, we have ARBITER 6, showing that niacin plus a statin seemingly reduced cardiovascular events compared with ezetimibe plus a statin. There were 2 events in the niacin arm (1%) and 9 in the ezetimibe arm (5%) and this was statistically significant. So, what can we conclude from this?

One possibility is that niacin actually does add some clinical benefit to a statin in secondary prevention, while ezetimibe does not.

Another possibility is that ezetimibe is actively harmful, while niacin is neutral. There were slightly more events in ENHANCE in the ezetimibe arm, but the numbers were tiny.

A third possibility is that this was due to chance. The actual results are quite "fragile" in the sense that only a few events moving from one arm to the other would make the difference disappear. While, in theory, a statistical test captures this, there's been a sense in the EBM world that very small event numbers create a fragility of results not adequately captured by a p value. It seems extremely unlikely that niacin really reduces events by 80%, since this should have been obvious in prior trials of niacin. As such, either niacin combined with a statin is much superior to niacin alone, or these results are in part due to the play of chance. (It's not likely that ezetimibe raises events by this degree when used with a statin, since we would likely have noticed this in other trials.)

ARBITER 6 was stopped early for benefit, based on CIMT results. This seems like an odd decision to say the least, since change in CIMT is not a clinical outcome requiring an ethical decision to stop a trial. The early stopping was unfortunate since with more events we might be more certain about whether the reduction in events was real.

I'll be interested to see how others interpret ARBITER 6. So far I've communicated with one lipid expert (who has not shared my views about non-statin therapy) and he doesn't believe that ezetimibe is clearly to be avoided, while another expert (who does share my views about non-statin therapy) doesn't feel that ARBITER 6 means that niacin should be used. Apparently neither one found ARBITER 6 substantially reshaped their positions on this.

My best guess is that niacin reduces events a bit, though not as much as ARBITER 6 suggests. If so, for the highest risk patients, such as those who have CV events while on a statin, adding niacin might make sense. Given the side effects with niacin (36% of patients in ARBITER 6 had  flushing, and many more in the niacin arm than the ezetimibe arm discontinued therapy), I'd hesitate to use niacin in lower risk patients even for secondary prevention, out of concern that they might discontinue both niacin and the statin.

I'd be quite hesitant to treat anyone with ezetimibe at this point. While I was a skeptic that ezetimibe did anything beneficial even before ARBITER 6, I'm now substantially more concerned that it could be causing real harm.

Opinion, Evidence, and Consensus Statements

In the future, I'll write more about this subject, since the distinction between expert opinion and clinical experience seems to cause unending confusion to people who write and talk about EBM.

For now, though, I want to write about the second recent event that caused me to decide to blog on evidence and medicine. I caught a snippet on NPR's Science Friday where they were talking about concussions in athletes and heard the following: "And take a very careful step-wise return not just to the playing of the sport, but also to even, you know, text messaging and doing things intellectually that can be harmful to their brains while they're still recovering from a concussion." www.npr.org/templates/story/story.php?storyId=120174333

I was startled to hear that an expert thought that someone should not be texting after a concussion (it turns out this is referred to as "cognitive rest"), and I had a hard time believing there was any evidence for it. And though I have no expertise in concussion management, the idea sounded somewhat contrived or even goofy to me.

It turns out that this mainly comes from consensus guidelines for the management of concussion (PMID 19433429) where, if they have any evidence to back this up, I couldn't find it. I certainly didn't find any published studies showing that cognitive rest improves outcomes after a concussion. If anyone reading this knows of such evidence, I'd be interested to see it.

To make recommendations, experts always need to interpret what evidence there is (expert opinion), and in the absence of high or moderate quality evidence about cognitive rest, the consensus panel presumably had to make their best guess for management. This particular recommendation seems a bit odd in that it has potentially high burdens on the injured athlete (don't read; don't text) in the absence of good evidence, and rest is not always such a good idea after a physical injury -- bed rest for low back pain and immobilization for an injured shoulder are both clearly a bad idea. But ultimately doctors need to either recommend cognitive rest or not recommend it and if some people think it's a good idea an expert recommendation is required.

However, a certain degree of humility would seem to be required when transmitting a recommendation grounded in such low quality evidence. Perhaps this is not such an instance, and I am just missing the better evidence, but it would certainly be common to hear expert panels and individual doctors make forceful pronouncements based on little or no foundation. If there really is nothing much to support this, perhaps the experts describing the recommendation (both in the guideline and on the radio) might have said, "Well, we think cognitive rest is a good idea, but we don't really know. In fact, for all we know, it might do more harm than good."

So, while I'll write more about expert opinion and clinical experience down the road, this episode of seeing yet another consensus guideline treated as gospel was an added push to create this blog.

First post and origins

Although an early participant in the world of Usenet newsgroups, I'm a latecomer to the world of blogging. I don't read or participate in blogs often, and as a newbie will undoubtedly make mistakes of etiquette and function much as I did in the very early days of Usenet before I got a feel for the music of the newsgroups. I hope anyone reading this will take this into account and cut me some slack!

I'd been considering for a while now blogging about evidence and medicine, something I spend a fair amount of my time thinking about in any case, but a couple of events pushed me to take the plunge.

The first was when a relative, a woman in her mid-40's, called to tell me that she'd seen her new primary care doctor. The doctor had run tests on her including checking a lipid profile. The lipid profile showed a total cholesterol of 160, an HDL of 44, and an LDL of 105. Based on these results, the doctor informed her that she should be very concerned about her "low" HDL number and told her to start taking fish oil.

This was an interesting interpretation of the results and an unusual suggestion for management, and it gives some sense of my concerns about how evidence is used, ignored, or misused in medicine. This relative had a low normal blood pressure, no smoking history, did not have diabetes, and had no family history of coronary heart disease. As such, there is even  some controversy as to whether she should have had a lipid profile checked at all.

Assuming a lipid profile was worth doing, she should certainly have been told that this was an excellent profile putting her at low risk for a cardiovascular event, and that there was little modern medicine could do to substantially affect her risk. Her 10-year risk of an event (using the 2008 calculator that overestimates vascular risk by including things that probably shouldn't be included -- a blog for another day) works out to about 2.2%. If we pretend that somehow fish oil could increase her HDL to 80 and that she would get all the same benefit as if her HDL were naturally 80, we could drop her calculated risk to about 1.4%. So, if she took medication daily for 10 years, she would reduce her risk of an event (under these outlandishly favorable assumptions) by 0.8%. Not something this relative was interested in doing.

But, do we even know that fish oil raises HDL? The main effect of fish oil on lipids is to lower triglycerides. Most studies suggest that fish oil only raises HDL levels by about 3%, so we might expect an increase from an HDL of 44 to an HDL of 45. It's hard to see how the doctor felt this would fix the woman's "low" HDL.

But, beyond that, we have no solid evidence that raising HDL with  medication improves outcomes. Changing HDL with medication is a surrogate outcome for that medication  -- that is, it is of no direct interest to the patient unless it also predicts better cardiovascular outcomes, and HDL appears to be a poor surrogate.  Some medications that raise HDL appear to cause harm and worse vascular outcomes.

As for whether fish oil supplements are beneficial through some non-HDL mechanism, I think we can safely say that the jury remains out as to whether a woman in her 40s at low cardiac risk could expect any important benefit from fish oil supplementation.

Interestingly, my relative's doctor also checked her vitamin D level and told her that a level of 34 (in the normal range) was concerningly low and that she should start taking 1000 IU of vitamin D per day (she was already taking 400 IU). Vitamin D supplementation is controversial, with lots of people claiming big benefits, but we can safely say for now (again, perhaps a longer blog for another day) that there is no solid evidence that having a woman with a normal vitamin D level start taking additional vitamin D is likely to be of any benefit to her.

My relative asked what I thought these events said about her doctor, but I'm not really sure. I see decisions like this made all the time, where evidence is misunderstood, misapplied, or ignored. I have the sense that this particular doctor may believe she is getting out ahead of the evidence -- suggesting therapies that she believes will be proven down the road to be of benefit. She could, of course, turn out to be right, but we should be very hesitant to give preventive medications without a decent evidentiary basis. Other drugs that raise HDL have increased mortality. Supplementation with other vitamins that were thought beneficial has turned out to cause cancer or increase mortality.

I said there were a couple of events that prompted me to start this blog, but this post seems long enough. I'll talk about other events in future postings.