Evidence In Medicine

Video now available

The video links at the beginning of my previous blog post were taken down close to the time of that posting, but are available again elsewhere on Youtube. I've edited that post, if anyone wishes to see the video presentation.

Evidence and Snake Oil

Recently a friend wrote to me and asked what I thought of a presentation she had heard about a product containing beta-glucan: 

Presentation part 1

Presentation part 2

Presentation part 3

I'm not suggesting anyone spend the time to watch all of this, but it is a 30 minute talk at a meeting (of salespeople, I think) of a company planning to market a version of an existing product with a preparation of beta-glucan added to it.

The speaker is excellent, and his audience is clearly impressed to have such a knowledgeable scientist speaking to them, and excited at the chance to market a new healthful product that the speaker promises will be one of the major revolutions in health in this half century or even century.

A claim like that is a bit astounding, and would obviously require high quality evidence to back it up. As Carl Sagan used to say, "extraordinary claims require extraordinary evidence". That someone would make such claims at a sales meeting, such that the attendees were hearing about this revolution in health there first, rather than from the mainstream media reporting on a health breakthrough, might cause the incredulous to suspect they were about to learn about snake oil.

The credulous, on the other hand, might believe that meetings like this are exactly where health breakthroughs are presented because the mainstream media and mainstream doctors would never allow such information to be presented in a medium that they control.

The speaker talks of the astounding immune stimulating virtues of beta-glucan to repair the ravages that modern society has wreaked on our immune systems and points to a number of clinical trials. He goes on to describe the benefits shown in the clinical trials (mainly for upper respiratory tract infections -- URIs) in glowing terms.

Since I'd been asked by a friend for my opinion, I decide to see what I could learn. I searched for clinical trials of the preparation for URIs and found a small randomized trial showing no apparent benefit of a beta-glucan product on URI in athletes.

Not surprisingly, this RCT had not been highlighted in the video. However, a number of other trials were mentioned and were linked on the product's website. Two of the trials listed were apparently only presented in meetings, one was reported as published and I was unable to find the full text (perhaps others will have more luck), but one trial was published in a form I could access.

On the product website, the accessible study was described as showing:

While there were no significant differences in the incidence of symptomatic respiratory infections among the study groups, the duration and severity of symptoms were alleviated in subjects receiving Wellmune WGP.

In the study results, the Wellmune WGP group reported:

• No missed work or school due to colds, compared with 1.38 days of work/school missed for the placebo group. (p = 0.026)

• No incidence of fever, compared with 3.50 incidence in the placebo group. (p = 0.042)

• An increase in quality of life, including physical energy and emotional well-being, as measured by a clinically validated health survey questionnaire (SF-36v-2). (p = 0.042)

• No adverse events were detected and no safety concerns were present.

This actually tracks fairly closely with what the published study states in its abstract. However, if you actually read the paper it becomes quickly apparent that an enormous number of comparisons were made, nearly all of which showed no benefit to beta-glucan (the missed work or school due to colds was one isolated positive finding). The reported effect on fever does not match the data in the text of the article and, even there, no effect on fever was seen in the intention to treat analysis or in the per protocol analysis, though the latter got close to achieving that all important p<0.05 though it didn't make it.

I communicated back to my friend my doubts that anything very interesting had been shown about the product and inquired about possible financial incentives influencing the interpretations of those who were reporting the results to her.

Why is any of this even worth thinking about? After all, the claims made in the video had raised my doubts as soon as I'd heard them, and there are many such claims for many products that are at least as dubious.

The reality, though, is that this beta-glucan product will be marketed and may gain traction. The friend who asked me about it actually currently sells an acai berry product in which she is a true believer. Acai berry products are being marketed all over the Web and doctors are now fielding questions from patients about their value. In my role as a medical editor, I was asked recently by a reader (a skeptical physician) to have our resource include an article discussing the evidence for and against clinical benefits from acai berry supplements.

If we wrote such an article about acai berry, what should it contain? There is no high quality evidence about acai berry and clinical outcomes. Should an article take an agnostic view about its benefits? Should it point out that lots of antioxidant supplements have failed to improve health or have even caused harm? Should it be written at all, or is there some threshold of believability or widespread use that must first be overcome.

The scientific medicine world generally claims it expects some reasonable evidence before administering a new preparation or trying a new procedure (though is not always as good in practice as in theory), and the pharmaceutical industry, for all its problems, can't release a drug without impressive clinical trials.

But when random claims of snake oil salesmen selling some new product spring up, eventually, if the product really does become widespread in the marketplace, the expectation of many in the media and the public seems to be that the burden has shifted to scientific medicine to disprove the benefit or else not take a position about whether a product is good or bad.

At the extreme, you have the most ludicrous ideas of alternative medicine like homeopathy. But even more plausible-sounding ideas when untested do not deserve much credence. That someone who is selling a product can weave a good story about why it is beneficial should not require the resources of the rest of us to test those claims.

We'd all be better off starting out as skeptics, particularly when those promoting an untested product have a financial stake in its success. Snake oil salesmen have been a bane of appropriate medical care for centuries, and there's no reason to think this will change. Those who care about science and evidence, though, must remember that we need not approach entirely untested claims from a position of neutrality. Most things you could imagine doing to improve health do not work, and many are harmful. Though absence of evidence is not evidence of absence, we need not spend time and thought disproving the claims of every snake oil salesman who comes along. It already takes enough resources testing just those claims that have some shred of plausibility as well as the products that have become so widespread that letting them stay untested endangers health.

Evidence, Medicine, and EBM

Lay of the Land

There have been some recent blog posts around yet again debating the concept of evidence-based medicine (EBM).

This blog is titled "Evidence in Medicine" not because "Evidence Based Medicine" was already taken as a URL and blog title, but rather because I didn't want some of the baggage and implications of the EBM label.

EBM is frequently defined as "the conscientious, explicit and judicious use of current best evidence in making decisions about the care or individual patients", and you would think it would be hard to object to that. And yet....

One blogger, Laika Spoetnik (Jacqueline), recently wrote a piece complaining about the attacks on EBM, and mentioned the humorous parachute example that I had referred to in an earlier post. She wrote about the parachute piece::

I found the article only mildly amusing. It is so unrealistic, that it becomes absurd. Not that I don’t enjoy absurdities at times, but  absurdities should not assume a live of their own.  In this way it doesn’t evoke a true discussion, but only worsens the prejudice some people already have.

(As I'd suggested in that same earlier post, EBM folks are not necessarily known for their great senses of humor.)

In response to Jacqueline's post, Kimball Atwood wrote a thoughtful piece explaining his problems with EBM, and linked to an earlier piece he'd written on the same subject. I would encourage people to read Dr. Atwood's posts.

Personal Systematic Reviews

So, how could any reasonable person possibly object to the use of best evidence in making medical decisions?

The answer is that they could not, but that EBM is not necessarily synonymous with using best evidence, despite the definition above. If I'd been writing a few years ago, I would have agreed with a number of the posters attacking EBM, since many of the people I have seen promulgate EBM do so in a way that misses some basic truths.

EBM,at least as it was promoted in academic internal medicine circles in the 1990s, was very focused on a specific hierarchy of evidence (RCTs > cohort studies > case control studies > case series) and on the belief that we should train primary care internists that the right way to answer a clinical question they encountered was to perform a literature search along with their own, on the fly, systematic review. I'll come back to the hierarchy of evidence in a bit.

The idea of teaching people to perform personal systematic reviews was being pushed at a time when I was co-directing the resident journal club at my hospital. This was couched as "teaching EBM", and the various residency course directors around the country seemed to be spending a lot of time telling each other what an important activity this would be for academic general internists to engage in. I and the other director of journal club pointed out repeatedly that real systematic reviews take months, and that it was a bizarre notion to think that the right way to answer a question that arose in a clinic would be to try to do this on the cheap. A quick review of randomized trials is fraught with the likelihood of missing important pieces of evidence. When residents try to do  this, they often end up focusing on the most recently published trial rather than the entirety of the available knowledge.

We argued, instead, that for most such questions residents would be far more likely to find the right answer in a textbook than by trying to interpret a body of literature in between seeing patients. (These days I would make the same argument but replace "textbooks" with "electronic resources", however that raises a conflict of interest with one of my day jobs, so I won't push that particular argument any further.)

As a result of our resistance, this curriculum never quite got implemented for our journal club, but my sense was that put our hospital in the minority. I think many programs in the late 90s and early 2000s tried to convince residents that it was sensible, practical, and necessary to perform their own systematic reviews whenever confronted with a clinical question.

Hierarchies of Evidence

Coming back to the hierarchy of evidence issue, it's clear that a system that focuses too much on the hierarchy mentioned above will reach absurd conclusions. For instance, one might decide that we have higher quality evidence that an HIV vaccine is protective than that people who wish to avoid lung cancer should not smoke. We have essentially no evidence from RCTs in humans supporting that latter hypothesis and yet we should clearly believe it to be true as surely as we believe anything in medicine, while it remains quite uncertain that an HIV vaccine is of any benefit despite the published RCT with a "statistically significant" p value.

And when effect sizes are large enough, even a couple of clinical events may provide higher quality evidence than some RCTs that try to examine questions of minor benefits of therapies. How many people have to survive meningococcal meningitis after treatment with penicillin before you can be quite sure that penicillin is lifesaving?

Dr. Atwood goes to the specific issue of biologic plausibility in the hierarchy of evidence as it applies to homeopathy. I actually agree with those who are suspicious of biologic plausibility arguments, since smart people can make almost anything sound biologically plausible. Dr. Atwood focuses more on biologic implausibility and I tend to find these arguments more compelling on average. In the case of homeopathy, he argues, it would be pointless and a waste of resources to perform RCTs examining homeopathic remedies. This is a fair point, since if homeopathy is true than our understanding of chemistry and physics is incorrect; thus it would require extreme levels of proof before we should begin to entertain the notion that a homeopathic preparation could be of benefit.

(I plan to write a future post about the burdens placed on the scientific medicine community by people who promote unproven remedies; valuable resources must then often be spent to prove the lack of benefit of the now widely popular yet worthless preparations and interventions.)

EBM and GRADE

Given what I've written you might conclude that I lean toward the anti-EBM camp. As mentioned, that's likely where I would have placed myself a few years ago. In the interim, though, I've started working with the GRADE group and in particular with Gordon Guyatt.

Through these interactions, I've come to realize that when a really smart and thoughtful clinician like Dr. Guyatt discusses EBM, the above problems and concerns mostly go away. For instance, GRADE recognizes the importance of randomized trials, but also that high quality evidence can be found in other places. The group works to formalize what might lead rational clinicians to conclude they have high quality evidence, and while I don't agree with GRADE 100% of the time, my disagreements mostly involve some edge cases that really don't arise very often.

And since Dr. Guyatt is generally credited with inventing the term EBM, it's hard to dismiss his interpretation as being unrepresentative of true EBM. Instead, I've come to believe that most of the people who have interpreted and disseminated "EBM" lacked either the clinical understanding or the epidemiologic knowledge to communicate a fair representation of EBM to the clinical world.

I'd encourage those who find themselves troubled by aspects of EBM to spend some time looking at what the GRADE Working Group is doing. Even better, if you have the chance to interact with Gordon Guyatt at one of his courses or lectures, jump at the opportunity. You likely will feel differently about EBM for having done so.