Evaluating Medical Claims

Today I was talking to an almost-doctor (graduates med school in two weeks) I know in a social/religious context. We were talking about various things and he said he'd like to see more of me and I said if he wanted to set up a schedule to do some studying, I'd be open to that- what would he like to study? I was intending for him to name a religious topic.
But he said he'd like it if we could be medical chevrusas*, maybe, if he has time for regular get togethers, which he isn't sure he does. If we could pick a medical topic and then both of us read up on it, and get together to discuss it.

And then I started thinking about ground rules for discussion of medical topics (I also started thinking about medical topics that might be interesting to discuss- the major ones that come to mind are issues of debate such as chronic pain management). That struck me as being relevant to a comment I got many posts back when I wrote about the credibility of the big pharma and the medical establishment. I was asked what I thought of alternative medicine. So here are my thoughts.

1. Speculation is interesting but takes a back seat to evidence.

2. The only evidence that counts is data, not theories, not what a doctor says, and not a textbook recommendation. If somebody doesn't tell you how he knows something, he doesn't know it. Evidence that I consider relevant is practically everything: animal and human studies, case studies, molecular studies, studies on the disease in question and on related diseases, and epidemiological data. Especially epidemiological data.

3. Evidence never proves anything other than that it exists. It suggests many things. When considering evidence, look for the data source, not the researchers' interpretation of it, and always keep the limitations in mind. For instance, the time and place, type of research subjects if any, lab norms if any, data not given, placebo effect, and other variables.

4. The placebo affect should always always always be considered in human studies, particularly case studies. This is why you will get testimonials that something works when it doesn't. People think they got better because of the pills when actually it was their belief in the pills.

5. The related phenomenon is false attribution, for instance, if a doctor (or patient) claims the patient got better because of the medication, whereas the patient got better just because he happened to get better.

6. Understand the values or motivations of researchers when evaluating their advice.

Medical studies or advice is often based on a value system, sometimes more obvious than other times. A recommendation to take a medication may stem from the belief on the part of the doctor or researcher that the disease is worse than the side effects of the medication. Identifying these values is important because many researchers and some doctors are not even aware that there are other value options.
For instance, your doctor may believe that it is better to take fewer shots even if this raises your A1c a bit, because he thinks that taking shots lowers your quality of life. You may or may not agree.
In a less obvious scenario, a study came out a few months ago claiming that obesity lowered life expectancy even more than previous studies said. Upon closer inspection, this was because the doctors were looking at quality of life years- and they had decided that quality of life with certain obesity related diseases was even
worse than other researchers thought. The issue was not life expectancy- it was researchers' opinions about how much fat people could enjoy life.

7. Understand the definitions of the people or diagnoses being studied.
I'm sure all of my readers can understand what happens when rule 7 is not followed- you get people talking to you about a diabetes that isn't the one you or your kid has.
When you know who was included in a study, you have a better idea of who it is relevant to, and what it means.

8. Understand sample bias. When looking at any study, it helps to understand how certain people came to be participating in the study. People included in a study on "natural medicine" are likely to be people who like the idea of natural medicine.

9. Sometimes people lie.
Mostly in looking at medical research, this is (hopefully) an accident where the conclusion doesn't really match the data. Sometimes however, doctors and scientists have been known to falsify data. In case studies, there is often a possibility that things didn't go exactly the way the patient or parent says they did. And when doctors or other people administer surveys with multiple choice questions or even open ended questions, there is a HUGE potential for misunderstanding.

10. Nobody is average about everything.
If a study shows that a medications lasts for 26 days in the average subject, that doesn't mean that it can't last 96 days in you. Or 2 days, for that matter.

11. Nothing affects just one thing.

12. If A and B happen in a given relationship every time, your researcher will often assume A causes B. The logical possibilities however are:
A causes B
B causes A
C causes A and B
the researcher didn't really test enough to be sure about every time and it was chance

*Chevrusa is a Hebrew word that refers to a study partner- typically a study partner with whom one studies a religious topic.