“Evidence-based medicine” has the same sort of status amongst modern doctors that the flat earth theory had amongst enforcers of the mediaeval Inquisition: it is so sacred that you just cannot argue with it. Except, of course, that a lot of the fashionable meta analyses may be meaningless. Health journalists are too lazy, too ignorant or too scared of the journal backed Inquisitors to point this out.

As my university statistics professors would have told you, it would be a mistake to rely on my statistical analysis. Look, though, at the common sense behind two studies that have got a lot of media attention over the past fortnight.

Sleeping pills double your risk of death

A review of about 70,000 patient records in the British Medical Journal concluded that hypnotic and anti-anxiolytic agents — medicines to treat anxiety and insomnia — doubled the likelihood of a patient dying from any cause over a long period (all cause mortality). Frightening stuff and it was peer-reviewed in the BMJ so why would any journalist doubt it?

First, the common sense reason: over a third of the patients the researchers included had received not one but two “study drugs” (medicines looked at in this analysis): a benzodiazepine (usually diazepam) and a second medicine for sleeping disorders or anxiety (usually zopiclone, a short-acting hypnotic). Given the hysteria about side-effects of the drugs in UK general practice, it is a safe bet that these were very sick people. Indeed, read on and you will see that the researchers say that, “Patients who were prescribed study drugs were more likely than controls to be current smokers and to have higher rates of all forms of physical morbidity, most notably cancer and respiratory disorders. The group prescribed study drugs also had higher rates of sleep … anxiety … and other psychiatric disorders … than controls, and received more prescriptions for non-study drugs.” The meta analyses tries to control for this by matching the roughly 35,000 records of people prescribing study drugs with about 69,000 who were not receiving them but had similar diagnoses (the researchers also tried to use this mechanism to adjust for other variables such as socio-economic status) . Of course, though, a study like this ignores severity of illness and the more ill you are, the more likely you are to be treated.

The lead researcher, Dr. Scott Weich of the University of Warwick told the New York Times that the research “adds to an accumulating body of evidence that these drugs are dangerous … I prescribe these drugs, and they are difficult to come off. The less time you spend on them the better.” It was faithfully reported with only a quick reference to what Dr Weich and his colleagues said in their paper: “these findings remain prone to many forms of bias. While we have largely excluded immortal time bias and selection bias, we are unable to exclude the possibility that the results were due to confounding by indication or to residual confounding by unmeasured or incompletely measured factors, such as socioeconomic status. This applies especially to deaths in the first year of observation. These results add to evidence of an association with mortality, but must be treated with caution.” If the Old Gray Lady of newspapers covered the story with this little balance, you can imagine how trashier media covered things.

Any self-respecting journalist would probably have focused more on the caution and less on the quote if they had used an old editor’s trick: could you apply the same methodology to a different situation and come up with an absurd result? Almost certainly yes. Take a large group of people with Type 2 diabetes being treated with insulin and compare them with a smaller group of diabetes patients not receiving insulin. Both share a diagnosis. Adjust for socio-economic variables and co-morbidity and you will still, almost certainly, conclude that use of insulin is associated with earlier death. You have, of course, ignored the point that a patient only gets insulin once her diabetes has progressed to a more serious stage.

Eating ten vegetables a day adds years to your life

The same kind of mindless, ooh -aah journalism was on display with a study that purported to show that everyone should eat up to ten portions of fruit or, preferably, vegetables a day rather than the five that the World Health Organization recommends. This finding came from a primary analysis of data from about 65,000 people who took part in the Health Survey for England. It was published in another BMJ Journal, the Journal of Epidemiology and Community Health, Here’s how Sarah Knapton reported the findings in the Daily Telegraph, “A healthy diet should include 10 portions of fruit and vegetables a day, doubling the five-a-day official advice, health experts have said. The research, which involved a 12-year study, also found that vegetables were four times healthier than fruit. The study, by University College London, found that eating large quantities of fruit and vegetables significantly lowered the risk of premature death. People who ate at least seven portions of fruit and vegetables each day were 42 per cent less likely to die from any cause over the course of the study.”

This all looks less true when you … er … ask questions. Possibly Sarah missed that day at journalism college.

First, the study relied on what people said they ate rather than attempting to figure out what they actually ate. We know from many previous studies that many people lie (or convince themselves that their habits are healthier than they actually are). The paper itself notes this, “There may be social desirability bias and random error (forgetting) in the recall of fruit and vegetable consumption.” It is a good bet that those who are careful about their health are more likely to adjust their estimates upwards than those who couldn’t care less.

Second, only tiny numbers claimed to eat ten — or even seven — portions of fruit and veg a day. The paper reports that the mean number of daily portions of fruit and vegetables consumed by participants in the study were 3.8 (SD — standard deviation –2.6): 2.3 portions of fruit (SD 2.0) and 1.5 portions of vegetables (SD 1.2). I couldn’t find it in the paper but in an interview, one of the authors said that under one percent of the population studied had the ideal behaviour she was now recommending. Quite apart from being very careful about basing any findings on such a small number of lives and deaths, this is likely to be a highly unusual group of people. Do you know anyone who eats ten 80 gramme portions of fruit and vegetables a day? Have you even thought about what 800 grammes of vegetables looks like? I’m a vegan and even I don’t know anybody who eats that many vegetables. If they actually exist, these people are probably part of a tiny ultra-cautious group that avoid almost all risks in life — risks such as meeting people.

Flawed studies usually throw up junk findings and — sure enough — this study shows that people who ate frozen or tinned fruit were actually more likely to die. The authors say, “this may be due to confounding for example by (poor) access to fresh groceries in deprived areas or among people with pre-existing ill-health or a more hectic lifestyle. ” Hints that the methodology is really not very reliable.

The medical journalists may be confused or they may just understand that professional survival requires delivering increasingly sensational stories