Last week, the US Centers for Disease Control posted an interesting article (in advance of publication in Emerging Infectious Diseases) on attitudes to antibiotic resistance amongst primary care providers. It is full of good ideas and it might even reflect what US providers think. As a basis for public policy, though, it is dangerously inadequate. Sadly, we see organisations in the public and private sectors making the same kinds of errors all the time. They fail to understand the strengths and limitations of qualitative market research and a proper process for integrating secondary, qualitative and quantitative research

This paper is actually among the better ones. The methodology is quite good and the paper does not make exaggerated claims but it does skip over vital caveats. First, the whole study is based on 36 interviews of which 27 were with doctors (and nine were with prescribing paramedics working in primary care). In 2010, there were over 200,000 MDs alone working actively in primary care in the US. It is a far from homogenous group: about 85,000 were family practitioners, about 90,000 were internists and about 11,000 were paediatricians. This is the US, of course, so Doctors of Osteopathy also prescribe (and they love prescribing antibiotics — although that is another story) and there are about 60,000 of them. There are probably another 50,000 prescribing paramedics. There are no hard and fast sampling rules in quali research but you need to be very, very brave to interview 1 in every 10,000 of your global sample and pronounce on that basis.

You need to be even braver given that you are almost guaranteed to have built in sampling errors when you talk to doctors. These researchers did the right thing: they did not identify their client before the interview (imagine the impact of knowing you might influence Federal policy?) and they paid an incentive. However, most doctors are inaccessible to researchers even under these circumstances. They are just too busy or too well insulated to do the screening questions or to set aside an hour (even a compensated hour) for an interview. You end up talking to the doctors who have nothing better to do or who have a slightly abnormal desire to tell the world what they think. The authors do note that, “…the clinicians who were screened, selected for participation, and agreed to be interviewed may hold stronger opinions about this topic than those who were excluded or declined to participate.”

One way to be aware of sampling errors is to check whether the doctors you recruited look like the global sample: are they roughly the same age distribution and the same racial and gender distributions as the MDs, DOs and paramedics working in the US as a whole? There is no indication that these researchers even looked (interviews were instructed to recruit “a mix”). Click through to the technical appendix and you will find that among the 36 interviews, only one was with a hispanic doctor and only three were with an Asian doctor. This is nothing like the profile of doctors in the USA. You will also see that paediatricians were vastly over-represented and both GPs and IMs were under-represented (DOs are not included at all for reasons that are never explained)

Quali research is a great way to uncover ideas and trends. It is also a very good way to talk about issues that are too complex to be included in multiple choice or “yes / no” answers. However, it never gives results that are statistically valid so researchers should be very careful when they write things like this: “PCPs commonly perceived that patients expect an antibiotic for clinical visits, contributing to a shared feeling of pressure among PCPs to satisfy patients.” I know that they meant the 36 they interviewed but it is an invitation to be misunderstood. As the authors write, “although in-depth interviews are an effective method to explore individual providers’ KAPs, we cannot generalize our findings to the PCP population as a whole because of the lack of external validity inherent in this type of qualitative research.”

The right way to do things is to uncover the ideas and the trends in quali research and to check how a valid sample responds to these hypotheses in well-planned quantitative studies. With a good researcher, you might feel confident enough to skip this step if you are doing your own policy planning or estimating demand for a product. We really hope that the Federal government is not thinking about that kind of a shortcut in policy making.

get smart