The "bad apples" debate

It’s amazing how history repeats itself. The March issue of Research magazine featured a cover story on what it termed ‘Bad Apples’, the fake and/or serial respondents who crop up in the course of qualitative work.

A search through AQR’s archives will reveal that this is an old chestnut, but one that has never been satisfactorily resolved. That’s why, following the Research article, the topic was put up for discussion on our website.

Members were asked to consider four central themes. First, is a central database thought to be the answer, combined with identity checks? If so, who should bear the cost of its creation and implementation? Secondly, is the root cause linked to the speed with which research and field agencies are asked to recruit candidate? And if this is the case, is the situation ever going to get any better? Thirdly, how much is the case of the ever-more-niche respondent exacerbating the issue, and finally, is there a dearth of discussion between clients and agencies in relation to recruitment briefs and the recruitment process?

One month on and, at the time of writing, there have been 25 responses to date with another seven letters on the topic featured in the April issue of Research. And if you still haven’t added your comment to the AQR website, it’s not too late.

The majority thought fraudulent respondents were the biggest problem, but adopted a more relaxed attitude towards those who come under the ‘repeat’ heading (see site for further details). The points raised included:

So where does the debate go from here?

At the April AQR Committee meeting the issue was thrashed over once more and it was agreed that it did need resolution.

The Committee decided to formulate a set of ‘best practice’ guidelines, endorsed and hopefully written by both clients and researchers, which will offer advice on how to deal with respondents — and recruiters — who aren’t perhaps participating as they should.

This will focus on problems that can occur with the original selection of respondents, maybe recommending random spot checks, to those that occur at the groups in question, even if this boils down to the researchers initiating post-mortems with recruiters the following day or shortly thereafter.

“We don’t live in a perfect world,” said AQR chair Fiona Jack. “Sometimes the client’s specifications are difficult to adhere to. There is always an element of time pressure, but we do need to try and create a shared understanding of limitations, manage people’s expectations better, and for everybody to make it their business to raise the bar.”

There is no question that the introduction of national ID cards, if and when it happens, will make a difference. Respondents will then be able to be checked when they arrive at groups to make sure they are who they say they are. But this won’t resolve all problems. What is to stop participants finding a way round the system? Or exaggerating in groups?

Human nature

As the Committee pointed out, it is human nature to behave differently according to your surroundings. A focus group is no exception, so all those participating — from the client, to the moderator and the respondents — will tend to present themselves in a different light to normal. In other words, there is no norm.

No matter what the original article in Research magazine would have us believe, ‘bad apples’ are spotted for the most part by experienced researchers because they stick out from the rest. They may not be ‘outed’ during the group, but their contribution will be viewed in a different light.

It is not the function of AQR to build a database of these ‘bad apples’. Cost is not the issue so much as it is not the Association’s remit to police this area.

Which is why the recruitment guidelines are being scanned, to see if there is any way that they could be tightened up, while the Committee sets out to write the new ‘best practice’ guidelines. These, when they are ready, will appear on the AQR web site and, in future, will be circulated with the guidelines.

A common aim

Will it be enough? Well, if it achieves a dialogue between researchers and clients as to the problems, and their causes, it will have moved this whole debate to a higher level. Those on both sides of the fence do, after all, want to achieve the same aim: answers, insights and understanding.

Did it need an article in Research magazine to set the ball rolling? If the AQR archives are anything to go by, it was time for this issue to be aired again and just happenstance that it happened this way. The challenge for the Committee — and AQR members as a whole — is to go some way towards resolving it at long last so that we don’t, in 10 years’ time, go through the whole process all over again.