Targeting Mr X:
but is he Mr Right?

ASC Conference - May 2002: report by Tim Macer

Bravely, the Association for Survey Computing tackled the twin subjects of sampling and weighting in its recent one-day conference, “Targeting Mr X: but is he Mr Right?” It is a topic that many would file under ‘worthy’ rather than ‘fascinating’ and the same was true of much of the day. As always, a few highlights gave a long day the perspective of being shorter, when looking back.
If you have problems understanding what weighting does to your data, then you are probably in good company, according to Paul Smith from ONS. Non-response is having a much more damaging effect on surveys than simply making it harder to get the numbers. Only including respondents who are “happy to participate and good at interviews,” he stated, plays havoc with representation and means interesting findings can be missed.
Smith sees a remedy in differential sampling, to boost groups less likely to respond, such as ethnic minorities. He also advocates ratio estimation or regression-based weighting to correct for under-representation, but advised caution.
Research that Smith has carried out confirms most researchers’ suspicions that a moderate amount of weighting will improve the quality of the data, but weighting too aggressively does the complete opposite. There is no handy statistic to identify the
optimum level of weighting, so the best Smith could recommend was to check indicator variables and “don’t over-do it!”
Tony Dent (Sample Answers), presenting a paper “Probably the best sample you can get”, was critical of the purists that were wedded to increasingly irrelevant sampling methods and who dismiss database samples as being unrepresentative.
“All methods suffer the inherent bias of the willing respondent and this is the real bias we face all the time” he said. “If you understand the bias you have, you can get results as good as from the most statistically pure (and expensive) methods.” Dent reminded his audience that sample is the response achieved, not selected.
While market researchers are preoccupied with sampling, they appear to have overlooked the value of working with and developing GIS (geodemographic information systems) data to meet MR’s needs, according to two different speakers.
Combining MR data with GIS data allowed Plato Marketing Solutions to create a dynamic support system for the Post Office to decide which branches it should develop and which it should close.
Blair Freebairn, from Plato, presenting the case study, observed: “Demand elasticity is typically seen as a function of price. But you get a similar effect with accessibility. There is a lot of interesting work to do on how the two are bizarrely related.”
Freebairn commented that GIS, which had originated within the MR industry, had gone its own way to the extent it was now a foreign territory to most market researchers. “To get most out of it, you need to speak two different languages,” he said.
He also commented on the fallacy that respondents can recall spatial information accurately in surveys. Observations such as ‘closest’ and ‘too far away’ should be tested against location data.
“When looking at research data, we need to look at spatial factors too – it offers a quick win,” he stated, describing how perceived distance can often reveal more about affinity and loyalty when compared to the actual distance on the ground.
James Debenham, John Stillwell and Graham Clarke (Leeds University) have developed a new GIS model that attempts to make the method both more dynamic and offer a better predictor of consumer behaviour by including so-called supplier-side drivers.
Overall, the day had the feel of not enough happening on the technology front to solve today’s problems of response and representatives. The revelation that weighting, unchecked, can easily do more harm than good calls for urgent attention before this becomes a panacea for falling response.

ASC: www.asc.org.uk

Tim Macer's website is at www.meaning.uk.com

Published in Research, the magazine of the Market Research Society, June2002, Issue 433.

© Copyright Tim Macer/Market Research Society 2002. All rights reserved. Reproduced with permission.

top of page | Back to list of reviews