|Aiming for near perfection in manufacturing processes is one thing, but can market research be made more perfect by applying industrial quality control models?
Demonstrably so, is the message from Honeywell's US market research team. After the firm successfully applied the Six-Sigma quality control evaluation model to achieve the near-perfect level of 99.9997% in the manufacturing of its industrial control hardware and software, it started to scrutinise its research processes in a similar fashion. In doing so, the company has forced a chain of its research suppliers, from software manufacturers to fieldwork agencies, to make some radical methodological and procedural changes that could have far-reaching implications for the research industry.
Even if you are a quantitative researcher, you may be forgiven for not recognising Honeywell's reliability figure: it is what six standard errors (or sigmas) look like - a level rarely applied in MR.
The Six-Sigma discipline, originated by Motorola and adopted by a wide range of large manufacturing corporations, works by applying a variety of rigorous methodological tools to identify sources of variability in any complex process as a first step to reducing the scope for error and thereby reducing overall cost through saving time, eliminating waste and inefficiency.
While Honeywells consultant researcher Bill Stone would not claim to have reached six sigmas of accuracy in his research, the journey is already proving to be worthwhile in cost reduction alone. He says: "We took a great gulp and said let's see if we can apply the Six-Sigma process to the survey process, generate some methods for improvement and really improve the quality of the data we are gathering."
Both CATI and web interviewing came under his early scrutiny. Both methods excelled in some areas but suffered from uncontrolled variability and "noise" in others. For example, the filtering effect of receptionists and secretaries when contacting respondents can eliminate those not only able but also willing to participate in Honeywell's predominantly B2B research.
Web was considered better for presenting complex ideas and getting a considered response, but the depth of data was lacking without the probing influence of an interviewer. Probing itself was investigated, and insensitive or inappropriate probing was revealed as having a significant negative impact on respondent cooperation.
As web interviewing has become more established, a large proportion of Honeywell's
Go with the flow: The automation of Honeywell's MR process
|research is being carried out in parallel, in CATI and on the web.
Investigations revealed that most mixed-mode surveys were incurring a double programming cost with few firms able to use the same software to deploy both, or even if they did, to use the same script.
"On average, with a fairly in-depth survey you will pay about $8,000 to $10,000 dollars to program a study for each mode; if you repeat this quarterly this means you are spending $80,000 not $40,000, and even for a large organization, this is a significant cost," says Stone.
Overall, Honeywell has reduced its research costs by 20% through improved efficiency identified in its Six-Sigma review. It has also seen improvements in the quality of data by focusing on the data collection process.
"We found that a number of the quality issues that were identified in Six-Sigma were met with web survey tool offerings, but there were also some genuine limitations," says Stone. "It made us start to look at some of the leading edge thinking on how you reintroduce interviewers into the process."
This encouraged Honeywell to switch to a fieldwork supplier using Humanvoice's SurveyGuardian product (reviewed in Research, March 2001), a so-called "e-interviewing" tool. SurveyGuardian is a wrap-around that sits alongside a web survey. Real interviewers communicate with respondents, online, through a chat window in the web browser used to complete the survey. Typically, an interviewer will monitor and assist up to three online respondents at once and eliminate many of the modal differences between CATI and web surveys.
Furthermore, interviews can migrate more easily from web to CATI or even be completed in a simultaneous mixture of the two. For example, a respondent hesitating, or repeatedly overtyping a box can trigger an alert to the "guardian" interviewer who can offer to assist; a box left empty can alert the interviewer to probe gently.
Over 90% of Honeywell's customer research in its Industrial Controls division is now conducted online using e-interviewers.
|"Our goal was to get better, more focused data from the people we need to hear from," says Stone. "We found that by being able to utilise these web tools, we were able to focus more precisely on some (though not all) customer segments, which gave us better data to put into our value models and understand the nuances of customer perception. Mixing phone and web methodology gives more breadth of coverage on some questions. Because of the improved comfort of the web for respondents - less pressure to respond, at their own pace and so on - this has allowed us to see a much higher quality of response on certain issues."
Applying Six-Sigma has caused Stone and the other researchers at Honeywell to "walk the process" and root out methodological inconsistencies. By exposing and explaining the research process, Stone has noticed an increase the confidence of a management that has been more comfortable with "hard" financial data placing trust in "soft" customer perception data.
Stone still has a long list of problems to be solved. He cites the difficulty in overcoming different database and file formats when attempting to implement realtime reporting. He is also critical of agencies for treating CATI and web as separate business units, as well as the agencies and software providers that have simply built web technology out from CATI. He is also concerned that the research industry lacks people experienced in the web, not just at the project director level, but he cites the difficulty they have had in finding web-literate CATI interviewers capable of handling both modes of interviewing.
Stone advises: "Field organization need to keep in mind that mixed methodology - more than web surveys - is going to be the need of the future and they must be able to accommodate that. I think there are some parts of the industry that have been quick to embrace the web but it seems the industry as a whole is still figuring out how do to it."
SurveyGuardian's website is at: www.humanvoice.com
Tim Macer writes as an independent specialist and advisor in software for market research. His website is at www.meaning.uk.com