Bending the rules a bit on who meets the quota should never be seen as a ‘technical’ breach of any research code of conduct, no matter what pressure we find we are under to meet the target. Karen Forcade’s admission that she passed off research findings in the USA that had been manipulated to meet the client’s sampling criteria (see “Youth Research boss pleads guilty to federal fraud charges”), and in some cases, simply fabricated interviews, is all the more reprehensible because the study in question was for child safety relating to lighters. Forcade, along with another researcher at this now-defunct agency are due to be sentenced: it could mean jail for them.
It’s an extreme case, but it serves to remind us that our combined labours are about delivering the truth. Often, what emerges from our inquiries is indistinct and contradictory and getting to the truth may legitimately involve editing both data and findings to bring it out. We also know that some respondents provide false data – and it is not a problem that only afflicts online surveys. Discretion is called for in choosing what to edit and how to edit it, and wherever discretion and professional judgement enters a process, so too does the potential for abuse. Respondents fake interviews because they’ve been promised trinkets and tokens. Forcade faked interviews because her firm gained around $15,000 for each safety study they did: higher stakes and also greater consequences, though quite why she did this, is still hard to comprehend.
Yet most errors are more mundane in origin. From my own observations, and conversations I’ve had with those who work with data on the technical side, data quality is an issue that large areas of the industry have become complacent about. Execs and project directors look far less now at actual respondent data than they used to. And while eyeballing the data will only uncover some problems: error detection is really best done using automated methods. Yet few research firms seem to be putting effort into innovating in this area. Manual processes for error checking seem to abound, focused on checking tables while other parts of the research process that will introduce error (scripting, editing, data processing, coding and report preparation) are largely left to their own devices.
Yet every time I’ve been involved in the migration of a tracker from one supplier or one technology to another, errors have emerged where published findings have been found retrospectively to be just plain wrong. Only yesterday, I was talking with a technology provider about automated error detection, and we both found we had several anecdotes that illustrated this very situation. In one case, it was simply that the labels for one regional report had been inverted – the more the manager tried to get his low satisfaction score up, the lower the scores went. He was about to lose his job when someone spotted the error.It seems he’d actually been doing rather well.
Research does not need to be wilfully misrepresentative to do real damage to lives and reputations.
I’m curious to know if others have observed this problem too, or how they police their data.
This entry was posted in Research Industry and tagged data quality, error, fraud.
You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
According to a recent white paper produced by Oracle’s BI Consulting Group (“The Great Debate: buy versus build”) it costs between two-and-a-half and three-and-a-half times as much to build your own business intelligence and analytics system than buy and adapt an off-the-shelf solution. It must be admitted, the white paper concerned is a marketing piece to support this firm’s services in – you guessed it – adapting standard Oracle systems to provide BI solutions. But their figures are based on experiences with some 250 large-scale implementations, which is a decent number to conjure with.
They also found that such projects when built from scratch by system developers averaged 34 weeks from start to finish and consumed 925 person days, whereas implementing then adapting a generic platform took half the time – 17 weeks – and less than a third of the effort, with an average of 290 days required. When you consider the amount of effort involved in managing a project twice as long, and the drain that this has internally on the stakeholders involved with prototyping and early adoption, there are also likely to be sizable hidden internal costs that can be avoided. MR applications may differ in some respects, and the development projects around them may be smaller in scale, but the experience is unlikely to be radically different from this scenario.
There is a surprisingly large amount of custom-built MR software around, developed in-house from scratch. It’s not all legacy stuff either. I continue to hear of firms developing their own tools across the range of research applications, from panel management through to dashboards and portals. As the article from Oracle points out, their experience is that the out-of-the-box solution tends to meet between 70% and 80% of needs, so development effort is concentrated instead on the 20% to 30% that needs to be accommodated.
Of course, it is always possible that, lurking in that 20-30% are some very nasty problems that require almost infinite resource to resolve, but that remains true whatever route you take. Perhaps decision-makers are not comparing like-with-like when opting for own-grown. An off-the-shelf solution has known limitations – these limits are usually cited as reason for rejection. Despite this, they are limitations that may often be overcome through customisation. Open systems now make it readily feasible to customise rather than bribe your software vendor into changing their core software product or live with the limitations. Yet it is a choice that is often neglected.
The build-your-own route is seductive – theoretically, it appears to have no limits in what is achievable. The danger lurks in those obscure user requirements where effort starts to spiral towards infinity in achieving them, and where hard decisions are needed – always assuming you’ve noticed in time.
If you can find something where 70% of the work was already done without having to think about it, surely that’s a good thing? Yet, “it only met 70% of our requirements” is often the siren call that lures project teams into the unfathomed depths of what astute system developers call “the vanity project”. It’s flattering to be told your requirements are so special that a system must be built from scratch to meet them. It’s also very very unlikely. But to counter vanity with vanity, who likes finding out they paid more than three times over the odds?
This entry was posted in meaning eye and tagged .
You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
We will only know whether we were at the turning point of an L, a W or a U when looking back. Nevertheless, my own unscientific poll of firms I’ve spoken to in the past few weeks confirms that, although the recession has been acutely felt by MR technology providers, things seem to have been looking up slightly since people got back to work after the summer. Some tech firms have been busy, even very busy, and some have continued to grow despite the downturn. Being inexpensive or on a short track to adoption seems to help here. Another factor seems to be the needs-led solution: an agency client needs a custom panel, a web-based analysis tool, a dashboard – reactive rather than strategic purchases.
At the same time, others have been putting on a brave face, weathering out the storm and continuing to develop their products. Exhibition organisers are going to have a tough of it time next year. Tech providers are wincing at the costs of going to the big shows at a time when very few are buying. For one, just the charge for electricity levied by the venue was sufficient to wipe out all profit.
Even the firms that have remained busy are reporting that it is taking much longer to close the deal. People will talk for 18 months or longer about a £25K order but it never seems to materialise. Others find the orders they do land have been scaled back considerably from what they were asked to bid for.
It makes me feel that MR firms are still not approaching their technology from a strategic point of view. As I reported in June, several research companies at CASRO Tech were seeing a slowdown in work as being the opportunity they needed to get their processes and tools in order for when life got busy again. Is this opportunity being squandered?
There are many tales out there of firms never coming to a decision, seeing almost everyone and rejecting them all, or having virtually an annual review and still sticking with the same set of ageing or complicated tools that require high levels of skill and effort to operate them. This is symptomatic of technology decisions being delegated down to those in the organisation who are perceived to understand them: unfortunately it is often those who have the greatest investment in being indispensable masters of the dark arts who hold most sway. Perhaps they did reach the right decision, though I’m often unconvinced. But even if they did, I’m often unconvinced it’s for the right reasons.
This entry was posted in Research Industry, meaning eye and tagged automation, recession.
You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
