Industry taking a twin-track approach to social media research

Social media research continues to be one of the hottest topics in research. I’ve just been reviewing the abstracts for this year’s CASRO Technology Conference in New York in June, which I will be co-chairing, and of all the topics, its the one with the longest string of submissions. Not only that, but there is some diversity of opinion into what it is, how to do it, and whether it adds anything at all to the existing researchers’ toolkit. Closer to home, it’s a topic that will be debated in next week’s Research conference in London too.

Analysis technology used on social media research projects, based on the 17% of firms who are active in social media research

Social media research is also one of the new topics we focused on in our 2010 annual software survey, sponsored by Globalpark, the results of which are published today. There are some curious findings – and some predictable ones too – that add perspective to the current debate.

Our survey of over 200 research companies of all size around the world, shows social media research is still at the early-adopter stage,  accounting for revenue-generating activity in just 17% of the firms surveyed. Close to the same number – 19% – say they are unlikely to offer social media research, and of the remaining 63% who gave an answer, 31% say they are either experimenting with it and 32% are considering it for the future. Small firms and research companies in Europe are the least likely to be doing social media research and are also the most likely to have ruled it out, whereas large firms are the ones that are most active. The actual volumes of work are still low – we also asked how much revenue social media research accounted for. It is 5% or under for  two-thirds of the agencies that do it and tails off beyond that – but there appear to be some specialists emerging, with a handful of firms deriving more than 20% of their income from it.

Many firms are bullish about the future, though, with 20% predicting strong growth, and a further 52% anticipating some growth, with North America, and again the larger firms, most optimistic about its future.

As a technologist, I was most interested to see what technology firms were applying to what is, after all, something born out of technology. Were the tech-savvy gaining the upper hand, or were researchers taking the conventional, low-tech approach beloved of qualitative researchers. Again, it’s a bit of both. Of all the software-based or statistical methods we suggested for data analysis, the one that came top, was “manual methods”, used by 57%. For analysis, this followed by 54% citing “text mining” (as correspondents could pick all that they used). Text mining, though it uses some computing power, is also very much a hands-on method – but it’s good to see more than half turning to this method. Other methods make much less of an appearance, and the method that I consider shows most promise for dealing with the deluge of data, machine learning-based text classification, was bottom of the list, cited by one in six practitioners.

For data collection, technology was much more apparent – although it is hard to avoid here. We were still intrigued by the massive 54% who say they are using manual methods to harvest their social media data from the web; 57% were using web technologies to collect the data, and the more exotic methods were also fairly abundant, including using bots (43%), crowdsourcing (41%) and avatars (24%).

I’ll pick up on some of the other intriguing findings from the study later. But as the report is out now, you can pick up your own copy by visiting this webpage – and there will be a full report in the May issue of Research magazine.

Research firms reveal nine month technology lag (and a secret affair with the Mac)

It’s not the first time I’ve postulated that MR firms can be laggards with their technology. An interesting early finding to emerge from the 2010 Globalpark MR Software Survey, a survey among research agencies worldwide carried out annually by meaning, provides some supporting evidence for this by looking at the actual technology being used to access the survey.

We’ve been able to analyse the browser string returned by the 550-some participants who responded to our survey invitations – which were targeted exclusively at MR companies across the globe. The browser string, which many MR survey packages capture automatically, reveals exactly what browser and also what operating system the survey participant used. It’s hard data, free from any response bias, as it is picked up from the routine chattering that goes on in the background between web server and web browser.

We thought it would be interesting to compare this with the current state of worldwide usage to see if MR differs, and if so, how. Overall, the figures are very close with respect to operating systems in use. We compared our figures with those from GlobalStats Stat Counter who measure usage in the same way, only on a somewhat grander scale: typically 15 billion page impressions from 3 million websites per month across the world.

The headline figure for Windows, at 93.3% among our MR participants, against the GlobalStat’s worldwide figure of 92.0% is within a whisker. Perhaps surprisingly too is the 5.9% figure for Mac OS X usage – a squeak away from the global figure of of 6.2%. Most MR software providers produce only Windows versions of their software, and even web-based software, such survey authoring tools or online analysis programs, which could be platform independent, are often locked down to Windows-only browsers.

Chart showing OS usage 2009 to 2010

The trend, which we can see by comparing browser string data from our 2010 survey with the equivalent from our 2009 survey, shows Mac usage has surged from 2% to virtually 6% in the last year. We cannot tell how many of these users were taking the survey at home on their own Macs (which is quite likely) but it’s clear they were responding to a work-oriented email on a platform that most MR software managers choose to ignore. If this is a continuing trend, this minority will be increasingly hard for software developers to dismiss as undeserving of their attention.

We can also see some interesting things happening with the flavours of Windows being used. It is well known that most corporate IT departments gave Windows Vista a miss when it came out in 2007, choosing to stick with the the ageing but more reliable XP. Even now, as XP approaches its tenth birthday, it is to be found on 70% of the PCs in our study that have Windows installed on them (65% overall). Among our 2009 participants, it was found on 74% of the Windows PCs. In the meantime, what little share Vista had accumulated has now mostly been ceded to the newest Windows 7, which launched in October last year, just before our fieldwork period. It has now grabbed nearly a quarter (23% among Windows PCs, 21% overall).

Mobile devices barely made an appearance – less than 1% of our sample. There again, we’d expect most researchers to realise that taking our survey on a handheld device was unlikely to be a joyous experience. The same is not true for iPad users. We tested it on iPad and it looked good: but only two showed up among 554 research professionals who clicked the invitation link.

Chart showing web browser usage in MR firms vs global usage

Browser usage was also remarkably close to the global figures overall. Between last year and this, the major trends are that Internet Explorer has been losing share to both Firefox and Google’s own Chrome browsers – and our MR sample show signs of the same trend. However, when we compared the trend with the global data, our November 2010 sample showed a remarkably close fit with what the rest of the world looked like around March and April this year. Worldwide, Internet Explorer dipped below the 50% mark in August and by November it had eased down further to 48%. Our November 2010 snapshot shows IE as the browser of choice among 54% of research users, down from 62% last year, while Firefox has gained strongly, and even Chrome and Safari have picked up users.

We’ve just started to look at the actual questions in the 2010 MR Software survey – we will have the results out in March 2011.

Online samples – where lighting a few fires might help

flame on a struck matchIn Greek mythology, it was Prometheus who stole the secret of fire from Zeus and gave it to humankind. The new non-profit organisation being launched by rival research giants GfK and Kantar to address industry-wide concerns about the online survey quality, seems to make a nod to the myth in its chosen name, the Promedius Group.

The industry’s concerns about online research are many and various, but a common complaint is the lack of transparency of sample providers in the composition of their samples and the extent to which these overlap. It’s worrying enough that, as response rates dwindle, research firms are probably already relying on less than 20% of the population to answer more than 80% of their surveys. But what if it is the hyperactive 0.1% of the population that turn out to be answering 10%, or 20%, as some fear, turning survey data into junk? Without the vantage point of the gods, no-one can really tell what is happening.

Good research is always a balance of art, craft and science. The risk is that if survey results are no longer generally reproduceable, any claims to scientific validity are lost. Those that spend the big money in research, like General Mills and P&G, have noticed this, and are highly likely to start putting their money in consumer intelligence gathering elsewhere unless research can be made more accountable again.

The solution is staring at us from the problem. There is a vast trail of data that can be associated with every web-based interaction – put it all together and it becomes possible to pinpoint individuals and identify, within reasonable probabilities, that they do seem to be taking 20 surveys a day, or that they are very unlikely to be a gynaecologist because the digital wake emanating from the same PC speaks more of college student. Getting at this data, however, is much more difficult. If you are a big player, with a large volume of interactions, you can do this – but even the industry’s own demi-gods face a major hindrance, in that most of the panel providers don’t reveal the key information you need to start putting this information together, like respondent IDs or IP addresses.

Promedius will, it appears, be making use of a lot of technology to match data and perform checks on data, and they will be making this technology available for other research companies to use. This is welcome news, as the problem has been proving too big for anyone to solve on their own. There are already commercial services – MarketTools’ Truesample and Peanut Lab’s Optimus to name two – and these have gained some traction. They also add cost, and are restricted to some extent by only ever showing part of the picture – from those samples and providers that have opted in.

With three major players backing this initiative (IPSOS were involved in the development of the technology behind Promedius) it is likely that it will have the critical mass that is needed for it to become established. What the technology does, and how affordable and convenient that is (the announcements do not say that this will be offered to the industry for free) remains to be seen. I’ll be looking to secure a software review as soon as it becomes available. But there is a good chance that Promedius will be putting fire into the hands of the people, as far as panel and online survey transparency is concerned.

Hopefully Promedius will enjoy a better fate than its near namesake, who after several other skirmishes, found himself bound to a rock by the vengeful Zeus, with an eagle visiting him every day into eternity to peck away his liver.