I read this very interesting post by Edward Appleton about the authenticity of peoples online behaviour published on the Greenbook blog. http://www.greenbookblog.org/2012/02/01/how-authentic-are-we-online/
The authenticity of online respondents is a very interesting philosophical question and is an area we have been looking at recently too - albeit in perhaps in a more literal way...
We have been examining ‘honesty’ within the online survey environment, by comparing answers to questions where we hold known norm figures about behavioural activity and building up a picture of levels of honest answering to different types of questions.
We have found that in the main most people in western* markets who respond to surveys are, on the whole, very honest in the answers they give.
Rolled up into averages, about 95% give "honest" answers to the average question - so it is not something you need to worry too much about for a typical survey.
But that figure of 95% is for the average question. This figure can vary dramatically and we have witnessed questions where up to 30% or respondents show signs of untruthfulness based upon the context of how you ask the question and the type of questions being answered and who you are asking.
The reasons for this, though, are complicated. We have identified a range of driving factors that determine the level of authenticity to an answer we may give, these are:
1. The Peacock syndrome: We like to show off and make others think we are smarter, wealthier, happier, better than other people. You can see this issue in effect on magazine readership studies where, for example, the claimed number of readers per copy of Vogue magazine is 12 - about four times higher than the average magazine. The number of people who claim to shop at Harrods the premium London on the TGI survey is about 2 times higher than the estimated footfall*. So you can never really trust stated figures about any claimed behavioural activity if it is socially desirable.
You must also remember that this process works in reverse. So as an example, people admitting to watching soap operas, smoking or noticing advertising, all things that we don't like to admit to doing, are all downgraded by us naturally.
*These figures themselves may not be completely truthful, they are plucked from my memory which is a well known source of untruthfulness if I am to be brutally honest!
2. Pleasers: This is perhaps more of a cultural issue. , Some people like to answer in certain ways to please you. Now, this is different to the peacock syndrome and it effects different types of questions, those that are less commercial questions and more social questions such as political opinions as an example. We are often very reticent to give our real views on certain topics for fear of perhaps offending. Or we may worry that our views are not socially acceptable. Or even in the extreme case of say, the fear of persecution if you were conducting research in an oppressive political regime.
3. Gamers: Now these are a set of people who adapt their answers to play the system, which is a big problem particularly in certain countries and for certain types of topic. For example, if I am trying to sample top end business people and I am offering a high incentive to take part then there will be those who will try to pretend to be business people so that they can qualify to participate in the surveys. But this is not just limited to a small group of fraudulent respondents. A lot of respondents start gaming the system when they simply get bored. They stop saying they do things or buy things to avoid having to answer extra follow-on questions.
4 Disengagement: This leads onto the last group which is the disengaged respondents who don't answer truthfully because they can't be bothered to think about it. It manifests itself on lots of different levels. Ad recall can vary by upwards of 100% between engaged and disengage respondent groups.
5. We lie to ourselves: We are all naturally quite self delusional. The image we have of ourselves photographically, for example, is often nothing like the image that other people see. So if you asked me to pick a picture which is a good likeness of yourself nine times out of ten it will be a different picture to what someone else will pick. It results in peacock answering but also a lot of delusional responses to social behaviour questions e.g. are you the sort of person who would recycle? We all like to think we do and a lot of us believe we do, but it is something we don't actually do in practice. Here is a link to a wonderful paper exploring this issue http://www.tandfonline.com/doi/pdf/10.1080/15298868.2011.617886.
6. The power of the unconscious: So much of what we do is controlled by unconscious decision making paths. The conscious v unconscious mind are often in conflict. We, as a result, often make irrational decisions and so we are not very good observers of aspects of our behaviour which is controlled by the unconscious. As a result we are hopeless predictors of certain things we do. Obviously this issue sits at the heart of behavioural economics.
7. General Ignorance: We simply don't know or don't remember answers to certain often very clearly obvious questions: do I have brown eyes? I don't know. Where do I live in the US - 10% of people don't know the answer to this, particularly if you show them a map of where they live (to the extent we had to stop using maps for this question in our surveys!). We also are unable to predict our future behaviour because we don't really know what we will do. , We are, in fact, particularly bad at this. You may as well pluck a random number out of your head instead of asking somebody if they are likely to buy a new product in the future or not as it is likely to be just as accurate.
Cross cultural differences in truthfulness
I have starred *Western markets earlier on because in non western markets the level of "honest" answering can drop significantly down to, in some cases, around 70%. In fact in some countries you can forget any other worry about panel quality and demographic sample biases. The level of honesty is the Number One issue that you need to contend with. The variance in levels of truthfulness is compounded by factors such as income. If the incentive to take part in a survey is relatively higher in one country v another you will see higher numbers of people motivated to be gamers for example.
The big questions
The big questions are to what extent can these relative factor be quantified? What impact do these things have relatively?, and what tools and techniques do we have to counter them?
This is something our Chief Scientist at GMI, Mitch Eggers has been extensively exploring over the last year and I hope we will be able to publish a paper on the topic at some stage in the near future.
No comments:
Post a Comment