How good really is your survey data? Are your decisions and recommendations based on actual truths? Or rather messy and inaccurate data?
The Problem with Survey Data:
We would all like to believe that we will remember things as we get older. But reality is that any given day a person is doing a million different things. These may include getting ready, preparing, or eating meal, having conversations, texting, browsing online, playing games, reading the news, transporting kids and having random conversations with them about the day, and probably more. There are so many things that it is no wonder one person can accurately remember everything they did. Some of these actions are more meaningful and can be recalled, while others are deemed less significant and are likely put deep into our subconscious where it makes it hard to remember the exact details or that you even did that.
In order to understand how to market to consumers and build better products, brands often ask consumers to recall things like ‘how often do you do this’, ‘how much time do you spend’, ‘where are you spending your time’, ‘how much did you spend’, or ‘did you see this advertisement’. This form of ask/answer has been the main form of insight collection for our industry for the past 100+ years. And the goal remains the same – to understand human behavior and to create better products that meet the needs of consumers. Our industry has evolved and improved the way we ask questions over time, however, much of how questions are asked has not changed and many surveys that are fielded are very poorly written.
And if you read some of these surveys, or try to take your own survey, they are not all good. Our industry is putting out questions that are leading, that are hard to answer, and sometimes that just does not make any sense.
Additionally, the industry is at a tipping point. Survey fraud and data quality are out of control. We hear countless people in the industry sharing the percentage of fraud and bad data they are removing on every survey – these numbers range from 30-80% depending on where the data is being collected.
But there are good questions, and there are good respondents who are ultimately trying to be honest and give accurate answers. But how accurate is accurate? How much of what honest people say is true?
So, we set out to compare actual consumer behavior versus what consumers recall was their behavior and found some disturbing outcomes.
How we Measured the Survey Recall Issue:
Qrious Insight specializes in behavioral data and has a network of over 100K consumers in the US. This digital behavioral data is collected from individuals who have opted in to share their data and in exchange are incentivized. In a world where everyone is taking your data for free, the idea of a fair exchange is very motivating.
The digital data collected includes:
- The apps people are using
- The websites they are visiting
- The searches they are making
- The products they are viewing, shopping, and purchasing in-app on Walmart, Target, Amazon
- The social media ads they are exposed to on YouTube, TikTok, Facebook, Snapchat, Instagram, and X
- The media content they are viewing on device from Netflix, Disney+, YouTube, Amazon Prime
This rich, deterministic behavioral data set reveals what people are doing without the need for recall. It is not modeled, or inferred, it is real actual behaviors from consumers. Additionally, because we see everything people are doing, we have fraud detection beyond traditional fingerprinting that identifies if people are using survey bot software or searching for how to cheat a survey. These people are removed from our panel immediately.
We paired our behavioral data from consumers to a traditional survey where we asked people to recall engagement on their device from the past week. The survey was short (less than 5 minutes) and included a variety of question types across many categories. We started the survey by telling people there are no right or wrong answers. That everyone qualifies. And that we are looking for them to just tell us what they did last week.
The survey data was then compared to the behavioral data for each individual to measure the say-do gap across questions.
The Say-Do Gap Results:
As researchers we strive to be accurate with our findings. We build sample plans and come up with sample sizes to minimize our margin of error and be able to make recommendations confidently. But even after all that cleaning, how confident should we be in the answers we are getting back from surveys?
We started by asking people which of the following things they did on their device last week. This list included things like email, social media, streaming/watching shows, reading news, exploring travel options/using travel apps, managing their finances/banking/investments, browsing the Internet, watching YouTube videos, communicating with others (via text, calls, video apps, etc.), listening to music/podcasts, and playing mobile games.
Comparing these same individuals survey data to their behavioral data, we found that the average gap was 33% – meaning that only 67% of stated survey responses to a simple yes/no question was correct.
And this gap, varied widely by the tasks asked about. In fact, the worst observed recall was about finances. 51% of people answered the finance question incorrectly. 43% failed to accurately recall that they had done this, when behaviors showed they actually had. And, 8% overstated telling us they had done it, when there was no digital record of it.
The category with the best recall was Travel. In this case 76% of responses were accurate, which still left a say-do gap of 24%. Where 12% did not remember engaging with travel sites or apps, and 12% told us they did, but there was no digital behavior.
“we found that the average gap was 33% – meaning that only 67% of stated survey responses to a simple yes/no question was correct”
When we think about broad categories like this. There are likely many things contributing to a person answering the question incorrectly. This could be…
- It is something they typically do, and for some reason did not do it last week, but still in their mind thought they did.
- The task was so out of the norm and not super significant to them, that they just forgot.
- That they did not interpret the question the way we meant it to be interpreted.
And likely there are other scenarios as well. But regardless, is 76% accuracy, ok? How about 49% accuracy like finances?
We also asked people to recall which apps they used on their device in the past week. This list included Snapchat, Facebook, Amazon, Walmart, YouTube, Walgreens, Email, Internet Browsers, and TikTok. The gap here ranges from 39% to 13%. Where Amazon has the largest gap of 39% incorrect answers. 27% are saying they engaged with Amazon and did not, likely because it is such a common thing, that they just figured they did. And Walgreens has the smallest gap of 13%. Meaning 87% of people can accurately recall if they engaged with the Walgreens app last week. When you break this down though, 84% of people did not use the app and accurately recalled that.
We then asked people to recall from memory how much time they spent on their device each day over the past week. Looking at what they told us versus what was observed in their digital behavior, we found 90% of people’s time did not match. This was giving them a +/- 4 hour window, and still 90% off people’s answers were wildly different from what they actually did.
So, flip this around for a minute, if you are asking people about specifics in a survey, you might only be obtaining 10% of correct answers. And yet, researchers are making critical decisions from inaccurate data like this.
And what does this tell us about recall and accuracy?
If someone never does something they will likely answer it correctly. But if someone is doing it at some frequency, it might be hard to recall when they did it. So, asking absolutes like ‘have you ever’ can produce fairly accurate results. But anything else presents a level of error that researchers must be aware of.
And this level of error does not mean that these individuals are bad people, or untruthful. It just means that they are human. They have a million things going on and being able to give detailed information is tough, even with the best intentions.
How You Can Improve Insights and Take Action on True Data:
While traditional survey methods can provide a glimpse into consumers’ behaviors, it simply is not the most accurate way to understand accurate behaviors. I have said it before, but at this point, it is not really a choice. It is our responsibility as an industry to stop asking consumers to recall behaviors that we know they can’t articulate. Now behavioral data is a great solution, but it is not always a replacement for surveys, but rather a complement.
Our recommendations are to let surveys do what they are good at – collect attitudes, perceptions, and opinions/reactions. Do not use surveys to ask about behaviors, especially specifics. This means attitude and usage surveys asking about specific details, segmentation surveys digging into behaviors, consumer journey surveys asking people to recall all touchpoints and the order in which they occur. If we are asking these in a survey, we probably should not make recommendations off them.
Behavioral data exists in many forms today. The digital behavior is just one type. There are forms of usage being passively collected through technology like QualSights, or apps like Peloton, FitBit, and more. Insurance companies even have behavioral tracking programs that track your driving and in-exchange give you a discount on your insurance.
At Qrious, we are using behavioral data to ask less, and know more about people. We are keeping surveys short and to the point but providing more data and information on people. We are accurately measuring consumer journeys and advertising effectiveness. We are understanding who people are, their media habits, their hobbies, their passions. We know what people are doing and when they are doing it. We know where a specific audience is spending time and can guide digital ad placement.
We truly believe behavioral insights are the way of the future. So just remember…
Behavioral data is here. It is here. It is now, And, we should all be embracing it!
