When you have a pile of qualitative data, how do you sift through it to understand what users really value?
Ben Labay is the Managing Director at Speero (formerly CXL). His work involves managing experimentation programs for mid-market to large enterprise companies.
As part of our Wynter Games series, Ben sat down with us to make dealing with data a little less intimidating.
Why do you need qualitative data?
Many brands assume that competitors in their vertical are their biggest threat. In reality, your biggest threat is a poor customer experience.
Ben explained that everything today is all about the customer. Google isn’t collecting feedback within one product area - they are collecting everything.
Your biggest competition has become well-funded companies that create easy, enjoyable user experiences - setting the bar high for everyone else.
And the way to create pleasant user experiences, is through qualitative research.
“They're listening to their customers and solving their customers’ problems by collecting, voice of customer data, collecting qualitative feedback,” said Ben.
“The way that they're more directly your competitors is that they're creating these friction frictionless experiences. They're raising the stakes for all of us.”
User experience trumps everything
Today, user experience is more important than price, quality, and even your product itself.
Modern users who are accustomed to a great brand experience elsewhere (even in an entirely different vertical) simply won’t be happy with a poorly designed experience.
“Companies used to compete on features maybe 50 years ago, when with more competition, they competed on price,” Ben explained.
“We are now selling experiences. And the only way to measure and account for this is to get feedback and to get voice of customer data.”
In the age of data, measuring and benchmarking users’ experiences becomes a vital task.
Knowing the name of something is not the same as knowing something
Ben quoted physicist Richard Feynman as saying “Knowing the name of something is not the same as knowing something.”
This quote inspires the importance of creating hypotheses and conducting research.
When you apply this mindset to the popular Jobs to be Done framework for customer research, it’s the same concept.
You need to go beyond identifying the product that a customer is using - you need to identify the “job” they are hiring the product for.
“This mental model is behind all of these kind of core concepts,” said Ben. “It’s also what explains why we suck at communicating and marketing and gathering voice of customer data. So we think the product is what we sell, but it's not the same.”
How do you code qualitative data?
Ben shared a recent qualitative research coding exercise that his team conducted for an online jewelry store.
The jewelry client wanted to understand what motivated first-time buyers. They chose to ask the question, “what matters to you most when buying jewelry?”
Taking the open-ended responses, Ben and his team created a list of different codes that seemed to describe the responses well.
The codes are 1-4 word summaries that capture the essence of the responses. In this case, the codes identified included things like quality, style, and price among others.
“With these codes, we have the signal strength of the names that people give to what matters to them,” said Ben.
“We need to now try to make that leap over to what they mean and try to get from the name of something, to something.”
Coding data = finding patterns
After creating the initial list of codes, it was time to take the analysis one step further.
Looking closely at the codes showed that some were connected. Things like quality and durability could be placed in the same thematic category.
Grouping similar comments together allowed Ben to look at the broader patterns in the data.
“In this case, meaning a material value are big core motivating themes. They want the jewelry to connect to something in their lives, right? That's a big thing to pivot on, to hang our hat onto in terms of a strategy and where we're going,” said Ben.
A framework for moving from data to knowledge
The process that we described above is the process of moving from raw data to knowledge that you can act upon.
It doesn’t happen all at once, and it can be a messy process. But the end goal is to go from qualitative data analysis, to insights, to action.
“This model stretches over a lot of cool hypotheses and marketing, and you can use it in your personal life a ton as well,” said Ben.
“The core of this, the heart is that data to code to category. This is the pattern finding.”
One response can be coded in several ways
With open-ended responses, there is a large degree of interpretation. Often, one customer survey response can lend itself to several different codes.
For example, one of the jewelry survey responses was “I really don't see why this is so expensive. Seems like I can get it elsewhere for less.”
This response can be viewed through the lens of price, value, or competition.
How you code it may depend on your end goal, and the type of information you are looking for.
“You can sort of have a different goal in mind, as you search for patterns and start to thrash around here,” said Ben. There is no right or wrong answer.
Look for similarities and dissimilarities
While patterns provided the foundational part of this analysis, you should also look for dissimilarities.
“There's gold in the anomalies,” Ben said. “You don't want to have too much of a scientific approach.”
In other words, you don’t want to become fixated on one correct way of streamlining everything into boxes. Sometimes the similarities will stand out; sometimes the differences will.
Codification process example
When Ben and his team logged the codes for the jewelry client, they used a spreadsheet.
Each of the customer surveys responses was a row, and each new code identified was a column.
As they read through each response one by one, they added more and more columns. Every time a response resonated with an existing or new column, they added an “X.”
Tips & Tactics
Ben recommended the following tips for conducting surveys and interviews in order to ensure you’re starting with the best possible raw data.
- Avoid ‘why’ and ‘was’ questions - these words trigger rationalizations
- Open with ‘how,’ ‘where,’ ‘what,’ ‘when,’ ‘which,’ ‘who’ questions
- Use at least one open-ended question
- Test open-ended questions by trying to answer with yes/no
- Use closed-ended questions for benchmarking
Helpful Software for Coding Qualitative Data
In terms of helpful tools, Ben recommended the below services. These may be particularly helpful for SaaS businesses.
- Amazon Comprehend - a DIY natural language processing service with API
- SEOScout - cheap/quick/free topic modeling (not NLP)
- UserLeap - survey + code tool for SaaS
- Chattermill, Luminoso, or Monkeylearn - for enterprise-level SaaS ML/NLP
Real-life case study
One of Ben’s clients was Native Deodorant, a pricey natural deodorant owned by Procter & Gamble.
The original e-commerce website for Native had the headline: Deodorant to stay fresh and clean. But that was not convincing anyone to buy a $12 stick of deodorant.
Using research, Ben rewrote the value-proposition to better fit the “job” customers were “hiring” for: Deodorant, without the chemistry experiment.
“This is the ‘something’,” said Ben. “This is why I would hire deodorant at $12 a stick. To, you know, not pollute my body.”
It’s always worth going in search of the perfect, powerful message.
You can watch Ben's full talk here.