Sumo Logic recently hosted an Enterprise UX Meetup that I got to open with an introductory presentation on “Designing by Numbers”. Many designers think of numbers only in conjunction with quantitative methods, but hybrid options allow designers to apply numbers to qualitative methods as well. The following blog is my attempt to create a simple four-step process to extract meaning from a hybrid testing method.
Results are everything
When I started thinking about the presentation that turned into this blog, I thought about the similarities between the world of Sumo wrestling and the world of design. Results are ingrained in the Sumo wrestling lifestyle. Everything that these amazing athletes do from the moment that they wake up to when they go to sleep at night is driven by results.
Similarly, in design, results are everything as well.
At Sumo Logic, we analyze close to 50 Petabytes of data and scan 175 Trillion records a day! The scope of our service is really amazing. But we have to make sense of all that data. The results of how our users use our product tells us whether our design is effective or not. At the end of the day, that’s what matters. We can always say, our designs are great and we love how cool they are, but like Sumo wrestlers, results drive everything we do.
Data is to design as strength is to Sumo.
A Sumo wrestler trains everyday to gain strength. Similarly in design, the more data you can gather, the more strength you have to inform your designs. It seems obvious, but you’d be surprised by how many companies that design without gathering data first. They assume they know it all, but in reality, designing without data can be pretty dangerous. For example, at Sumo Logic, we know that our product has to help our customers manage critical applications like DevOps, IT, and Security. So in that case, if our designs don’t focus on our customers’ goals, it could literally be dangerous to their—and our—bottom lines.
So how do we think like a Sumo when it comes to design?
1. Know your opponent
In this case, the opponent is the design problem.
First, break down the problem that you’re trying to solve.
This is a big challenge when you’re just starting to gather data. Often, designers may make the mistake of wanting to tackle all the problems at once, when in reality the first thing you must do is decide, what is the main problem we’re trying to solve?
Next, remember that problems are attached to people.
When you’re studying the problem, remember to study the person who is having the problem.To try to solve a problem without concentrating on the person is like saying a Sumo is wrestling “fighting moves” instead of another Sumo wrestler.
Finally, it’s a good rule not to measure “likes” or “hates”.
These are words that make people feel like they have to react to or defend against them. And they’re not really measurable. Concentrate on the reasons for the “likes” or “hates”. Ask users, what is it exactly about that feature that you really like or hate? UX designers, Product Managers, or anyone in the UX field need to get around vague answers and not let them sway you one way or another out of a knee-jerk reaction. Focus on the reasons your users give you, and make sure you know what you’re really measuring as you collect data to define your problem.
2. Identify high risk areas
A Sumo wrestler has to know his opponents weaknesses in order to exploit them, but even more importantly, he needs to know his own weaknesses in order to defend against them.
High risk areas are the strikes that can knock you out.
But once you identify them, you can defend against them.
For our Metrics project, the UX team developed many creative deliverables, including this Customer Journey diagram below. Even if it looks a little bit like a comic strip, with it we were able to identify the high risk points in our customers’ day-to-day work, and efficiently communicate them with internal stakeholders, executives, and customers.
Illustration by Nitesh Jain
Often, designers confuse “high risk” with “high friction”.
For example, if you have a web form that’s difficult for your customer to get through, but they can get through and then continue, that would be considered high friction, not high risk. High risk means if your customer can’t get through that form, they’re out. Game over. So the high risk points are the main areas you want to identify, tackle, and make sure are working well before you move on in your design process.
3. Hypothesize
Once you know who is having the problem and understand the high risk areas, you can prepare to strike—with a hypothesis.
In the Sumo wrestling world, when the opponents take their positions and prepare to strike, it’s an elaborate setup that takes a long time before the moment of attack.
This is similar to creating your hypothesis in design. But you’ve already gathered data, talked to users, and done your research. Your hypothesis is clear. Now, it’s time to attack—with creativity and testing.
4. Be creative and test
Just like how a Sumo wrestler has learned many different moves, there are many ways to attack a problem. In design, these require creativity and user testing, as illustrated in the following diagram.
Qualitative. On one side of the spectrum you have a qualitative attack, which is usually direct contact with a person, and the results are measured in words. This method requires smaller sets of user testers.
Quantitative. On the other side of the spectrum you have a quantitative attack, which is usually indirect, and the results are measured in numbers. This method requires larger sets of user testers.
What People Say. On the bottom of the spectrum, you can base your attack on what people say in user testing. But often what they say what they would do in a situation, and what they actually do is very different.
What People Do. At the top of the spectrum, you can base your attack on what people actually do in user testing.
Which is the best method? It depends on the problem, the people, and the high-risk factors. You need to be creative and try different methods to determine what is the best way to tackle your design problem.
Relying on one method is like a Sumo wrestler relying on only one move.
Here’s another example. Let’s say we’re using a qualitative method to measure pizza diets.
In this test, you have four different diet categories:
- Carnivores
- Only New York pizza
- Vegan
- Gluten free
From this result, you could say that the number of “Carnivores” and “Only NY” is just about equal. With this method though, the words you use to describe the results are vague. For example, you could say that not many people are interested in “Gluten Free” pizza diets.
To make your results more specific, you can take your qualitative results, and superimpose numbers on the results, to get a quantitative result. Convert your results to ones and zeroes, do the math, and get actual number results. Then you can say “Two out of six people are interested in Vegan pizza diets.”
Once you have qualitative results, you can find out the reasons behind these answers, so they’re not just numbers. For example, one test participant says he or she is both a “Carnivore” and a “Vegan.” You might want to follow up with that person and see why they answered that way.
Overall, combining these two methods is much more powerful than using only one method, like a Sumo wrestler using a combination of moves to throw his opponent off balance. Together, they can lead to a new jumping off point to ask more details questions to get even better results.
How do I know if I got it right?
From experience, I can tell you that you’ll know when you have it right. You’ll also know when you have it wrong. There’s a lot of vagueness in between, but not on the ends of the spectrum. Ultimately, your customers will tell you when you have it right with how they respond to your product. We’ve had great responses from our Sumo Logic customers, and with your continued help, we’re making our product better all the time.