6 Reasons Not to Test with Users (and Why to Test Anyway)

5 April 2016
Categories: Blog

Every design ever made has a user – but not every experience is designed with the user in mind. Sometimes it’s because designers don’t know enough about their users; sometimes it’s because deadlines are too tight to bother. But in the end, users are the ones who interact daily with designs, and users are the ones who can help make designs better.

Usability testing catches sticky user problems when the product is still in the design phase. So why are so many organizations reluctant to get started? Anecdotally speaking, it seems as though every UX practitioner has one—or a dozen—stories of higher ups and team leads who said “we can’t usability test. It’s just….

  • too expensive
  • a waste of time and resources
  • biased and artificial
  • impossible to test enough users
  • unnecessary, since users aren’t experts
  • redundant, since we have market research and competitive analyses.”

In this article, we’ll respond to all of these objections, and give UX practitioners the ammunition they need to convince any doubters that usability testing is, in fact, right for every project.

Objection #1: User testing is too expensive

While we’d love to do limitless user testing, we do have to acknowledge project constraints, and budget is often a huge one. Sometimes, it’s enough to say “usability testing will save us from possibly producing a product no one wants,” but when the budget absolutely can’t be expanded, that’s ok too!

Some user testing platforms can be incredibly expensive, but they’re not the only options. Platforms like UserTesting.com are expensive because they allow us to customize every detail of our test, including the demographic of people we’re testing. For highly specific products or services, a narrow user field is perfect—a divorce attorney isn’t interested in a happily married person’s opinion of his website. However, for the local coffee company trying to sell delivered-to-you coffee, their customer base will range from a 20-something hipster who’s into the newest thing to a house-bound retiree who can’t get out to buy groceries.

Assuming user demographics vary, we can run cheap, DIY user tests early and often. We call these impromptu tests guerrilla testing, which is exactly what it sounds like: testing in the wild with randomly selected users. It can reveal blind spots in the project before too much money or time is spent creating something users won’t want in the end. All it requires is brief coffee shop sessions, where a team member asks a variety of customers to offer their thoughts on the website or product in exchange for a cup of coffee.

For teams operating in a digital-only environment, it can be hard to do guerrilla testing. But cheap testing can still be done. Affordable programs like Qualaroo allow testers to use surveys or questionnaires to capture opinions from current site visitors. And UsabilityHub offers quick initial impressions, preference tests, and navigation tests at a reasonable cost. The possibilities for quick user feedback are only as limited as your imagination. Just be sure to approach potential testers with clear respect for their time and opinions, and a defined goal behind each question.

Objection #2: User testing is a waste of time and resources

Really?! Did they not just hear us say that user testing prevents the launch of unwanted, unusable projects?

Whether a test succeeds or fails, if the team learned something about how they could improve the website, app, business, or product, they came out ahead. Sure, we could save some money and time by not testing, but how else will we learn if users like what we’re offering? Even just running the product by three actual consumers will turn up obstacles that the team may never have recognized on their own.

For example, Drupal developers at Digital Loom were tasked with rebuilding the New England Foundation for the Arts (NEFA) website, and were on the fence about user testing. One of the primary purposes for NEFA’s website is collecting donations for the foundation. After a few user tests, it quickly became obvious that people were overlooking the “donate” button, even when they were tasked with making a donation. If Digital Loom hadn’t bothered to watch a few users run through the site, they would never have caught the button blindness and could have cost NEFA donations.

When it comes down to it, “wasted resources” is only a good objection if the team has no intention of fixing the issues that come up in testing. And if the team isn’t interested in improving the product, then there are bigger problems at hand.

Objection #3: User testing is not objective, so the results are unrealistic

We know that many user tests are administered in settings that feel contrived for the user, and the moderator might not be an impartial observer based on her role in developing the test project. Plus, the test questions will never be perfectly neutral, no matter how much time we spend on them. But that’s ok!

As the founder of MeasuringU, Jeff Sauro says,

Usability testing is artificial.

We do the best we can to simulate a scenario that is as close to what users would actually do with the software while we observe or record them.

However, no amount of realism in the tasks, data, software or environment can change the fact that the whole thing is contrived.

This doesn’t mean it’s not worth doing.”

There are lots of resources to help overcome common moderating biases, and simple tactics for avoiding them. Sauro offers nine of the most common user testing biases and UX Agony Aunt highlights some copywriting tips to help you avoid bias in your test questions. The key is not to give up because all tests are flawed. Let go of your perfectionist tendencies for just a minute, and focus on ways to improve the objectivity.

Objection #4: We’ll never be able to test enough users to reach statistical significance!

This is an easy one. The idea that we need to test hundreds of users in order to understand usability issues is a myth. It comes from the comparisons between user testing and market research: market research identifies preferences and facts, and therefore we need to talk to a huge number of people in order to find the overarching patterns. However, user testing is intended to find usability issues.

The Nielsen Norman group has actually found that we don’t need to test more than five users to identify usability issues. At that point, key design problems begin to repeat. If we test three users, we’ll find most of the major flaws right away; if we test with five or more, we’ll see the major flaws repeated and likely not learn anything new.

When the sand goes in the jar first, the rocks don't fit.

Photo source: Children’s Ministry

Consider the story of the rocks and pebbles and sand in a jar: if you put the sand in first, the rocks won’t have room to fit. But when the rocks go in first, there’s plenty of room for the sand. This applies to testing priorities as well. User testing is the rocks—the obstacles to conversion, the missing information, the form that doesn’t make sense—the largest problems a user will face with your website or product. The pebbles and sand are all the copy improvements, FAQ articles, and A/B tests you’ll conduct after the product is launched. There is time to fit them in later, but usability testing identifies the rocks, which need to be taken care of immediately.

Objection #5: Users aren’t experts—why do I need their opinions?

This is probably my favorite objection. It assumes I have somehow managed to absorb millions of personalities, preferences, and patterns into myself and can look at a website or product through the lens of all of them. I have a lot of research under my belt, and I have performed a lot of tests myself, yes, but all this tells me is that no one user test is the same as another.

Now, let’s be clear for just a moment—user testing is inherently biased. It is one emotional, unpredictable human asking another emotional, unpredictable human for an opinion. But, we can use that humanness to our advantage. Our goal is to identify user biases, and find out where the emotional, unpredictable, human aspects need to be accounted for in our designs. Because, after all, our end users are also humans—and they’re not design experts.

Plus, by watching a user manipulate designs, we learn short cuts they’ve developed to find buried or hidden content, obstacles they can’t overcome, as well as goals, pain points, and missing information. We don’t want advice from experts—we want advice from the people who know nothing about code or design but have to use our website every day.

Working closely on building a product or website, we tend to develop what I call “website blindness” — we’re so involved in the back end functionality, in the details, and the layout that we can navigate it better than anyone else. Think about your house: after you’ve lived in a house for a few years, you can find your way around in the dark without ever stubbing a toe or missing a light switch. Your website is like your house—you have a deep understanding of how it goes together and how to navigate it, but your users are completely in the dark. User testing helps you identify where people are stubbing their toes, and where you should put a light switch, so to speak.

Objection #6: Why can’t I just use market research and competitor research?

As we mentioned before, user testing is not the same as market research. Market research largely measures how consumers feel about your brand or product. That’s why it requires enormous sample sizes, like those mail-in surveys you get once a year to explore your shopping habits. User research can work with market research, but has a different goal entirely: user research looks at behavior. Focus groups, mail-in surveys, phone quizzes—all are asking users what they want (see Objection #4) but user research watches how they behave in real time and makes decisions off of this information.

One example that most of us can relate to is Apple. Apple is famous for not using market research, according to Jobs himself. But that doesn’t mean the iPhone in your back pocket just happened to be great the first time around. Apple has made a point of not asking users what they think they want, but giving users product designs and observing how they use the design.

Legend has it Apple once packed a pizza box with bricks to simulate the size and weight of a portable computer, and then followed the users around, watching how and when they used the “computer.” The prototype and resulting observations led them to develop a more manageably sized product.

Think about this: how can users with no design experience or understanding of the possibilities of technology dream up answers to “What should a smartphone do?” They don’t have the capacity to even guess at what they would want the device to do for them. But if you provided them with a device that did 75 things, and watched which tasks they performed and which apps they never used, you’d have a better sense of users’ true intentions for technology.

Go research the competition, and when you’re ready, and you think you’ve got an equally great or uniquely better product, then watch your users test both products.

What Comes Next?

Overcoming objections isn’t the end of the project. It’s just the beginning! Now that the higher ups are on board, it’s time to actually do user testing. So in Part 2, we’ll talk through how to prepare for user testing and set the team up for success.

Have you experienced other objections to user testing? Check out these resources: