User Testing or A/B Testing: Which is Better? (Hint: Use Them Both)

User Testing or A/B Testing: Which is Better? (Hint: Use Them Both)

Katlyn Droke

Our tech and strategy teams recently came across an article (“Why User Testing Beats A/B Testing All Day Long”) that took a strong stance on different types of user experience research and testing. Whenever an interesting article like this one pops up, we have a tendency to share it around the office and give our own perspectives on lucrative topics like this. The article makes a case for user testing (or usability testing) being a better, faster, more-reliable option.

But what’s right for you and your projects? Here’s our take: It isn’t a cut and dry decision and it ultimately depends on the situation. First, let’s see what our experts had to say in response to that article.

Kristen, our UX Strategist, says:

Depending on your background, whether marketing or UX, you may have a different view of A/B testing.

In the article, the author says: "A/B testing is done on a live website with real visitors who have no idea that they are part of a test. And, as we noted above, user testing is faster and requires fewer tests for useful data [than A/B Testing]."

While that may be the way marketers typically view A/B testing for websites, that doesn’t have to be true. The way that I have always conducted A/B testing is through a very short, unmoderated click test, usually with the assistance of a testing program like Chalkmark by Optimal Sort.

Here’s the process: Take 2 screenshots that look like the live page with an A variable and a B variable - modified using Photoshop, Axure, etc. Divide the number of users into two, so that half get Screenshot A and the other half get Screenshot B. During the test, ask both groups the exact same question pertaining to the changed variable and see which behaves better towards your desired goal.

These types of tests as outlined above can be done within a couple of hours if you have an available user testing pool (which most large companies have if they have a UX team), and are therefore much less expensive and time consuming than full-fledged moderated usability testing. This type of testing also works well to make changes quickly, as you can easily run the test in the morning and implement changes by the afternoon - something we’ve done in the past for clients.

I also think the author underestimates the work that moderated usability testing requires. I've seen several experienced moderators ask leading or biased questions because they haven't taken the time to prepare their test questions accordingly, which skew results. And more often than not the moderator error goes unrecorded and unconsidered when looking at the results.

However, I do agree that moderated usability testing is the overall best way to get user feedback because it can be done on a quantitative and qualitative basis. I think there are innumerable benefits that usability testing can give to a website design, and the author did a great job outlining some statistics on the successful outcomes of usability testing. UX testing always seems to be the first thing that is dropped from a project plan, but it could be one of the most beneficial activities.

Christen, our Senior Strategist, says:

User testing, whether usability testing or A/B testing, is a great way to get actual user feedback to ensure the experience is working as intended and leading to the intended results for the business. Also, user testing can be the tiebreaker between teams to ensure that user-centric design wins, especially when there may be competing priorities within an organization.

When planning for a project to meet your goals and objectives, you obviously have to judge the various types of testing according to desired outcomes, available resources, budget, time, and scope. I’ve found that most user testing and A/B testing can be scaled to meet the needs of any project, whether you are testing an MVP (which could be flat comps) or testing a full site experience, there are various levels of testing that can achieve the necessary results. Actionable insights are key, as they help inform decision making for how to improve to the project at hand.

My biggest piece of advice? Find a way to implement some level of testing in your projects. The data and insights you’ll gain will be worth every penny, since it will help ensure an experience that meets user needs and aligns to your goals and objectives.

When it comes to deciding what kind of user experience testing you need to do, it helps to take a step back and consider what you want to know.

Let’s say you own a t-shirt company and you’re building a website for it. Ask yourself exactly what kind of decisions you need to make. For example: Do I need inputs on my site’s navigation? If it’s a structural decision that can have a major impact on whether or not your site succeeds or fails with users, then you may have to do in depth user testing. What if I’m not sure what color a button should be? As a t-shirt company, a button isn’t likely to make or break your site, so A/B testing the button color may be the way to go. Others who follow a “lean” approach to web development may consider that the very existence of that button is the MVP (or minimum viable product), and they would prefer to do a live A/B test that could inform the next release.

What if you weren’t a t-shirt company? Let’s say you were a nonprofit instead. That button could be much more important to you. A donate button for a nonprofit would be a major make-or-break point because it relates to profit. If people can’t find your button because it’s in the wrong place or the wrong color, then that may require both usability testing AND A/B testing.

That kind of thinking should help you narrow down what’s best your business, but your situation could be complicated or may even require both. Prior to implementing testing, be sure to gather your team to determine clear goals and objectives for the tests to ensure you get the right insights and feedback to make necessary improvements to the overall user experience. Because, that is the ultimate goal of testing, right?