Quantitative Vs. Qualitative Research. Which is best?

Recently I was able to attend the UX Masterclass conference in Madrid, Spain.

The UX Masterclass is an event held bi-annually by the UXalliance, a global collaborative of 25 separate UX companies across the world.

As usual at the UX Masterclass, there are key speakers and presenters covering a range of topics in the world of UX. Everything from the history of UX through to future UX trends and the new digital generation.

However, at this particular Masterclass there were quite a few presentations and key speakers focusing on the topic of Quantitative Un-moderated Remote User Testing, such as Userzoom.

This raised quite a few questions and conversation points over which testing method was better between Quantitative testing and Qualitative testing. I.e. Is it more beneficial to use Quantitative un-moderated remote testing tools such as Userzoom, compared with doing Qualitative testing in-lab? Opinions were very much divided but this made for an interesting and educational day.

There are of course many benefits and advantages to Quantitative remote user testing, using such tools as Userzoom.

First and foremost with quantitative research is, the data obtained is objective and quantifiable, e.g. you can show stakeholders exact numbers of users that clicked on a certain link, and this data is much easier for business decisions makers to accept and understand.

Secondly, remote user testing can collect vast amounts of data from a large number of participants in a much shorter space of time, from anywhere in the world. There are no locations barriers, no worries about trying to schedule participants for tests or no-shows.

Not only can you collect as much data as needed, it can also be analysed and converted into graphs and tables instantly, without needing to wait for any written reports or videos.

The data produced can be on a large enough scale to make assumptions and decisions on the test product. I.e. If hundreds of people across the country all preferred “Page A” over “Page B”, then with these numbers and data, we can assume and generalise that the rest of the general population will have the same preference.

Finally and possibly one of the biggest points for researchers and stake holders, is that remote user testing is viewed as being much cheaper to perform than e.g. in-lab Qualitative testing. For example, there is no need for facility hire, translators or moderators and it would be assumed to be much easier to get approval and budget for running such tests.

Therefore, in the eyes of many, of course Quantitative remote user research makes much more sense than Qualitative testing.

As much as remote testing can objectively provide you with data and numbers which are helpful for making decisions, it cannot however provide you with deeper answers regarding questions such as “How” and “Why?”

With remote testing, it's easy to see data for “time on task”, or “click through rates” but none of that data will help explain why a user may have spent so long on a page, or why they clicked / didn't click on a certain area.

It may be easy to see that nearly users preferred e.g. “Page A” to “Page B”.

However, “Why did they prefer it?”, “What was wrong with “Page B” that they didn't like?”,

“Was it just the design or was it hard to understand and use?”

Information such as what the users were thinking, why they think a certain problem happened or even how they would solve that problem etc.; answers to these questions cannot be easily gained from remote Quantitative testing. It will require a real person, a moderator sitting in the same room asking those questions. It requires a deeper conversation, more questions and instant adaptability from an experienced moderator to steer the questions in the right direction in order to gain useful answers. Such is not possible with a static questionnaire sheet on a screen in remote testing.

The simple answer would be, both are best. They each have their advantages and disadvantages, but mainly, the first and main point should be what type of answers you are looking for, even before testing begins.

If your reason for research is to gain objective, clear figures and data to certain questions, such as “Which design did users like more?”, then use Quantitative to find out, and this may help with making future design choices.

If on the other hand, you would like to find out answers to questions related to e.g. user habits and reasons for trusting certain sites and brands; then use Qualitative research to gain deeper insights and to be able to see a user's reactions and expressions when answering or using the product.

Therefore, having heard many Quantitative remote research presentations in Madrid, as well as comments from the audience regarding why Qualitative is also better; it may just be that mixing both types of testing would be an ideal solution.

Perhaps beginning with a broad remote user test to find e.g. user patterns and clear statistics regarding the site or product, and finally move onto qualitative to discover whether those issues need to be fixed, do they affect user's feelings etc. and to how to prevent them from happening again.

To conclude, there is no clear “best testing method” or a winner between Quantitative and Qualitative research.

Just remember, next time before planning a test with a client, ask them what answers or information they would like to find out, and then choose the method which would be “best” to in order to get them.

That will be the answer to which type of testing is best.