Are Surveys Effective in User Research?

The effectiveness of user surveys have always been debated in the field of user experience. Despite them being ubiquitous, there are quite a few voices that caution you against using or over-using them.

Given that your research is only as good as your data, establishing the effectiveness of the tools you use to collect said data is important.

So let's talk about the tool each of us have probably used at one time or another — surveys. Do surveys universally suck? Or can they actually be used to collect actionable, unbiased data?

Three benefits of using a UX research survey

There's no denying that UX surveys have several advantages over other data collection methods. These qualities are what makes them so pervasive in research gathering.

Let's examine a few reasons why surveys are effective in user research:

1. Survey participants are less prone to Hawthorn Effect

While innately a UX survey suffers from sampling bias, (which can be fixed via correct incentives - more on this shortly) it is actually less prone to the Hawthorne effect (also called the observer effect) compared to some other forms of data gathering.

The Hawthorne effect is a type of reactivity in which individuals modify an aspect of their behavior in response to their awareness of being observed.

— The Hawthorne Effect: a randomised, controlled trial

As researchers, we've probably all seen this in action — when users are being observed, they're more likely to give us answers they think we want to hear. In the case of user research surveys (especially anonymous ones), individuals feel less pressure to modify their responses because typically, they're responding from the comfort of their environment without the fear of being judged.

2. User research surveys are inexpensive

Perhaps the main reason why surveys are so ubiquitous is that they're among the cheapest ways of gathering user research data. Most surveys cost a grand total of nothing to send out.

Even if you incentivize the surveys (we'll discuss later why we prefer incentivized surveys), the relative cost is minimal compared to what it takes to recruit participants, conduct interviews, or contextual inquiries. As a researcher, gathering data without allocating a large part of your budget to it is always ideal.

3. User research surveys are scalable

Scalability is another area where surveys can outshine other research data collection methods. Active research data collection methods like interviews and ethnographic studies get exponentially more expensive in terms of time and resources as you start scaling them.

For instance, imagine that you were synthesizing just 20 interviews with 20 freeform answers each. Creating an affinity diagram to bubble up important concepts from 400 responses may become incomprehensible. Unless of course you're using UserBit's tag analysis. 😉

Affinity diagram created directly from highlight tags on UserBit

On the other hand, surveys for user research can easily be sent out to a larger audience. And if used appropriately, quantifying their responses can happen quickly. In fact, most survey tools these days automatically create charts for your data as the responses come in.

Google Forms automatically generate analysis for your data

Disadvantages of UX research surveys

So why do so many people complain about surveys being a valid user research method? Let's look through some shortcomings of this data collection method:

1. Surveys may result in skewed data

Fact of the matter is, no one likes to be on the receiving end of user research surveys. Unlike Buzzfeed quizzes, there's no instant gratification at the end of a product survey telling you what Harry Potter character you're. This means, by default survey responders fall into 'love you' or 'hate you' buckets.

You'll get a proportionally higher number of responses from people who really like your product or really hate it. In statistics, this is called Sampling Bias. This is a problem. You need your data to be unbiased so the synthesized insights don't lead you towards creating products that only a small subset of your users want or don't want.

The fix

Incentivize the survey with a reward. A gift card or some form of credit goes a long way in getting people to take your survey not because they feel a certain way about you, but for material benefit they get to receive.

2. Surveys often result in poor qualitative data

A lot of surveys contain questions with free-form answers. The hope here is that the user will take time out of their busy life to write essays about how they feel about your product. I hope that sounded as ridiculous as it was for me to type it.

In truth, you'll most likely get one word answers (one sentence if they really like you) which gives you little to no qualitative data.

The fix

Just don't do it. Don't use surveys for collecting qualitative data. There are other methods much more suited for that (in-person interviews, contextual inquiries, ethnographic studies). You should use surveys to get short quantifiable responses via Likert scale or binary answers.

3. It is easy to create bad surveys

Creating surveys is easy, which would explain why we encounter a ton of terrible surveys. Bad questions in a survey are just as problematic as asking bad questions in an interview. Every leading or biased question in a survey is compounded and more damaging because of the scalability that surveys offer. And yet, because creating surveys is so inexpensive, they don't go through the same scrutiny as interview questions.

The fix

Surveys should be subjected to the same level of scrutiny as interview questions. Don't ask questions that are leading or biased. Also don't expect your users to write long thoughtful responses to survey qustions.

When should you use surveys in your user research workflow?

Now that we've examined both the good and the bad of surveys, let's talk about when it makes sense to use them during our research process. Because surveys are so easy to create and use, it might be tempting to send them out every opportunity we get. However, due to the shortcomings above, surveys are not a good fit for all situations.

For example, they cannot be relied upon to get good qualitative feedback, and therefore, they should not be used as a primary data collection method. On the other hand, surveys make one of the most effective supplemental tools for corroborating existing hypothesis.

UX survey example: Will anyone pay for my product?

Let's say you wanted to know if your target audience would pay for the app you're making. You'd start with conducting user interviews as your primary validation method. Then, you would ask your interviewees things like:

• What was the last app they paid for?
• How much did they pay for it?
• etc.

Setup and organize interviews notes with UserBit's interview module

The idea is to validate that the given problem is:

1. Something users would indeed pay to solve
2. How much would they be willing to pay.

You've gone ahead and conducted your interviews and now you want to corroborate your findings. A survey can help you do this at scale!

Specifically, you can send out something like the Van Westendrop's price sensitivity analysis questions as a survey to your bigger user pool.

Van Westendrop Sensitivity Meter

Not only does this help you validate your hypothesis, but also quantifiably corroborate it at scale.

If you do use surveys as a supplementary data collection method, you're in good company. Siddhi Sundar, Sr. UX researcher at Netflix also talks about how the triangulation of data using surveys helps them uncover users' needs.

Survey research and behavioral data, combining the results from a comprehensive, multimarket attitudinal and behavioral survey with analysis of Netflix behavioral data.

Conclusion

Remember that surveys are just tools and they aren't intrinsically bad or good. But like any other tool, if you use them incorrectly, you can expect suboptimal results. Surveys do have shortcomings when it comes to collecting qualitative data, but nothing beats it as a scalable supplemental method to corroborate your insights.

UserBit is an all-in-one user research platform for user UX and product teams. Your home for research repository,analysis, user personas,journey mapping,card sort studies, and more.