Are Surveys Effective in UX Research?

The effectiveness of user surveys has always been debated in the field of user experience. Despite them being a commonplace tool, there are quite a few voices that caution you against using or over-using them.

Given that your research is only as good as your data, establishing the effectiveness of the tools you use to collect said data is important.

So let's talk about the tool each of us has probably used at one time or another — surveys. Do surveys universally suck? Or can they actually be used to collect actionable, unbiased data?

3 benefits of using a UX research survey

There's no denying that UX surveys have several advantages over other data collection methods. These qualities are what makes them so pervasive in research gathering.

Let's examine a few reasons why surveys are effective in user research:

1. Survey participants are less prone to Hawthorne Effect

While innately a UX survey suffers from sampling bias, (which can be fixed via correct incentives - more on this shortly) it’s actually less prone to the Hawthorne effect (also called the observer effect) compared to some other forms of data gathering.

The Hawthorne effect is a type of reactivity in which individuals modify an aspect of their behavior in response to their awareness of being observed.

— The Hawthorne Effect: a randomised, controlled trial

As researchers, we've probably all seen this in action — when users are being observed, they're more likely to give us answers they think we want to hear. 

In the case of user research surveys (especially anonymous ones), individuals feel less pressure to modify their responses because typically, they're responding from the comfort of their environment without the fear of being judged.

2. User research surveys are inexpensive

Perhaps the main reason why surveys are so widespread is that they're among the cheapest ways of gathering user research data. Most surveys cost a grand total of nothing to send out.

Even if you incentivize the surveys (we'll discuss later why we prefer incentivized surveys), the relative cost is minimal compared to what it takes to recruit participants, conduct interviews, or contextual inquiries. As a researcher, gathering data without allocating a large part of your budget to it is always ideal.

3. User research surveys are scalable

Scalability is another area where surveys can outshine other research data collection methods. Active research data collection methods like interviews and ethnographic studies get exponentially more expensive in terms of time and resources as you start scaling them.

For instance, imagine that you were synthesizing just 20 interviews with 20 freeform answers each. Creating an affinity diagram to bubble up important concepts from 400 responses may become incomprehensible unless, of course, you're using UserBit's tag analysis. 😉

Affinity diagram created directly from highlight tags on UserBit

On the other hand, surveys for user research can easily be sent out to a larger audience. And if used appropriately, quantifying their responses can happen quickly. In fact, most survey tools these days automatically create charts for your data as the responses come in.

Google Forms automatically generate analysis for your data

Disadvantages of UX research surveys

So why do so many people complain about surveys being a valid user research method? 

Let's look through some shortcomings of this data collection method:

1. Surveys may result in skewed data

The fact of the matter is, no one likes to be on the receiving end of user research surveys. Unlike Buzzfeed quizzes, there's no instant gratification at the end of a product survey telling you what Harry Potter character you are. This fact means, by default survey responders fall into “love you” or “hate you” buckets.

You'll get a proportionally higher number of responses from people who really like your product or really hate it. In statistics, this phenomenon is called sampling bias, and it’s a problem. You need your data to be unbiased so the synthesized insights don't lead you toward creating products that only a small subset of your users want or don't want.

The solution

Incentivize the survey with a reward. A gift card or some form of credit goes a long way in getting people to take your survey not because they feel a certain way about you, but for the material benefit they get to receive.

2. Surveys often result in poor qualitative data

A lot of surveys contain questions with free-form answers. The hope here is that the user will take time out of their busy life to write essays about how they feel about your product. I hope that sounded as ridiculous as it was for me to type it.

In truth, you'll most likely get one-word answers (one sentence if they really like you) which gives you little to no qualitative data.

The solution

Just don't do it. Don't use surveys for collecting qualitative data. There are other methods much more suited for that (in-person interviews, contextual inquiries, ethnographic studies). You should use surveys to get short quantifiable responses via a Likert scale or binary answers.

3. It’s easy to create bad surveys

Creating surveys is easy, which would explain why we encounter a ton of terrible surveys. Bad questions in a survey are just as problematic as asking bad questions in an interview. 

Every leading or biased question in a survey is compounded and more damaging because of the scalability that surveys offer. And yet, because creating surveys is so inexpensive, they don't go through the same scrutiny as interview questions.

The solution

Surveys should be subjected to the same level of scrutiny as interview questions. Don't ask questions that are leading or biased. Also don't expect your users to write long thoughtful responses to survey questions.

When should you use surveys in your user research workflow?

Now that we've examined both the good and the bad of surveys, let's talk about when it makes sense to use them during our research process. Because surveys are so easy to create and use, it might be tempting to send them out every opportunity we get. However, due to the shortcomings above, surveys aren’t a good fit for all situations.

For example, they can’t be relied upon to get good qualitative feedback, and therefore, they should not be used as a primary data collection method. On the other hand, surveys make one of the most effective supplemental tools for corroborating existing hypotheses.

UX survey example: Will anyone pay for my product?

Let's say you wanted to know if your target audience would pay for the app you're making. You'd start with conducting user interviews as your primary validation method. Then, you would ask your interviewees things like:

• What was the last app they paid for?

• How much did they pay for it?

Setup and organize interviews notes with UserBit's interview module

The idea is to validate that the given problem is:

1. Something users would indeed pay to solve

2. How much would they be willing to pay

You've gone ahead and conducted your interviews and now you want to corroborate your findings. A survey can help you do this at scale!

Specifically, you can send out something like Van Westendrop's price sensitivity analysis questions to survey your larger user pool.

Van Westendrop Sensitivity Meter

Not only does this help you validate your hypothesis, but also quantifiably corroborates it at scale.

If you do use surveys as a supplementary data collection method, you're in good company. 

Siddhi Sundar, Sr. UX researcher at Netflix also talks about how triangulating data using surveys helps them uncover users' needs.

Survey research and behavioral data, combining the results from a comprehensive, multimarket attitudinal and behavioral survey with analysis of Netflix behavioral data.

Conclusion

Remember that surveys are just tools and they aren't intrinsically bad or good. But like any other tool, if you use them incorrectly, you can expect suboptimal results. Surveys do have shortcomings when it comes to collecting qualitative data, but nothing beats it as a scalable supplemental method to corroborate your insights.