Usability testing, whether remote or in-person, requires researchers to observe and converse with users. From those users interactions, you’ll build a qualitative dataset that can be used to analyze your product.
How do you observe unobtrusively and ask questions to make sure you have a well-populated dataset? Let’s review some techniques to effectively moderate usability tests.
Usability testing refers to evaluating a product or service by testing it with representative users.
https://www.usability.gov/how-to-and-tools/methods/usability-testing.html
Observation Tips
Embrace Silence
Users should practice a think-aloud protocol. This protocol can shed light on what a user is thinking. Narrating what you’re doing isn’t necessarily hard, but it can interrupt natural (internal) processing of information.
Give users breaks to think to themselves if they get stuck or are trying to solve a problem. If you’re worried about missing insights on their thought processes, note the instance for review in the post-test follow-up questions.
Let the User Struggle
Users make mistakes all the time. Where would we be without auto-correct? No one ducking knows.
When the users make a mistake they will ask things like:
- Is this right?
- How do I get there?
- What does that mean?
We need to see users make mistakes so we can resolve issues and create effective design patterns. Avoid the temptation to answer questions right away. Instead, follow-up with a question of your own like What do you think it means to get this right?
It can be painful to watch users struggle, and repetitive if you’ve got a large sample size, nevertheless let the users encounter a fair share of difficulty so you can identify the areas of your product that need to be improved.
Interview Tips
Ask Open-ended Questions
To create a well-populated dataset, you should ask questions that encourage the user to reflect and elaborate on previous experiences. Compare these open-ended vs. closed questions and consider what you think would be more effective at generating conversation points?
| Closed | Open-ended |
|---|---|
| Did you notice this warning message? | Tell me what you thought about the warning messages. |
| Did you find what you were looking for on this page? | Tell me about the process you went through to search for [content]. |
| Is this page what you expected? | How does this [page/workflow] compare to others you’ve encountered online? |
Open-ended interview questions create opportunities to collect unexpected insights about a product.
Don’t Ask Leading Questions
Leading questions bias a user’s responses before they have the chance to share their unfiltered reactions. It’s perfectly normal for a UX researcher to bring your own expectations into a study. These expectations, however, can lead to confirmation bias. Thus, as a result of inaccurate analysis, the product moves further away from a user’s mental model.
To encourage alignment of your product with your user’s expectations, ask questions that promote open dialog. My favorite interview question to start dialog is What stood out about that?
I use this question all the time and when I compare it to other questions like these, I find it to be issue free.
| Question | Issue |
| Tell me about how hard this [workflow] was. | You frame the workflow as hard. This tells the user you expect them to have had difficulty and may inflate difficulty ratings. |
| What parts of the page seem cluttered? | You’re telling the user you think this page is cluttered and asking them to agree with you. |
| If you had to improve one thing, what would it be and why? | Users will cherry-pick something to improve that might be working perfectly fine. |
When you ask a user What stood out about that? they can organically say things about a workflow being hard, content being cluttered, or wanting to improve a layout. In an artificial usability testing environment, creating access to organic, unfiltered reactions is crucial.
Ask One Question at a Time
Interviews of any kind require the use of memory to recreate situations from previous events. Since memory recall is far from flawless, ask one question at a time to avoid negatively impacting user recollection. Present one question at a time, otherwise your user’s responses could be mapped incorrectly.
Open-ended interview questions create opportunities to collect unexpected insights about a product.
Tweet
Any Question is Better than no Question
Products grow quickly and priorities can change without warning. When these changes inevitably disrupt your workflows, it’s a reminder that you will always be having conversations with new users. Get to know them and remember to ask lots of questions.