Best practices for moderated user testing
What makes a good usability test moderator?
Usability testing is a collection of techniques used to measure the way a user interacts with a product. It plays a crucial role in helping product teams understand usability issues with their products. Usability testing can be either moderated or unmoderated. Moderated testing requires the active participation of a moderator (usually, a user researcher or a designer who will facilitate the testing) and can be conducted in person or remotely. Unmoderated usability testing is completed without having a moderator present in the user’s own environment.
In moderated user testing, the results are highly dependent on the skills and abilities of the moderator (Barnum, 2011). For example, the cognitive biases of the moderators can have a significant effect on the results of the study by affecting user performance.
Research suggests that most UX practitioners don’t have formal training in moderation techniques and they tend to learn how to moderate usability testing through trial and error and from relevant literature.
A number of different frameworks exist to help with this. According to Dumas and Loring (2008) the moderator has three main roles:
The Gracious Host: Ensuring participants feel welcome and the session goes smoothly leading to a positive experience overall
The Leader: Participants expect moderators to know what they’re doing and take the lead. Moderators should also control the pace of the session and take charge whenever issues come up. They are also expected to know what to do when participants need encouragement, prompting, or assistance.
The Neutral Observer: The goal of the moderator is to collect accurate data while playing the other roles by letting the user speak, asking unbiased questions and avoid defending the product design that is being tested.
They have also come up with 10 Golden Rules for interacting with users duing moderated testing:
Decide how to interact based on the purpose of the test.
Protect the test participants’ rights.
Remember your responsibility to future users.
Respect the test participants as experts, but remain in charge.
Be professional, which includes being genuine.
Let the test participants speak.
Remember that your intuition can hurt and help you.
Be unbiased.
Don’t give away information inadvertently.
Watch yourself to keep sharp.
In practice, few practitioners follow a specific framework when conducting user testing. Hertzum and Kristoffersen (2018) studied a number of usability sessions and came up with the following suggestions:
Moderators should try not talking much. The focus should be on the user verbalizations.
Moderators should be more conscious about their affirmations (e.g., ‘Mm hm’, ‘Okay’, ‘Uh-huh’) and keep them neutral.
Closed questions, which may also be leading, should be avoided. Open questions should be used as they are less leading and allow the user to express their thoughts.
Moderators should avoid asking users to answers hypothetical situations, for example, “What do you think would happen if you …?”
The moderator verbalizations can have a significant effect on the user.
CUE (Comparative Usability Evaluation) is a series of ten studies that investigate the reproducibility of usability evaluations and explore common practices among usability professionals. In the most recent one, CUE-10, researchers focused on moderated usability testing. In particular, they examined the following:
What are the key issues that distinguish good from poor moderation?
How can we avoid moderation errors?
What is good moderation practice?
From February to April 2018, moderators independently moderated usability test sessions of the website Ryanair.com (see Figure below). Each of the 16 moderators moderated three usability test sessions (full instructions and the tasks can be found here).
Moderators were given two primary instructions for the test:
Each session should be limited to at most 40 minutes.
All sessions should be recorded and show both the test participant and the moderator in addition to the participant’s screen.
The key observations from the study are presented below.
Building Trust and Rapport
One of the most underrated aspects of conducting a user study is the importance of building rapport with your users. It begins during recruiting and continues throughout the entire test session. In the CUE10 study, moderators used a variety of techniques to build trust and rapport. The most helpful ones were the following:
They informed users that it is important to find out what does not work well and that it is okay if things go wrong or are certain tasks are difficult to perform.
Asking background information about the user. For example, enquiring whether they are already familiar with the website used in the study or if they have any similar experience that could affect their interaction.
Provide encouragement to the test participant if they self-blame.
Offer compliments to the users during testing.
Inform the users about the role of the moderator as a neutral observer.
Managing Time
Ensuring that usability sessions don’t overrun is an important responsibility of the moderator. The user’s time should be respected and test sessions should not exceed the agreed length of the test session.
The moderators who managed their time successfully did the following:
They stayed focused on the given task throught the testing.
When the usability problem was clear to the moderator, they gave hints to the users.
They stopped a task when the usability problem was clear to them.
Giving Tasks to Test Participants
The way tasks are presented to participants can affect usability testing. Molich and colleagues observed inconsistencies in the way this is done by the moderators; some handed the tasks to test participants and asked them to read the task aloud, while others read the task to the users.
The most effective strategies were the following:
Presenting users with one task at a time and provide them with a copy of the task (e.g., instructions on paper or on screen).
Moderators should not tell users how many tasks they have prepared at the start of the session in case they need to skip the task. It also ensures that participants are not overwhelmed.
Structuring Usability Test Sessions
The typical usability session followed the structure described below:
Briefing: The moderator informs the user about the purpose of the usability test, the procedure, their role and contribution, the approximate length of the test session, and their rights (e.g., privacy, data handling). At this stage permission to record the session is obtained and users are told that it is not them that are being tested but the specific product design.
Pre-task interview: Moderators gather background information about the test participant (e.g., previous experience, education) that might affect the way they interact with the design. The pre-task interview is also a helpful way to build rapport.
Task moderation: This is the main part of the session. The user performs each task, one at a time.
Debrief after each task has been completed (optional): Some moderators conduct a short debrief after each task had been completed.
Final debriefing: At this stage the user has a chance to talk about their experience and any general thoughts they might have about the system being tested. Moderators can ask open-ended questions such as “What did you like most?” and “What is most in need of improvement?” or focus on events that occured during testing.
Giving Prompts, Probes, and Assists
Even though moderators need to be quiet during the task moderation part of the session, they sometimes need to intervene. This can be achieved with the use of prompts, probes, and assists.
A prompt is an act of encouraging a hesitating test participant. It’s a way of letting the participant know that the moderator is engaged and listening.
A probe is a question to a test participant during task solution (“And why is that?”). It is asked to find out more about the way a user reaches to a particular solution and their mental model.
Finally, an assist is an act of helping a test participant. A moderator gives an assist when a participant is struggling to move on or to skip a well-known or unimportant usability problem.
Preparing for the Sessions
To ensure sessions run smoothly it is crucial that moderators are familiar with the product being tested and are aware of potential issues and limitations. This can be achieved by spending time with the prototype/product or by conducting pilot sessions.
What does this mean for me?
So what are the takehome messages from Molich et al’s research? They can be summarised in 9 statements.
Time management is important and can be achieved by focusing on list of tasks, stopping participants if they stray, and proceeding to the next task (by skipping or providing assists) when the usability problem is clear.
Preparation is key! Be familiar with the tasks at hand and the product you are testing. Pilot testing can help with this. This also ensure that the tasks that were chosen are appropriate for the allocated time.
Build trust and rapport with the user. This can be achieved by getting background information about the user, ensuring them that they are not the ones being tested and compliment them for their work (e.g., “You’re being very helpful”, “You’re giving us the kind of information we need to make this product better”).
Try not talking much when the participant is carrying out tasks. Well-timed, deliberate periods of silence elicit thoughtful, accurate responses and insights, and build trust with participants.
Do not encourage opinions in briefings or during moderation. Asking participants “What are you thinking?” when they are silent is preferrable to asking leading or probing questions.
Ensure the usability test sessions follow a clear structure.
Speak up if you think there is a problem with the brief or the tasks provided to you by the client.
Learn from your peers. Watching sessions run by other moderators can be very insightful and help you re-evaluate your own moderating technique.
Be open to criticism and review your own work. Qualities such as humility and openness to criticism are important preconditions for improving the quality of moderated test sessions. Practitioners should be review colleagues' sessions and ask for regular reviewing of their skills.