Unmoderated usability testing

Learn more about what unmoderated usability testing is, and how you can use it to develop and optimize the user experience for your intended audience.

Tip:

Need to get started with unmoderated usability testing fast? Check out our quick start guides:

What is unmoderated usability testing?

An unmoderated usability test allows you to evaluate the user's experience while interacting with a product, website, app, feature, or prototype. It helps your designers, product teams, and key stakeholders assess how intuitive and easy to use products are. It also allows you to test assumptions and identify potential problems and areas for improvement.

The goal is to identify issues you might have missed by asking real users to complete a series of usability tasks. Then, you analyze the results, success rate, and paths taken to completion. Ultimately, you can use this data to improve your designs, help users achieve their objectives, and enable users to have a positive product experience.

An unmoderated usability test is:
  • Unmoderated: The participant is completing the test on their own.
  • Remote: The session is conducted virtually via desktop, camera, and microphone. The session is recorded.
  • Self-paced: Participants complete the usability test in their own time, in their own environment, while the activity is open.

Alida's unmoderated usability testing solution lets you collect quantitative and qualitative data to assess your products' usability.

Quantitative data
  • Survey questions that precede and follow the usability test
  • Customizable task success question (for example, "Were you successful in completing the task?")
  • Task duration in minutes
Qualitative data
  • Screen recordings
  • Camera and microphone recordings (if enabled)
  • Transcripts of audio

Differences between unmoderated and moderated usability tests

Tip: Need to run a moderated test? Consider using video discussions, which allows one or more moderators to engage with a small group of users and collect qualitative feedback. Recordings, transcripts, and rich reporting capabilities are available.
Unmoderated usability test Moderated usability test
Moderation
  • There is no moderator present.
  • A moderator runs the session, interacts with participants, and asks questions.
Location
  • The session occurs virtually.
  • Screen recordings are captured.
  • If camera and microphone are enabled, video footage and audio are also captured.
  • The session can occur in-person or virtually.
  • The session may or may not be recorded.
Scheduling
  • Participants can complete the usability test in their own time while the activity is open.
  • Participants must be able to attend a session at a specific date and time.
Participants
  • Participants are interacting with your product while in their own environments. This is closer to a "real world" scenario.
  • The virtual, self-paced nature of unmoderated usability tests makes them more flexible for participants. Therefore, you can attract a more diverse range of participants who represent different backgrounds, regions, demographics, and abilities.
  • The virtual, self-paced nature of unmoderated usability tests also makes it easier to attract a greater overall number of participants.
  • If the session is in-person instead of virtual, participants are interacting with your products in an unfamiliar setting.
  • Participants have to be able to attend scheduled sessions, which can pose built-in obstacles. For example, for in-person sessions, scheduling and travel considerations would limit who could participate. For scheduled virtual sessions, scheduling a session during work or school hours could mean the difference between whether the potential participant could join or not.
Budget
  • Less expensive
Tip: Consider budgeting for incentives.

Usability tests are more time-intensive than typical surveys, and incentives are a nice way to motivate participants and thank them for their participation. For more information about incentive options in the Alida platform, see Incentives.

  • Costlier
Tip: Consider budgeting for incentives.

Usability tests are more time-intensive than typical surveys, and incentives are a nice way to motivate participants and thank them for their participation. For more information about incentive options in the Alida platform, see Incentives.

Benefits
  • Ideal for testing high-fidelity prototypes or when you're further along in the design process
  • Can also be used to test navigation, prototypes, hypotheses, and product messaging
  • Better fit for tight budgets.
  • Can be combined with quantitative data collection in a way that feels organic.
  • Easier to engage with more diverse participants, and more participants overall.
  • An experienced facilitator can deep-dive into unplanned topics that emerge in-the-moment, and follow organic conversational threads to uncover unexpected insights.
Limitations and considerations

  • The lack of an in-person moderator means you have to take more care in designing the activity because participants are left to their own devices. They have no one to turn to if they need clarification or guidance.

    For this reason, you need to ensure the tasks are focused and the instructions are clear. For more information about best practices around formulating questions, see Writing effective unmoderated usability test questions.

  • Have a clear idea of the possible paths participants can take to complete the task, and the hypotheses you want to validate.
  • Moderated usability testing may attract less diverse test participants. For virtual sessions, scheduling conflicts can pose a constraint. For in-person sessions, researchers need to rely on physical proximity or travel to implement their tests.

Best practices for unmoderated usability testing

In general, keep these considerations in mind as you conduct unmoderated usability tests:

  • Challenge your assumptions.

    You may think you know how users will navigate through a design and where the pitfalls might be, but will your users think the same? This is your chance to test your hypotheses.

  • Decide on your metrics to measure.

    Out of the box, you can report on the following metrics:

    • Task success question: Were participants successful in completing the task, yes or no?
      Tip: The task success question is customizable. "Were you successful in completing the task?" is a common example, but this does not have to be the question you pose to participants.
    • Task duration: What was the average time participants needed to complete the task, in minutes?
  • The metrics you need may influence your overall activity design. For example, if you want to filter results based on whether participants have prior experience using your product, you may need to include a "Have you used this product before?" question before the usability tasks.
  • Get consent.

    Unmoderated usability testing involves recording participants through their devices, so it's crucial that participants give their consent explicitly before they begin.

    Luckily, collecting consent is built into Alida's unmoderated usability testing solution. Participants cannot proceed without doing so.

  • Run a pilot test of your activity.

    We highly recommend finding internal stakeholders to do a dry run of your unmoderated usability test, or doing a soft launch before distributing the activity to your full list of participants. This way, you validate the clarity of the exercise and ensure you're capturing the metrics you need so that when you do the full distribution, you are not "wasting" completes.

  • Be inclusive and recruit participants that represent your entire customer base.

    You want to make sure your designs are intuitive for all potential users of different backgrounds, demographics, and abilities.