Usability testing how many participants

Small iterative usability tests reduce certain kinds of risks: timeliness of feedback, impact of feedback, and time to market. In particular, small tests early in the design process can help the team converge more quickly on usable designs. But when lives are at stake, there are other risks to consider. Tests must be designed to find uncommon but critical usability problems.

For example, take medical devices. That is a summative or validation test, and requires about 25 participants per user group FDA, Even during formative testing sample sizes closer to 10 are more common Wiklund, We know one researcher who routinely tests 15 or 20 participants per group.

This mitigates two types of risks: not only the use-safety risk to the human user, but also the business risk of missing critical problems, failing validation, and having to wait months to re-test and resubmit the product for regulatory approval!

Those of us developing unregulated systems and products might still choose larger sample sizes of The larger number may be more persuasive to stakeholders. It may allow us to handle more variability within each user group. Or it may be a small incremental cost to the project Nielsen, Consider using a larger sample size. It depends on assumptions about problem discoverability and has implications for design process and business risk.

Assess these three factors to determine whether five participants is enough for you:. At the very least, make sure to assess how variable your participants are. If they differ significantly in expertise or in typical tasks, recruit at least five participants per user group.

I really think the idea that a 5 person test is appropriate has substantially hurt the industry. It started the whole wave of discount approaches that give the feeling that UX is being done, when it is not. And how can we say we have identified a problem because ONE person has an issue? How sure are we that the one person who is challenged by a task is not just having a bad day? Or is an odd person?

We need to move in the other direction. We need to test more people. We need to test a wider range of tasks. If UX is a major business imperative we need to invest in quality work.

A part of that is running more participants across more tasks when we do testing. All rights reserved Privacy policy Terms and conditions for public training courses. HFI reserves the right, at its discretion, to change, modify, add or remove portions of this Privacy Policy at any time by posting such changes to this page. You understand that you have the affirmative obligation to check this Privacy Policy periodically for changes, and you hereby agree to periodically review this Privacy Policy for such changes.

The continued use of the Website following the posting of changes to this Privacy Policy constitutes an acceptance of those changes. Users may adjust their web browser software if they do not wish to accept cookies. To withdraw your consent after accepting a cookie, delete the cookie from your computer.

HFI believes that every User should know how it utilizes the information collected from Users. The Website is not directed at children under 13 years of age, and HFI does not knowingly collect personally identifiable information from children under 13 years of age online.

Please note that the Website may contain links to other websites. These linked sites may not be operated or controlled by HFI. HFI is not responsible for the privacy practices of these or any other websites, and you access these websites entirely at your own risk. HFI recommends that you review the privacy practices of any other websites that you choose to visit. If User is from the European Union or other regions of the world with laws governing data collection and use that may differ from U.

Any such personally identifiable information provided will be processed and stored in the United States by HFI or a service provider acting on its behalf.

By providing your personally identifiable information, User hereby specifically and expressly consents to such transfer and processing and the uses and disclosures set forth herein. In the course of its business, HFI may perform expert reviews, usability testing, and other consulting work where personal privacy is a concern. Users browsing the Website without registering an account or affirmatively providing personally identifiable information to HFI do so anonymously.

Otherwise, HFI may collect personally identifiable information from Users in a variety of ways. If a User communicates with HFI by e-mail or otherwise, posts messages to any forums, completes online forms, surveys or entries or otherwise interacts with or uses the features on the Website, any information provided in such communications may be collected by HFI. HFI may also collect information about how Users use the Website, for example, by tracking the number of unique views received by the pages of the Website, or the domains and IP addresses from which Users originate.

While not all of the information that HFI collects from Users is personally identifiable, it may be associated with personally identifiable information that Users provide HFI through the Website or otherwise. If the User opts out of certain services, User information may still be collected for those services to which the User elects to subscribe. For those elected services, this Privacy Policy will apply. HFI may also use information collected through the Website for research regarding the effectiveness of the Website and the business planning, marketing, advertising and sales efforts of HFI.

HFI does not sell any User information under any circumstances. HFI may disclose personally identifiable information collected from Users to its parent, subsidiary and other related companies to use the information for the purposes outlined above, as necessary to provide the services offered by HFI and to provide the Website itself, and for the specific purposes for which the information was collected.

HFI may disclose personally identifiable information in order to protect the rights, property or safety of a User or any other person. HFI may disclose personally identifiable information to investigate or prevent a violation by User of any contractual or other relationship with HFI or the perpetration of any illegal or harmful activity. HFI may also disclose aggregate, anonymous data based on information collected from Users to investors and potential partners.

Finally, HFI may disclose or transfer personally identifiable information collected from Users in connection with or in contemplation of a sale of its assets or business or a merger, consolidation or other reorganization of its business. For example, during the first stage, you might test with five participants, and the primary intent of the testing would be to catch any show-stoppers. If, at any stage, you catch show-stoppers or produce statistically significant results, you can terminate the study early and allocate your time and financial resources elsewhere.

This iterative approach, though uncommon, is often optimal and fits well with the philosophy of user-centered design and agile approaches. However, it can be harder to predict the time and money your study would require, which can make things difficult from a management angle.

There is no one-size-fits-all solution for determining the optimal number of participants for a usability study. Rather, we should think more in terms of ranges like those shown in Figure 1. Problem-discovery studies, which are subjective in nature, typically require between three and twenty participants, with five to ten being a good baseline. Generally, the number of participants should increase with study complexity and product criticality, but decrease with design novelty. For comparative studies—which are typically more objective than problem-discovery studies because of their heavy reliance on metrics—group sizes of between eight and 25 participants typically provide valid results, with ten to twelve being a good baseline.

Generally, group size should increase if you want statistically significant results; punctuated studies can be an efficient way of achieving this goal. Whatever the number of participants you use for a particular study, you should always understand the assumptions, limitations, and risks that are associated with your decision.

Caulton, D. National Institute of Standards and Technology. Retrieved October 1, Faulkner, L. Landauer, Thomas K. Martin G. Amsterdam: North Holland, Macefield, Ritch. Nielsen, Jakob. Nielsen, Jakob, and Thomas K. Perfetti, Christine, and Lori Landesman. Spyridakis, J. Turner, Carl W. Lewis, and Jakob Nielsen. Virzi, R. And discovered the real joy and value of user research—without worrying about numbers at all.

Author completely missed the paper of Borsci et al. It is a big lack in the reference list and in the text. To me the paper was a brilliant example of how you can handle the sample size. There is a problem with this approach, as described in this post. I would also be careful about adding participants until a result becomes significant, because this greatly increases the likelihood of getting a significant result due to pure chance. These insights PDF come from experimental psychology, but I believe they should apply here as well.

Janet M. Six helps companies design easier-to-use products within their financial, time, and technical constraints. The proceedings of conferences on Graph Drawing, Information Visualization, and Algorithm Engineering and Experiments have also included the results of her research.

Read More. Ritch has worked in UX design since Ritch has lectured at the Masters level in five countries—on user-centered design, UX design, usability engineering, IT strategy, business analysis, and IT development methodology. He also has numerous internationally recognized qualifications in IT-related training and education. His international experience spans 15 countries. Ritch presently heads Ax-Stream, an approved Axure training partner. Internal staff may be used for pilot testing since you are testing the technology and the flow of the test and the data is not factored into the final results.

Internal staff should never be used to supplement during testing. Nielsen outlines the number of participants that you need based on a number of case studies:. If you are testing in the federal space, please review OMB guidelines related to the Paperwork Reduction Act for usability testing. For diagnostic usability testing, six to eight users of a given target audience are usually enough to uncover the major problems in a product.

Note: If you plan to do iterative repeated usability testing over the course of developing the site, you will need to recruit a new group for each test. You will need to factor that into your planning, recruitment, and budgeting.

Participant screeners are composed of questions that will help those recruiting for your test to rule individuals in or out of contention. They may be as simple as gender and age or as complex as your target audience dictates. For examples, please see our screener templates. A successful recruit is one that meets the criteria, appears for testing and is able to complete the test.

A good recruiter will screen, schedule and remind the participants about their test appointment to assure all of their recruits are successful.

If needs be, you may also engage the recruiter to handle additional administrative duties such as administering incentives for participants i. There will be a fee for additional services so it will be best to discuss any additional services needed by your team during your initial discussions with the recruiter.

If the team has access to representative users you can recruit from those individuals. If the team does not have access to representative users, you will have to hire a commercial recruiting company.