Content uploaded by James R. Lewis
Author content
All content in this area was uploaded by James R. Lewis on Aug 07, 2014
Content may be subject to copyright.
A preview of the PDF is not available
Problem Discovery in Usability Studies: A Model Based on the Binomial Probability Formula.
... Jeffrey (1994) mentioned that in order to obtain generalised results, the test needs 10 to 12 participants per user testing, according to classic experiment test recommendations. Lewis (1993) and Virzi (1990) agree that the first few users discover the majority of the problems; their data did not demonstrate that frequency and severity are connected. ...
... We agree with the common assumption that the need to improve products must be balanced with constraints such as cost and schedule [11,26]. However, when the goal is to improve rather than assess products, concerns about cost and schedule should not dominate discussions of planned usability research. ...
While some practitioners have argued that five users are enough to conduct a usability study, others advocate larger sample sizes or formulas to determine the appropriate number. Although productive, this debate has largely ignored the distinction between formative and summative research leaving many practitioners unable to clearly articulate the circumstances that determine whether a small or large sample is required. This has led to an overemphasis of quantitative measures at the expense of qualitative insight and the specific practice of relying on numerous observations of a usability issue to establish validity. In our view, accounts of user difficulty that include a description of the problem along with its potential cause and impact do not require large sample sizes to drive meaningful design change. By addressing arguments central to this debate, we intend to clarify the appropriate uses of the usability study methodology and improve the credibility and impact of usability professionals in practical settings. Author Keywords
The topic of "how many users" is of great interest to usability specialists who need to balance project concerns over ROI and timelines with their own concerns about designing usable interfaces. The panel will review the current controversies over usability test sample size, test validity, and reliability.
"Really, how many users do you need to test? Three answers, all different."---User Experience, Vol. 4, Issue 4, 2005
ResearchGate has not been able to resolve any references for this publication.