ArticlePDF Available

Toward a Usability Evaluation of BCIs

Article

Toward a Usability Evaluation of BCIs

Abstract

If BCIs could be considered and evaluated as assistive technology facilitating life activities, they could avoid dissatisfaction and prevent abandonment. We tested two Java BCI prototypes, based on the Thought Translation Device and the P300 Speller, on 61 participants with different computer skills performing a Copy Spelling Task. We then administered usability and cognitive workload questionnaires. The results have shown significant differences in the number of performed errors, in satisfaction, and in the cognitive workload invested in the task. We found that the Thought Translation Device was more error-resistant, less stressful, and more satisfactory for the users.
121
International Journal of Bioelectromagnetism www.ijbem.org
Vol. 13, No. 3, pp. 121 - 122, 2011
Toward a Usability Evaluation of BCIs
Emanuele Pasqualotto
ab
, Alessandro Simonetta
ac
, Veronica Gnisci
a
, Stefano Federici
ad
,
Marta Olivetti Belardinelli
ae
aInteruniversity Center for Research on Cognitive Processing in Natural and Artificial Systems
(ECoNA), Sapienza University of Rome, Rome, Italy
b Institute of Medical Psychology and Behavioral Neurobiology, University of Tübingen,
Tübingen, Germany
c INAIL – Consulenza per l’Innovazione Tecnologica, Rome, Italy
d Department of Human and Education Sciences, University of Perugia, Perugia, Italy e
Correspondence: Emanuele Pasqualotto, Institute of Medical Psychology and Behavioral Neurobiology, University of Tübingen,
Gartenstrasse 29, D-72074 Tübingen, Germany. E-mail: pasqualotto.emanuele@gmail.com
A
bstract. If BCIs could be considered and evaluated as assistive technology facilitating life activities,
they could avoid dissatisfaction and prevent abandonment. We tested two keyboard controlled Java BCI
prototypes, based on the Thought Translation Device and the P300 Speller, on 61 participants with
different computer skills performing a Copy Spelling Task. We then administered usability and
cognitive workload questionnaires. The results have shown significant differences in the number o
f
performed errors, in satisfaction, and in the cognitive workload invested in the task. We found that the
Thought Translation Device was more error-resistant, less stressful, and more satisfactory for the users.
K
eywords: BCI, Usability, Cognitive Workload, NASA TLX, SUS
1. Introduction
Since BCIs could provide paralysed people a means of communication, this technology should be
considered and treated as an assistive technology, which facilitates life activities [Pasqualotto et al.,
2009]. Therefore, BCIs need an assessment process to match person and technology, in order to avoid
dissatisfaction and abandonment [Scherer, 2000]. The tradition of Human Computer Interaction has
already given us most of the tools needed to analyse and evaluate technology; although researches
aimed at evaluating human factors in BCIs’ design are still scarce. In this study we aim at assessing the
usability of two BCI prototypes by measuring the interaction with the systems in context, considering
the performance, cognitive workload, and satisfaction of non-disabled users.
2. Material and Methods
2.1 Interface prototype
In order to evaluate only visual layout features, removing all the possible EEG-related issues, we
developed two keyboard controlled Java prototypes. We decided to investigate the Language Support
Program used in the Thought Translation Device (TTD) [Neumann, et al., 2003] and the P300 Speller (P3S)
[Sellers and Donchin, 2006]. While the TTD is based on Slow Cortical Potentials and allows users to select
b
etween two alternatives (e.g. yes/no), the P3S is based on P300 event related potential and allows users to
use a flashing 6 × 6 matrix of characters. We developed the prototypes in order to be controlled using a PC
keyboard, considering the original nature of the input: TTD is controlled by using 2 keys, one for each half o
f
the alphabet, P3S is controlled by using one key to be selected when the target appears. We chose these two
interfaces because, due to the EEG components used, they represent two opposite ways of using an input in a
speller device: dichotomic versus multi-choice.
2.2 Procedure
We tested our BCI prototypes on 61 participants (31 F, 30 M; age: mean 22.3, min 18, max 38) with
different computer skills, randomly assigned to one of two groups, one for each prototype.
122
Using a Copy Spelling Task (CST), wherein users are asked to write on the screen through the interface,
our participants wrote in one session a list of ten randomized words (111 characters) previously selected by
means of a frequency list of words in Italian written language. We then administered the ‘System Usability
Scale’ (SUS) [Brooks, 1996] in order to measure usability and the NASA TLX [Hart and Staveland, 1988] in
order to measure the cognitive workload. Due to the rigid time constrain of the P3S flashing matrix compared
to the constrain-free use of the TTD, bit rates were not comparable and we did not use it as a measure.
3. Results
We analysed the data using a GLM procedure and a 2 × 2 × 3 design, where the variables were the
Interface (PS3 and TTD), Gender, and Computer Skills (Low, Medium, High). We used the Number o
f
Errors as a measure; moreover, we differentiated between two kinds of error: a choice-
b
ased error and a
recognition-
b
ased error. While the first one represents a wrong choice, a choice that led to the wrong
result, the second one is a lack of recognition of the letter to select, which led to pursue the research o
f
the desired target. We used the SUS and NASA TLX values as measures. We also analysed each single
component of the NASA TLX. Based on recent findings [Borsci et al., 2009] we analysed SUS as a
global score and as a two-factor scale, Usability and Learnability. Gender and Computer Skills did not
interact with the measures, showing that performance and satisfaction are not influenced by users’
p
ersonal characteristics. On the contrary, the Interface played a major role, interacting with the Number
of Errors (F(1,60) = 24.8, p < .00; LSP: M 9.87, SD 8.76; P3S: M 32.35, SD 21.58), with the Usability
score (F(1,60) = 8.7 p = .005; LSP: M 68.25, SD 15.77; P3S: M 55.00, SD 17.26), and with the
cognitive workload invested in the use (F(1,60) = 4.08 p = .049; LSP: M 43.01, SD 20.93; P3S: M
56.11, SD 26.12). Moreover, we found significant differences in the score of the wrong choice (F(1,60)
= 30.1 p < .00; LSP: M 5.70, SD 4.80; P3S: M 26.84, SD 19.23) and in the NASA-TLX Frustration
score (F(1,60) = 5.9 p = .019; LSP: M 6.39, SD 8.95; P3S: M 13.83, SD 13.19). The analysis of the two
factors of SUS confirmed a difference in the Usability score (F(1,60) = 10,76, p = .002; LSP: M 52.42,
SD 13.32; P3S: M 40.48, SD 14.02) but not in the Learnability score.
4. Discussion
Although several years have passed since the beginning of BCI research, there is still a lack in the
use of usability and psychological methods to evaluate this technology. Even though an improvement is
required, our methodology allowed us to begin an exploration of the usability of BCIs. The Thought
Translation Device represents a binary modality that we can find even in sensory-motor-rhythm base
d
BCI, in contrast with the multi-choice possibility of the P300 Speller. Disregarding any possible EEG-
based issue, the TTD allowed our participants to be more accurate, reducing the possibility of error.
Recognition is usually considered easier than recollection, and our data are consistent with this
assumption. In fact, we did not find any difference in the recognition-
ased errors, while we found the
in the choice-based error. Consistently with the error scores, NASA-TLX has shown that participants
found the TTD less stressful and less frustrating. In the same way, the SUS global scores and the
Usability factor have shown that the TTD is generally considered preferable.
References
Borsci S, Federici S, Lauriola M. On the dimensionality of the System Usability Scale: a test of alternative measurement models.
Cognitive Processing, 10(3), 193-197, 2009. (journal articles) Brook J. SUS: a “quick and dirty” usability scale, in Usability
Evaluation in Industry. Jordan PW, Thomas B, Weerdmeester BA, McClelland AL, Editors. Taylor and Francis, London, 1996,
189-194.
Hart, SG, Staveland, LE. Development of NASA-TLX (task load index): Results of empirical and theoretical research, in Human
mental workload. Hancock PA and Meshkati N, Editors. North-Holland Press, Amsterdam, 1988, 139-184. Neumann N, Kübler A,
Kaiser J, Hinterberger T, Birbaumer N. Conscious perception of brain states: mental strategies for brain– computer
communication. Neuropsychologia, 41(8): 1028-1036, 2003. (journal articles)
Pasqualotto E, Simonetta A, Federici S, Olivetti Belardinelli M. Usability Evaluation of BCIs, in Assistive Technology Research
Series Vol. 25. Assistive Technology from Adapted Equipment to Inclusive Environments -AAATE 2009. Emiliani PL, Burzagli
L, Como A., Gabbanini F., and Salminen AL Editors. Cooper RA, Editor. IOS Press, Amsterdam, 2009, 882.
Scherer MJ. Living in the state of stuck: How technology impacts the lives of people with disabilities. Brookline Books,
Cambridge, 2000. Sellers EW, Donchin E. A P300-based brain–computer interface: Initial tests by ALS patients. Clinical
N
europhysiology 117(3): 538-548, 2006. (journal articles)
... For instance, "ALS patients as a group often show cognitive impairment in working memory, attention response inhibition, naming and other functions" (Hinterberger et al., 2005). Differences in cognitive ability need to be accounted for when devising the operational context of a BCI and screening tools and assessment procedures have been proposed (Cheng et al., 2002;Felton et al., 2012;Pasqualotto et al., 2011). Also, some users require high levels of support and consequently, BCI may be judged more feasible for some rather than others; hence mechanisms for assessing whether to accept a subject onto a programme or otherwise have been proposed (Neumann and Kübler, 2003) and attention to defining and specifying user needs within the BCI context has been addressed, (Zickler et al., 2009). ...
... The key aspects of the NASA TLX are mental demands, physical demands, temporal demands, performance, effort and frustration -a numerical value being determined for each and an overall value calculated. Pasqualotto et al. (2011) also used the NASA TLX to measure cognitive workload for BCI, combined with system usability scale for usability. ...
Article
Brain-computer interfaces (BCIs) aim to provide a mechanism by which individuals with severe forms of paralysis can communicate their wishes to others or gain control over devices in their environment. This paper describes the evaluation of an interface designed for use with BCI, which is dedicated to providing a flexible and extensible framework for controlling multiple devices within a domestic environment. A structured format for assessing the usability aspects of this interface is presented. The goal-question-metric (GQM) approach derived from measurement in software engineering was used, providing a mechanism that can be readily adapted to assess other interfaces in assistive technology. The assessment was based on interaction of five participants currently using assistive technology who provided an initial representation of the target user group. Their input, in addition to guidance from a lead user, was used to influence subsequent design choices and directions and highlight the importance of user-centred design within BCI development.
... Still, the workload was higher than in the validation study conducted with healthy end-users (M = 52, see [16]). It was also higher than in P300-based BCIs for communication tested with healthy participants (M = 56, see [27]; M = 49, see [28]; M = 32 and M = 46, see [29]), and with patients undergoing lower-limb rehabilitation (M = 20, see [30]). In a study with non-locked-in Figure 7. Randomly generated paintings with BP2, all obtained by selecting 300 random selections in the painting matrices. ...
... end-users relying on assistive technology for their daily computer use (M = 30, see [31]) the NASA-TLX subscale 'physical demand' was close to 0 while JT rated it at 17 (26% of the workload). Importantly, all but two participants of those studies [16,[27][28][29][30] were between 18 and 34 years old. As evoked by JT and HS in the QUEST2.0 and in the post-evaluation feedback, the physical demand in the workload might not only be due to the pain in the neck, but it might also be explained by the state of his disease and his age (64), which inevitably lead to physical fatigue. ...
Article
Full-text available
Brain Painting is a brain-computer interface (BCI) application that allows for painting on a virtual canvas without requiring motor input. Following the user-centered design, we developed Brain Painting V2 (BP2) for two end-users with ALS (‘HP’ and ‘JT’) who use Brain Painting independently at home (BP1). After including line drawing and multiple shapes in BP2, we report the evaluation of 27 home use days by JT over 3.5 months. On scales from 0 to 10, the end-user indicated good satisfaction M = 7.1 (SD = 1.7), good enjoyment M = 6.9 (SD = 1.9) and low frustration M = 2.3 (SD = 2.5). Workload as measured once with the NASA Task Load Index was substantial (65/100). Satisfaction as assessed once with the QUEST2.0 was high (4.8/5) and maximum (5/5) for the BCI-specific aspects. Level of control predicted satisfaction, indicating the importance of maintaining high BCI accuracy. As results are based on one end-user only, they cannot be generalized to the population of potential BP2 end-users. However, the high satisfaction and the constant independent home use of BP2 by end-user JT illustrates that we provided JT with a BCI-controlled application which matches his needs and demonstrates the benefit of adopting user-centered design in BCI development.
... Through a recent literature review, we have shown that although there is a large amount of studies on BCIs, most of them focus on methodological approaches, neglecting usability aspects. 9,21 Most of the BCI studies, in fact, do not consider that users often discard assistive technology after only a few attempts and that personal factors (eg, mood, motivation, belief, and predispositions) should be taken into account because they can serve as both barriers and facilitators in sustaining efficient use. 22,23 Although the effectiveness of BCIs in terms of character selection is assessed in most studies, the efficiency and the satisfaction from the users' perspective are not always addressed; however, they are part of a recent area of growth. ...
... The copy-spelling task is widely used in the BCI field. 12,21,32 The general idea behind this task is to provide the user with a set of words or sentences to write with the communication interface. In each session with the BCI or the eye tracker, participants were asked to select a sequence of predefined characters in the matrix, representing common Internet tasks. ...
Article
Full-text available
Background. Eye trackers are widely used among people with amyotrophic lateral sclerosis, and their benefits to quality of life have been previously shown. On the contrary, Brain-computer interfaces (BCIs) are still quite a novel technology, which also serves as an access technology for people with severe motor impairment. Objective. To compare a visual P300-based BCI and an eye tracker in terms of information transfer rate (ITR), usability, and cognitive workload in users with motor impairments. Methods. Each participant performed 3 spelling tasks, over 4 total sessions, using an Internet browser, which was controlled by a spelling interface that was suitable for use with either the BCI or the eye tracker. At the end of each session, participants evaluated usability and cognitive workload of the system. Results. ITR and System Usability Scale (SUS) score were higher for the eye tracker (Wilcoxon signed-rank test: ITR T = 9, P = .016; SUS T = 12.50, P = .035). Cognitive workload was higher for the BCI (T = 4; P = .003). Conclusions. Although BCIs could be potentially useful for people with severe physical disabilities, we showed that the usability of BCIs based on the visual P300 remains inferior to eye tracking. We suggest that future research on visual BCIs should use eye tracking-based control as a comparison to evaluate performance or focus on nonvisual paradigms for persons who have lost gaze control. © The Author(s) 2015.
... For example, Lorenz and colleagues investigated several usability aspects such as accuracy, workload and learnability with regards to hybrid BCIs in a sample of 12 healthy participants [14]. Likewise, Pasqualotto and colleagues compared accuracy, usability, and workload in two BCIs controlled by different input signals, namely slow cortical and event-related potentials [17,18]. In contrast, McCane and colleagues included a large sample of N525 end-users with amyotrophic lateral sclerosis in all stages of the disease, but evaluated only accuracy [19]. ...
... A weighting procedure combines the individual scores for each dimension into one total score. It has previously been used to assess workload of healthy subjects during BCI operation [14,18]. ...
Article
Full-text available
Albeit research on brain-computer interfaces (BCI) for controlling applications has expanded tremendously, we still face a translational gap when bringing BCI to end-users. To bridge this gap, we adapted the user-centered design (UCD) to BCI research and development which implies a shift from focusing on single aspects, such as accuracy and information transfer rate (ITR), to a more holistic user experience. The UCD implements an iterative process between end-users and developers based on a valid evaluation procedure. Within the UCD framework usability of a device can be defined with regard to its effectiveness, efficiency, and satisfaction. We operationalized these aspects to evaluate BCI-controlled applications. Effectiveness was regarded equivalent to accuracy of selections and efficiency to the amount of information transferred per time unit and the effort invested (workload). Satisfaction was assessed with questionnaires and visual-analogue scales. These metrics have been successfully applied to several BCI-controlled applications for communication and entertainment, which were evaluated by end-users with severe motor impairment. Results of four studies, involving a total of N = 19 end-users revealed: effectiveness was moderate to high; efficiency in terms of ITR was low to high and workload low to medium; depending on the match between user and technology, and type of application satisfaction was moderate to high. The here suggested evaluation metrics within the framework of the UCD proved to be an applicable and informative approach to evaluate BCI controlled applications, and end-users with severe impairment and in the locked-in state were able to participate in this process.
... For example, Lorenz and colleagues investigated several usability aspects such as accuracy, workload and learnability with regards to hybrid BCIs in a sample of 12 healthy participants [14]. Likewise, Pasqualotto and colleagues compared accuracy, usability, and workload in two BCIs controlled by different input signals, namely slow cortical and event-related potentials [17,18]. In contrast, McCane and colleagues included a large sample of N525 end-users with amyotrophic lateral sclerosis in all stages of the disease, but evaluated only accuracy [19]. ...
... A weighting procedure combines the individual scores for each dimension into one total score. It has previously been used to assess workload of healthy subjects during BCI operation [14,18]. ...
... For example, BCI systems are mainly evaluated in the perspective of efficiency, such as classification accuracy and communication speed [57]. Usability dimensions, including efficiency measures, workload and satisfaction, have been assessed by Riccio Pasqualotto [60,61] investigated error rate and learnability of keyboard-controlled BCI prototypes. However, there were some notable limitations in these previous studies. ...
Article
Full-text available
A new Brain-Computer Interface (BCI) technique, which is called a hybrid BCI, has recently been proposed to address the limitations of conventional single BCI system. Although some hybrid BCI studies have shown promising results, the field of hybrid BCI is still in its infancy and there is much to be done. Especially, since the hybrid BCI systems are so complicated and complex, it is difficult to understand the constituent and role of a hybrid BCI system at a glance. Also, the complicated and complex systems make it difficult to evaluate the usability of the systems. We systematically reviewed and analyzed the current state-of-the-art hybrid BCI studies, and proposed a systematic taxonomy for classifying the types of hybrid BCIs with multiple taxonomic criteria. After reviewing 74 journal articles, hybrid BCIs could be categorized with respect to 1) the source of brain signals, 2) the characteristics of the brain signal, and 3) the characteristics of operation in each system. In addition, we exhaustively reviewed recent literature on usability of BCIs. To identify the key evaluation dimensions of usability, we focused on task and measurement characteristics of BCI usability. We classified and summarized 31 BCI usability journal articles according to task characteristics (type and description of task) and measurement characteristics (subjective and objective measures). Afterwards, we proposed usability dimensions for BCI and hybrid BCI systems according to three core-constructs: Satisfaction, effectiveness, and efficiency with recommendations for further research. This paper can help BCI researchers, even those who are new to the field, can easily understand the complex structure of the hybrid systems at a glance. Recommendations for future research can also be helpful in establishing research directions and gaining insight in how to solve ergonomics and HCI design issues surrounding BCI and hybrid BCI systems by usability evaluation.
... The system might recognize user actions perfectly but might be difficult to use. For this purpose, some methods to evaluate the ease of use (i.e., usability) of BCIs [Nam et al. 2009;Pasqualotto et al. 2011] have been proposed. However, usability alone does not imply user satisfaction either. ...
Article
Brain-computer interfaces (BCIs) are often evaluated in terms of performance and seldom for usability. However in some application domains, such as entertainment computing, user experience evaluation is vital. User experience evaluation in BCI systems, especially in entertainment applications such as games, can be biased due to the novelty of the interface. However, as the novelty will eventually vanish, what matters is the user experience related to the unique features offered by BCI. Therefore it is a viable approach to compare BCI to other novel modalities, such as a speech or motion recognizer, rather than the traditional mouse and keyboard. In the study that we present in this article, our participants played a computer game with a BCI and an automatic speech recognizer (ASR), and they rated their expectations and experiences for both modalities. Our analysis on subjective ratings revealed that both ASR and BCI were successful in satisfying participants' expectations in general. Participants found speech control easier to learn than BCI control. They indicated that BCI control induced more fatigue than they expected.
... To evaluate the workload of each condition, participants were asked to fill in the NASA TLX questionnaire (NASA Human Performance Research Group, 1987) after the calibration phase of each condition. It was introduced as a measure of usability in BCI by Pasqualotto et al. [35] and Riccio et al. [36], and it has been used to compare the workload for a visual and an auditory speller by Kä thner et al. [37]. It was used here as a multidimensional rating procedure to derive an overall workload score based on a weighted average of ratings on six sub-scales (Mental workload, Physical workload, Temporal workload, Performance, Effort needed and Frustration) for each condition. ...
Article
Full-text available
For Brain-Computer Interface (BCI) systems that are designed for users with severe impairments of the oculomotor system, an appropriate mode of presenting stimuli to the user is crucial. To investigate whether multi-sensory integration can be exploited in the gaze-independent event-related potentials (ERP) speller and to enhance BCI performance, we designed a visual-auditory speller. We investigate the possibility to enhance stimulus presentation by combining visual and auditory stimuli within gaze-independent spellers. In this study with N = 15 healthy users, two different ways of combining the two sensory modalities are proposed: simultaneous redundant streams (Combined-Speller) and interleaved independent streams (Parallel-Speller). Unimodal stimuli were applied as control conditions. The workload, ERP components, classification accuracy and resulting spelling speed were analyzed for each condition. The Combined-speller showed a lower workload than uni-modal paradigms, without the sacrifice of spelling performance. Besides, shorter latencies, lower amplitudes, as well as a shift of the temporal and spatial distribution of discriminative information were observed for Combined-speller. These results are important and are inspirations for future studies to search the reason for these differences. For the more innovative and demanding Parallel-Speller, where the auditory and visual domains are independent from each other, a proof of concept was obtained: fifteen users could spell online with a mean accuracy of 87.7% (chance level <3%) showing a competitive average speed of 1.65 symbols per minute. The fact that it requires only one selection period per symbol makes it a good candidate for a fast communication channel. It brings a new insight into the true multisensory stimuli paradigms. Novel approaches for combining two sensory modalities were designed here, which are valuable for the development of ERP-based BCI paradigms.
Article
Multi-device user-interaction behavior creates a need to design cross-device menus that support users to re-locate menu items efficiently and effectively after the transition from one device to another. We conducted two laboratory experiments to investigate the effects of specific designs of cross-device menus on post-transitioning user performance (selection time and selection error). In Experiment 1, a total of 72 participants moved between 32 desktop and mobile menu interfaces (representing 16 different designs of cross-device menus) to re-locate menu items. We investigated the effect on user performance of the cross-device menu configuration (complementary versus redundant items), menu length (short versus medium), menu item order (consistent versus inconsistent), and menu format (text-only versus text-and-icon items). The results showed significant main effects of configuration, length and item order with large partial eta-squared sizes. Users performed better with redundant menus, short menus, and menus with consistent item order than they did with complementary menus, medium menus, and menus with inconsistent item order. In Experiment 2, 20 participants moved between eight desktop and mobile menu interfaces (representing four different designs of cross-device menus) to re-locate menu items. We focused on the effect on user performance of the menu layout (horizontal-desktop and vertical-mobile versus vertical-desktop and vertical-mobile) at two different lengths (short versus medium). The results showed no significant effect of menu layout, and a significant effect of length with a small partial eta-squared size. Our study provides a foundation to researchers interested in further investigation of cross-device menu designs, and to practitioners in designing cross-device menus that allow more efficient and effective re-locating of menu items when moving between devices.
Conference Paper
The EU-project "Tools for Brain-Computer Interaction" (TOBI) aims at developing practical technology for brain-computer interaction To elucidate the most urgent needs and requirements of potential users 77 persons with disabilities from Austria, Italy, and Germany were assessed with a newly developed questionnaire Participants had a high degree of experience in using assistive technology (AT) Despite high satisfaction ratings with current AT solutions there was still a need for better or alternative AT solutions in the areas of manipulation, communication, computer access/entertainment, and environmental control The results also confirm the relevance of the set of applications the TOBI project is pursuing to develop Main end-user requirements of new AT were 'functionality" possibility of independent use", and 'easiness of use
Article
Full-text available
The System Usability Scale (SUS), developed by Brooke (Usability evaluation in industry, Taylor & Francis, London, pp 189-194, 1996), had a great success among usability practitioners since it is a quick and easy to use measure for collecting users' usability evaluation of a system. Recently, Lewis and Sauro (Proceedings of the human computer interaction international conference (HCII 2009), San Diego CA, USA, 2009) have proposed a two-factor structure-Usability (8 items) and Learnability (2 items)-suggesting that practitioners might take advantage of these new factors to extract additional information from SUS data. In order to verify the dimensionality in the SUS' two-component structure, we estimated the parameters and tested with a structural equation model the SUS structure on a sample of 196 university users. Our data indicated that both the unidimensional model and the two-factor model with uncorrelated factors proposed by Lewis and Sauro (Proceedings of the human computer interaction international conference (HCII 2009), San Diego CA, USA, 2009) had a not satisfactory fit to the data. We thus released the hypothesis that Usability and Learnability are independent components of SUS ratings and tested a less restrictive model with correlated factors. This model not only yielded a good fit to the data, but it was also significantly more appropriate to represent the structure of SUS ratings.
Living in the state of stuck: How technology impacts the lives of people with disabilities. Brookline Books Sellers EW, Donchin E. A P300-based brain–computer interface: Initial tests by ALS patients
  • Scherer
Scherer MJ. Living in the state of stuck: How technology impacts the lives of people with disabilities. Brookline Books, Cambridge, 2000. Sellers EW, Donchin E. A P300-based brain–computer interface: Initial tests by ALS patients. Clinical Neurophysiology 117(3): 538-548, 2006. (journal articles)
(journal articles) Brook J SUS: a “quick and dirty” usability scale, in Usability Evaluation in Industry
  • Pw Jordan
  • B Thomas
  • Ba Weerdmeester
  • Mcclelland
Cognitive Processing, 10(3), 193-197, 2009. (journal articles) Brook J. SUS: a “quick and dirty” usability scale, in Usability Evaluation in Industry. Jordan PW, Thomas B, Weerdmeester BA, McClelland AL, Editors. Taylor and Francis, London, 1996, 189-194
(journal articles) Brook J. SUS: a "quick and dirty" usability scale
Cognitive Processing, 10(3), 193-197, 2009. (journal articles) Brook J. SUS: a "quick and dirty" usability scale, in Usability Evaluation in Industry. Jordan PW, Thomas B, Weerdmeester BA, McClelland AL, Editors. Taylor and Francis, London, 1996, 189-194.
Assistive Technology from Adapted Equipment to Inclusive Environments-AAATE
  • E Pasqualotto
  • A Simonetta
  • S Federici
  • Olivetti Belardinelli
Pasqualotto E, Simonetta A, Federici S, Olivetti Belardinelli M. Usability Evaluation of BCIs, in Assistive Technology Research Series Vol. 25. Assistive Technology from Adapted Equipment to Inclusive Environments-AAATE 2009. Emiliani PL, Burzagli L, Como A., Gabbanini F., and Salminen AL Editors. Cooper RA, Editor. IOS Press, Amsterdam, 2009, 882.
Brookline Books, Cambridge, 2000. Sellers EW, Donchin E. A P300-based brain-computer interface: Initial tests by ALS patients
  • M J Scherer
Scherer MJ. Living in the state of stuck: How technology impacts the lives of people with disabilities. Brookline Books, Cambridge, 2000. Sellers EW, Donchin E. A P300-based brain-computer interface: Initial tests by ALS patients. Clinical Neurophysiology 117(3): 538-548, 2006. (journal articles)