Available via license: CC BY 4.0
Content may be subject to copyright.
Citation: Formicola, R.; Amici, C.;
Mor, M.; Bissolotti, L.; Borboni, A.
Design of Medical Devices with
Usability in Mind: A Theoretical
Proposal and Experimental Case
Study Using the LEPRE Device.
Designs 2023,7, 9. https://doi.org/
10.3390/designs7010009
Academic Editor: Subburaj
Karupppasamy
Received: 25 November 2022
Revised: 27 December 2022
Accepted: 3 January 2023
Published: 6 January 2023
Copyright: © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Design of Medical Devices with Usability in Mind:
A Theoretical Proposal and Experimental Case Study Using the
LEPRE Device
Raffaele Formicola 1, Cinzia Amici 1,* , Maurizio Mor 2, Luciano Bissolotti 3and Alberto Borboni 1, *
1Department of Mechanical and Industrial Engineering, University of Brescia, Via Branze 38,
25123 Brescia, Italy
2Polibrixia s.r.l., Via Branze 43, 25123 Brescia, Italy
3Domus Salutis Rehabilitation Clinic, Teresa Camplani Foundation, Via Lazzaretto 3, 25123 Brescia, Italy
*Correspondence: cinzia.amici@unibs.it (C.A.); alberto.borboni@unibs.it (A.B.)
Abstract:
Usability is a critical product feature and is required for widespread market adoption.
Standards on usability are highly focused on evaluation procedures and specific aspects, such as
software issues or human–machine interaction, whereas the relative scientific literature is very
normative oriented. The few methodological works dealing with usability either consider it as one
of the many attributes that a particular project must satisfy or present very general methods. No
design methods systematically oriented toward the integration of usability and usability-related
constraints have been developed to date. This paper presents a usability-oriented model for the
design of medical devices and its application to the design of LEPRE, a medical device for upper- and
lower-limb robotic rehabilitation. Two methods were used to assess the device’s usability: interviews
with experts to outline qualitative evaluations and System Usability Scale (SUS) questionnaires on
eight physiotherapists, two physiatrists, and 12 patients, enabling a quantitative assessment. Results
support the intention of providing an integrated methodological approach to be applied from the
early stages of the project, thus saving time and costs, leading to a more linear product development
for this application.
Keywords:
usability; design process; medical device; robotic rehabilitation; normative framework;
product development
1. Introduction
The usability of a product or service, in conjunction with its safety, is a critical feature
required for widespread market adoption of the good itself [
1
], and even more in the field
of medical devices. Nonetheless, few scientific works and regulations address usability
methodologically, considering it one of the many attributes that the project must satisfy [
2
],
or otherwise proposing very general methods [3].
According to International Organization for Standardization (ISO), usability is defined
as “the extent to which a system, product or service can be used by specified users to
achieve specified goals with effectiveness, efficiency and satisfaction in a specified context
of use” [
3
]. The concept of usability is, therefore, extremely broad, and the scientific
literature includes further attributes to the list of ISO parameters, interpreting and offering
additional shades of meaning to the ISO definition, such as: ease of use, learnability,
flexibility, attitude, and memorability [
2
,
4
]. Furthermore, since usability is applicable to
both software and hardware, it encompasses a wide range of applications, including apps,
websites, computer mice, smartphones and machinery. For this paper, usability is meant as
described by the ISO definition.
The scientific literature on usability, particularly in the medical field, is very normative-
oriented [
5
–
7
]. Themes such as usability risk assessment [
8
], usability reports [
9
], documen-
tation [
10
] and protocols [
11
] are of particular interest. Moreover, most of the articles are
Designs 2023,7, 9. https://doi.org/10.3390/designs7010009 https://www.mdpi.com/journal/designs
Designs 2023,7, 9 2 of 27
software-related, with an emphasis on the human–machine interface [
12
–
14
], evaluation
procedures [
15
–
18
], and numerous but very specific application cases [
19
–
21
]. For instance,
Pei et al. [
22
] described the usability of a novel robotic bilateral arm rehabilitation device
for patients with stroke [
22
], Surma-aho et al. [
23
] analyzed the usability issues of operating
room devices [
23
]. Heinemann et al. [
24
] focused on the usability of medical devices for
patients with diabetes who are visually impaired or blind [
24
]. As the application cases
cited above demonstrate, in addition to being circumstantial to a particular medical field
(stroke, operating room and diabetes), they often do not cover the entire relative range of
design possibilities. For instance, regarding stroke technologies, the first application exam-
ple [
22
] is restricted to a robotic bilateral arm rehabilitation device, while the case regarding
diabetes [
24
] is confined to patients who are visually impaired or blind. Apart from the
application cases, all the usability-related literature mainly concerns software applications
and the human–machine interface so it does not extend beyond what is outlined in the
standards. Most of all, it does not address the topic of usability with a methodological
approach.
For the European market, the normative framework that governs the usability domain
of devices in the medical field is comprised of three references: (i) IEC 62366-1:2015 “Ap-
plication of usability engineering to medical devices” [
25
], which describes a process for a
manufacturer to analyze, specify, develop, and evaluate the usability of a medical device in
terms of safety; (ii) ISO/TR 16982:2002 “Usability methods supporting human-centered
design” [
26
], which provides information on human-centered usability methods that can be
used for design and evaluation; and (iii) ISO 9241-11:2018 “Ergonomics of human–system
interaction—Part 11: Usability: Definitions and concepts” [
3
], which specifies a framework
for understanding the concept of usability and applies to situations where people use
interactive systems, products (including industrial and consumer products) and services.
As the last reference provides a global perspective on usability, the first two are primarily
concerned with the software domain.
In other words, the normative framework analyzes usability almost as a final output of
the design process for a medical device, providing indications, for instance, on its post-hoc
evaluation. On the contrary, fewer indications are provided about how to actively manage
usability-related aspects during the design evolution itself. In addition, the fulfillment of
usability-based requirements could strongly affect the final design of a device, especially
in the medical field, and it is a given that every unexpected change applied to a process
introduces additional costs to a project, e.g., personnel hours, economic costs, or resources.
Accordingly, several works describe design techniques and methodologies devoted
to identifying device requirements [
27
,
28
], as well as detecting or anticipating, if possible,
project criticalities. Among them, design-for-X [
29
] and open innovation approaches [
30
]
especially emerged, although not specifically devoted to the integration of usability in the
design process.
Other approaches, such as Design Thinking and Human-Centered Design, represent
valuable, complementary design tools [
31
,
32
], even if they do not properly consider design
models, as they are not contextualized into all the various stages of product design. In
fact, Design Thinking is a new paradigm for dealing with problems in many professions,
most notably Information Technology (IT) and Business [
33
,
34
], but does not result in either
specifically to the medical field or referable to one or more specific stages of the product
design. Likewise, Human-Centered Design is an approach to interactive systems develop-
ment that aims at making systems usable and useful by focusing on the users, their needs
and requirements, and by applying human factors/ergonomics and usability knowledge
and techniques [35,36]. Although Human-Centered Design does provide guidance on the
key aspects to be considered throughout the entire design process, significantly affecting
the definition of requirements, this tool is broad and does not define specific engineering
tasks. Therefore, according to the authors’ knowledge, no design methods systematically
oriented towards the integration of usability and usability-related constraints at all the
phases of the design process have been developed up to date.
Designs 2023,7, 9 3 of 27
In this context, the objective of this study is to propose a usability-oriented model for
the design of medical devices based on an extension of the concept of usability, which is
intended not only as an outcome, but more significantly, as an integrated methodological
approach to be applied during the early stages of the project. Therefore, this approach
embraces all the stages of the design process. Involves a more linear product development
and allows for reducing time and costs
Furthermore, the innovative aspect of this process is also grounded in its practicality
and broad applicability, distinguishing it, for example, from the generic iterative approach
proposed by the standards [35].
The usability-oriented model will be discussed by demonstrating its application in
the design process of the LEPRE medical device for robotic rehabilitation of the lower and
upper limbs. This robotic device has an end-effector-based architecture and can perform
rehabilitation treatment in different modalities: passive (CPM), active-assisted, or active
with biofeedback. It is composed of three main groups, i.e., the frame, the main body, and
the monitor. A compact differential system in the main body enables the implementation
of any motion profile in the desired plane. The extended description of the device is given
in Section 2.2: Case Study: LEPRE Rehabilitation Device.
2. Method
2.1. Usability-Oriented Model for the Design of Medical Devices
The Usability-Oriented Model for the Design of Medical Devices is represented in
Figure 1. This results from more than 15 years of research and analysis of scientific literature,
regulatory framework, and practical applications. It is the outcome of two implementation
steps that transformed and incrementally improved an original model. This original model,
called the “soft open innovation model” [
37
], has been applied in the last 10 years to the
design of the five medical devices depicted in previous literature by the authors [
37
] and
was implemented from two points of view: (a) through a more detailed specification of the
actors involved in the various stages of the design process, and (b) with the integration
of the regulatory aspect [
38
]. The result of this first improvement is the model named
“Open innovation-inspired model for the design process of medical devices” [
39
]. From
this second model, the role of usability was investigated and contextualized within the
design process, obtaining the approach proposed in this work.
According to the Usability-Oriented Model for the Design of Medical Devices, shown
in Figure 1, the design process begins with a specific unmet need that requires innovation
and progresses through the following phases to the final prototype: “Concept Definition”,
“Simulations”, “Prototype”, “On-Field Evaluation”, and “Final Prototype”.
In some cases, the emergence of usability issues itself may stimulate the launch of a
new innovative project. Referring to Figure 1, the “Goal” is not included in the usability
area (green dashed box) because this is not a condition that always occurs. Nevertheless,
usability desiderata can be the reason for defining the “Goal”, i.e., the model’s starting
point. For example, suppose products on the market in a given area of use have poor
usability with regard to a specific function. In that case, an industrial company could cover
that market segment by developing a new innovative solution that satisfies users in terms
of usability.
Focusing communication on usability is critical during the “Concept Definition” and,
in particular, when identifying “Functional Specifications” of the project by addressing with
stakeholders the main features that are involved, such as functionality, ease of use, comfort,
flexibility and learnability. Interviews are important for eliciting usability requirements [
40
],
particularly during the “Concept Definition” and subsequent phases. Interviews are
more effective when conducted in groups [
41
,
42
] because conflicting requirements are
reduced. These are primarily used in the search for requirements rather than for a posteriori
evaluation, emphasizing the usability paradigm shift once more.
Designs 2023,7, 9 4 of 27
Designs 2023, 7, x FOR PEER REVIEW 4 of 31
Figure 1. Usability-Oriented Model for the Design of Medical Devices.
According to the Usability-Oriented Model for the Design of Medical Devices, shown
in Figure 1, the design process begins with a specific unmet need that requires innovation
and progresses through the following phases to the final prototype: “Concept Definition”,
“Simulations”, “Prototype”, “On-Field Evaluation”, and “Final Prototype”.
In some cases, the emergence of usability issues itself may stimulate the launch of a
new innovative project. Referring to Figure 1, the “Goal” is not included in the usability
area (green dashed box) because this is not a condition that always occurs. Nevertheless,
usability desiderata can be the reason for defining the “Goal”, i.e., the model’s starting
point. For example, suppose products on the market in a given area of use have poor us-
ability with regard to a specific function. In that case, an industrial company could cover
that market segment by developing a new innovative solution that satisfies users in terms
of usability.
Focusing communication on usability is critical during the “Concept Definition” and,
in particular, when identifying “Functional Specifications” of the project by addressing
with stakeholders the main features that are involved, such as functionality, ease of use,
Figure 1. Usability-Oriented Model for the Design of Medical Devices.
The detailed design and early production phases, represented in the scheme by “Simu-
lations” and “Prototype”, remain primarily engineering-related tasks. As a result, usability
improvements are less likely to emerge at these stages. However, there are times when
technicians become aware of issues during the detailed design stages of a project, like in
the case of previously unknown conflicts of requirements. These issues must be discussed
with stakeholders, and as a result, new usability requirements may emerge.
Once the prototype has been created, it must pass a test before being used in the field.
This check reveals “unexpressed requirements”, or requirements deemed necessary by a
stakeholder but not expressed until that point. In most cases, these requirements are related
to usability.
If the prototype passes the test, the next phase is the “On-Field Evaluation”. This is the
most important phase after the “Concept Definition” because it validates the prototype’s
usability. Typically, if the previous steps of the design process were correctly carried out
using the usability-oriented approach; in this case, the usability requirements emerge that
do not imply design revisions but rather improvements to the prototype under evaluation.
Designs 2023,7, 9 5 of 27
The “Final Prototype” is the result of passing the “On-field evaluation” check, although
the implementation of further minor design changes could still occur also after the “Final
Prototype” is completed.
The Beginning-of-Life (BOL) phase concludes with the production of the “Final Proto-
type”. The usability-validated product is then allowed to proceed through the remaining
phases of the product life cycle [
43
]: the Middle-of-Life (MOL), which includes use and
maintenance, and the End-of-Life (EOL), which comprises disposal or recycling.
Devices intended for the European market are partially guided to be compliant with
usability since they require the set of mandatory procedures of the CE marking to be
freely commercialized within the European market itself. Nevertheless, additional usability
requirements that lead to improvements may also emerge at subsequent stages, like in
after-sales phases, for example, during the post-market surveillance procedures required
by the normative framework
2.2. Case Study: LEPRE Rehabilitation Device
The LEPRE Rehabilitation Device is an end-effector-based robotic device for upper-
and lower-limb rehabilitation. It is characterized by a compact differential system, with
two degrees of freedom that enable the implementation of any motion profile in the desired
plane [
44
], and can work either through pre-set exercises or with exercises customized ad
hoc by the clinician. The device is suitable for rehabilitation in passive modality (Con-
tinuous Passive Motion, CPM), active assisted, and active modality with biofeedback.
Figure 2depicts the current CE-marked version of the device, validated for use in clinical
practice [45,46].
Designs 2023, 7, x FOR PEER REVIEW 6 of 31
Figure 2. LEPRE Rehabilitation Device (Polibrixia s.r.l., Brescia, Italy).
Rehabilitation of both upper and lower limbs is due to the device's central body's
ability to tilt and thanks to the interchangeability of accessories, allowing for two setups:
upper-limb configuration (Figure 3a) and lower-limb configuration (Figure 3b).
The device can also perform rehabilitation treatment in different modalities: passive
(CPM), active-assisted or active with biofeedback.
(a) (b)
Figure 3. From the left (modified with permission from [47], person’s face obscured): (a) LEPRE
Rehabilitation Device in the upper-limb setup: central body in upper position and accessories that
allow hand grip, and (b) LEPRE Rehabilitation Device in the lower-limb setup: central body in lower
position and accessories that allow the anchorage of the feet.
2.3. Suitability Analysis of Usability Evaluation Methods
Several tools exist to elicit requirements. Especially in relation to the usability field,
the standard ISO/TR 16982:2002 [26] is fundamental. This technical report provides an
overview of existing usability methods that can be used on their own or in combination
to support design and evaluation. Methods presented in the standard include the obser-
vation of users, performance-related measurements, critical incidents analysis, question-
naire, interviews, thinking aloud, collaborative design and evaluation, creativity methods,
document-based methods, model-based approaches, expert evaluation and automated
evaluation. In addition, the standard provides guidelines for choosing the optimal method
based on the life-cycle process. Specifically, the standard divides the Life Cycle of a project
into three macro phases: acquisition and supply processes, development process, and op-
eration and maintenance processes. Since the model presented in this paper concerns the
Figure 2. LEPRE Rehabilitation Device (Polibrixia s.r.l., Brescia, Italy).
Rehabilitation of both upper and lower limbs is due to the device’s central body’s
ability to tilt and thanks to the interchangeability of accessories, allowing for two setups:
upper-limb configuration (Figure 3a) and lower-limb configuration (Figure 3b).
The device can also perform rehabilitation treatment in different modalities: passive
(CPM), active-assisted or active with biofeedback.
Designs 2023,7, 9 6 of 27
Designs 2023, 7, x FOR PEER REVIEW 6 of 31
Figure 2. LEPRE Rehabilitation Device (Polibrixia s.r.l., Brescia, Italy).
Rehabilitation of both upper and lower limbs is due to the device's central body's
ability to tilt and thanks to the interchangeability of accessories, allowing for two setups:
upper-limb configuration (Figure 3a) and lower-limb configuration (Figure 3b).
The device can also perform rehabilitation treatment in different modalities: passive
(CPM), active-assisted or active with biofeedback.
(a) (b)
Figure 3. From the left (modified with permission from [47], person’s face obscured): (a) LEPRE
Rehabilitation Device in the upper-limb setup: central body in upper position and accessories that
allow hand grip, and (b) LEPRE Rehabilitation Device in the lower-limb setup: central body in lower
position and accessories that allow the anchorage of the feet.
2.3. Suitability Analysis of Usability Evaluation Methods
Several tools exist to elicit requirements. Especially in relation to the usability field,
the standard ISO/TR 16982:2002 [26] is fundamental. This technical report provides an
overview of existing usability methods that can be used on their own or in combination
to support design and evaluation. Methods presented in the standard include the obser-
vation of users, performance-related measurements, critical incidents analysis, question-
naire, interviews, thinking aloud, collaborative design and evaluation, creativity methods,
document-based methods, model-based approaches, expert evaluation and automated
evaluation. In addition, the standard provides guidelines for choosing the optimal method
based on the life-cycle process. Specifically, the standard divides the Life Cycle of a project
into three macro phases: acquisition and supply processes, development process, and op-
eration and maintenance processes. Since the model presented in this paper concerns the
Figure 3.
From the left (modified with permission from [
47
], person’s face obscured): (
a
) LEPRE
Rehabilitation Device in the upper-limb setup: central body in upper position and accessories that
allow hand grip, and (
b
) LEPRE Rehabilitation Device in the lower-limb setup: central body in lower
position and accessories that allow the anchorage of the feet.
2.3. Suitability Analysis of Usability Evaluation Methods
Several tools exist to elicit requirements. Especially in relation to the usability field,
the standard ISO/TR 16982:2002 [
26
] is fundamental. This technical report provides an
overview of existing usability methods that can be used on their own or in combination
to support design and evaluation. Methods presented in the standard include the obser-
vation of users, performance-related measurements, critical incidents analysis, question-
naire, interviews, thinking aloud, collaborative design and evaluation, creativity methods,
document-based methods, model-based approaches, expert evaluation and automated
evaluation. In addition, the standard provides guidelines for choosing the optimal method
based on the life-cycle process. Specifically, the standard divides the Life Cycle of a project
into three macro phases: acquisition and supply processes, development process, and
operation and maintenance processes. Since the model presented in this paper concerns the
development phase, only the macro phase of development was considered for choosing the
most suitable usability evaluation method. In turn, the development process is divided by
the standard into three stages: requirements analysis, architectural design, and qualification
testing.
In fact, no linear correspondence between the model presented in this paper and the
model proposed by the standard can be established. In fact, the latter defines the stages of
architectural design and qualification testing as follows:
•
Architectural design: “During the design phases, usability methods will be imple-
mented to confirm, modify or refine the previous findings”;
•
Qualification testing: “Is the activity where usability methods are applied to test the
match with the requirements”.
In the Usability-Oriented Model for the Design of Medical Devices, on the other
hand, there is no clear distinction between these two phases because both are evaluated
in the various testing phases: the fulfillment of initial requirements is provided by the
qualification testing phase of the standard, and the architectural design phase provides the
possibility of changes and improvements.
Figure 4synthesizes the correspondence between the different phase nomenclatures
of the standard and the model presented in this paper.
For each of the three stages of development, the standard indicates which usability
evaluation methods are most appropriate (see Table 1for an extract from the standard).
For the requirement analysis phase, the standard identifies the following methods
as most recommended (i.e., methods marked “++”): observation of users, questionnaires,
interviews, and thinking aloud. The observation of users, in the sense of a collection of
information about the behavior and the performance of users, is not feasible in our case
Designs 2023,7, 9 7 of 27
as both clinical actors and patients cannot fully interact with the device as a functioning
prototype has not yet been produced at this early stage of defining the specifications. For
the same reason, thinking aloud, intended as the verbalization by users of their ideas,
beliefs, expectations, doubts, and discoveries during their use of the system under test, is
not applicable as there is no complete prototype. Moreover, valid questionnaires in these
early stages are qualitative and aimed at eliciting opinions on the device. Consequently,
interviews are the best tool for evaluating usability in the requirements analysis phase as
they allow for greater flexibility than the qualitative questionnaires and enable unexpressed
opinions and requirements to be more easily revealed.
Designs 2023, 7, x FOR PEER REVIEW 8 of 31
Figure 4. Correspondences between the Usability-Oriented Model for the Design of Medical Devices
and the development process defined by the ISO ISO/TR 16982:2002 standard.
For each of the three stages of development, the standard indicates which usability
evaluation methods are most appropriate (see Table 1 for an extract from the standard).
Figure 4.
Correspondences between the Usability-Oriented Model for the Design of Medical Devices
and the development process defined by the ISO ISO/TR 16982:2002 standard.
Designs 2023,7, 9 8 of 27
Table 1. Usability evaluation methods recommended based on the development stage.
Development Stages
Usability Evaluation Methods
Observation of Users
Performance-Related Measurements
Critical Incident Analysis
Questionnaires
Interviews
Thinking Aloud
Collaborative Design and Evaluation
Creativity Methods
Document-Based Methods
Model-Based Methods
Expert Evaluation
Automated Evaluation
Requirements analysis ++ + + ++ ++ ++ + + + + +
Architectural design + ++ + + ++ + ++ ++ + + +
Qualification testing + ++ + ++ ++ + + + + + +
Legend
++ Recommended
+ Appropriate
empty cell Neutral
Since, for the model presented in this paper, the two phases of architectural design and
qualification testing are not separate, a combined evaluation was carried out to understand
the best usability evaluation method. The method that would be most suitable for these
phases is the performance-related measurements, as it has the higher mark (“++”) for both
phases. The commonly used quantifiable performance measurements include time spent
to complete a task, number of errors, and number of tasks that can be completed within a
predefined duration. It was decided not to adopt this method for the evaluation of usability
in these phases because the large number of tasks that can be performed with the device
would have made a complete evaluation difficult and because this tool does not allow an
evaluation of the overall device but in terms of human–machine interface only.
Consequently, the methods most suited by the standard to assess usability at these
stages are questionnaires, interviews, thinking aloud and document-based methods
(i.e., those marked with a “++” and a “+”). The latter method was not considered, as
it involves pre-existing documentation, which is, however, not available as the first pro-
totype is being evaluated. Interviews were continually held with healthcare personnel,
who also reported the patients’ feelings about the device in their care. Interviews also
made it possible to replace and supplement the thinking aloud, avoiding specific tests and
leaving the clinical actors free to use the device according to instructions for use and clinical
practice. Questionnaires are mostly employed in this phase, as they allow a quantitative
evaluation of usability concerning the whole device and can be applied to both clinical
actors and patients.
Following the instructions of the standard, the tools that emerged as the best ones to
support the design and evaluate usability were interviews, especially in the initial stages
indicated by the standard under the heading of “Development–Requirements Analysis,”
and questionnaires, especially in the final stages indicated by the standard under the head-
ing of “Development–Architectural Design” and “Development–Qualification Testing”.
Therefore, in the present work, the Usability-Oriented Model for the Design of Medical
Devices has been applied to the LEPRE Rehabilitation Device (Polibrixia s.r.l., Padova,
Italy), and interviews and questionnaires were used to evaluate the usability of the device.
Designs 2023,7, 9 9 of 27
2.4. Usability Evaluation Protocol
Two methods available in the scientific literature [
6
,
15
] and the standards [
26
] were
used to evaluate the obtained usability of the LEPRE Rehabilitation Device: interviews to
outline a first qualitative evaluation and questionnaires to enable a quantitative assessment.
The subjects provided the proper consent for the data treatment.
The interviews were conducted with more than 15 physiatrists (at least 45 years
old) and physiotherapists (at least 35 years old) with at least 10 years of experience from
different rehabilitation clinical facilities, treating patients with acute-phase impairments
with orthopedic and neurological genesis by the same trained operator. The interviews
were conducted throughout the different stages of the device design. In particular, as the
phases progressed, the interview questions went from being open-ended to increasingly
specific ones. For example, in the early stages of “Concept Definition”, typical questions
were: “What are the rehabilitation modalities that you expect from the device?” or “What
are the trajectories that the device should be able to perform?”. Differently, at the end of the
prototyping phases, the usual questions were: “Which usability criticalities do you think can
be present with the implemented structural shape?” or “Can the usability of the 3D mouse
be improved by modifying its position?”. A multidisciplinary team of three engineers
analyzed the answers provided by the experts to capture suggestions or desiderata aimed at
improving the device’s usability. In the current work, usability desiderata were considered
new design constraints and translated into device requirements.
For the quantitative analysis, the System Usability Scale (SUS) [
48
–
50
] was used, i.e., a
survey composed of ten statements with a final score ranging from 0 to 100, which is the
most valid and widely used questionnaire, both scientifically and industrially [
51
]. In partic-
ular, the applied SUS totally coincides with the original version proposed by Brooke [
48
–
50
],
but for the translation of the statements in the Italian language. Appendix Apresents the
detail of the questions used and the formula to evaluate the SUS value. Generally, a usabil-
ity rating is considered fully acceptable when it exceeds the SUS value of 70: Table 2collects
the adopted convention for interpreting the ratings, compliant with the values proposed by
Bangor et al. in [52].
In particular, the SUS questionnaire tool was used to assess the usability of the device
at three distinct stages of the development of LEPRE. The device usability was evaluated for
the first time after the first prototype was created: this moment is indicated in the following
elaborations as T0. As it was impossible to assess patient usability because the device had
not been CE marked yet, the questionnaire was distributed to eight physiotherapists and
two physiatrists. After the device was CE-marked, a clinical investigation in accordance
with ISO 14155:2020 [
53
] was carried out to evaluate the LEPRE device’s usability on
patients. At that moment, as indicated in the following elaborations as T1, the questionnaire
was submitted to 12 post-stroke patients, both ischemic (50%) and hemorrhagic (50%), and
once again to the clinical operators: the same eight physiotherapists and the same two
physiatrists. The SUS evaluation was then repeated on the same patients at the end of the
second therapy session with the device to capture possible evolution in the evaluation.
This moment is indicated in the following elaborations as T2. For both the questionnaires,
the SUS values were analyzed with descriptive statistics, and in particular, computing
minimum and maximum values (min and max), as well as mean and standard deviation
(SD) values, of the data samples (combined in mean
±
SD). In addition to the exposed
parameters, the p-value calculated according to the Mann–Whitney U-test with a level
α= 0.05 is also reported to analyze statistical consistency.
In addition to the overall SUS values as a function of groups of participants (physio-
therapists, physiatrists, and patients), which represent the primary endpoints, SUS results
were also analyzed by considering the two components of which it is composed: Usability
(SUS(U)) and Learning (SUS(L)). In fact, among the 10 questions of which the SUS ques-
tionnaire is constituted, in accordance with what is reported in the literature [
22
], eight
questions (Q1, Q2, Q3, Q5, Q6, Q7, Q8, Q9) are intended to express an evaluation in the
usability domain (SUS(U)) and two (Q4, Q10) in the learning domain (SUS(L)). Appendix A
Designs 2023,7, 9 10 of 27
presents the detail of the formula to evaluate the value of the SUS components: Usability
(SUS(U)) and Learning (SUS(L)).
Moreover, the Mann–Whitney U-test and single-sample t-test with significance level
α
= 0.05 are applied to compare the physiotherapists’ scores to the average SUS score
obtained by Sauro [
54
], who used a population of 5000 different products that have been
sold in the market.
Regarding the quantitative analyses, the flowcharts of the activities conducted to
obtain the resulting SUS values are presented in Figures 5and 6.
Table 3summarizes the descriptive data of the participants involved in the study.
Designs 2023, 7, x FOR PEER REVIEW 12 of 31
(a) (b)
Figure 5. Flowcharts of clinical operators: (a) physiotherapists’ flowchart; (b) physiatrists’ flowchart.
Figure 5.
Flowcharts of clinical operators: (
a
) physiotherapists’ flowchart; (
b
) physiatrists’ flowchart.
Designs 2023,7, 9 11 of 27
Designs 2023, 7, x FOR PEER REVIEW 13 of 31
Figure 6. Patients’ flowchart.
Table 3 summarizes the descriptive data of the participants involved in the study.
Table 3. Study population description.
Category Parameter Value
Patients
Age 54.08 ± 9.77
Sex M = 8, F = 4
Stroke type (I = Ischemic, H = Hemorrhagic) I = 6, H = 6
Physiotherapists
Age 44.88 ± 7.79
Sex M = 5, F = 3
Experience (years) 19.63 ± 7.48
Physiatrists
Age 48.00 ± 1.41
Sex M = 1, F = 1
Experience (years) 18.50 ± 2.12
3. Results
The Usability-Oriented Model for the Design of Medical Devices was adopted to de-
velop the LEPRE Rehabilitation Device (Polibrixia s.r.l., Italy).
Figure 6. Patients’ flowchart.
Table 2.
Adopted convention for the interpretation of the SUS values as Adjective Ratings. The
interpretation complies with Bangor et al.’s classification [52].
Adjective SUS Value
Worst Imaginable 12.5
Awful 20.3
Poor 35.7
OK 50.9
Good 71.4
Excellent 85.5
Best Imaginable 90.9
Table 3. Study population description.
Category Parameter Value
Patients
Age 54.08 ±9.77
Sex M = 8, F = 4
Stroke type (I = Ischemic, H = Hemorrhagic) I = 6, H = 6
Physiotherapists
Age 44.88 ±7.79
Sex M = 5, F = 3
Experience (years) 19.63 ±7.48
Physiatrists
Age 48.00 ±1.41
Sex M = 1, F = 1
Experience (years) 18.50 ±2.12
Designs 2023,7, 9 12 of 27
3. Results
The Usability-Oriented Model for the Design of Medical Devices was adopted to
develop the LEPRE Rehabilitation Device (Polibrixia s.r.l., Padova, Italy).
The following are examples of changes, improvements, and implementations that
occurred in the LEPRE design process due to implementing the usability-oriented method.
These are usability requirements, in addition to those prescribed by regulations, that results
from the extension of the concept of usability foreshadowed by the model’s paradigm
change. The following examples are graphically depicted in Figure 5: Carter Shape,
Structural Shape, Mirror Function, and Mouse 3D Height. The figure also indicates the
suitability of the proposed model with respect to the project timeline, indicating at which
stage of the LEPRE design process the usability changes were performed.
Furthermore, usability evaluation tests were conducted at various stages of the device
development to assess the usability achieved thanks to the application of the usability-
oriented model. The moments when usability measurements were performed, i.e., T0, T1
and T2, are shown in blue in Figure 7.
Designs 2023, 7, x FOR PEER REVIEW 14 of 31
The following are examples of changes, improvements, and implementations that
occurred in the LEPRE design process due to implementing the usability-oriented
method. These are usability requirements, in addition to those prescribed by regulations,
that results from the extension of the concept of usability foreshadowed by the model’s
paradigm change. The following examples are graphically depicted in Figure 5: Carter
Shape, Structural Shape, Mirror Function, and Mouse 3D Height. The figure also indicates
the suitability of the proposed model with respect to the project timeline, indicating at
which stage of the LEPRE design process the usability changes were performed.
Furthermore, usability evaluation tests were conducted at various stages of the de-
vice development to assess the usability achieved thanks to the application of the usabil-
ity-oriented model. The moments when usability measurements were performed, i.e., T0,
T1 and T2, are shown in blue in Figure 7.
Figure 7. LEPRE Case Study: Application of the Usability-Oriented Model for the Design of Medical
Devices.
3.1. Carter Shape
The original LEPRE Carter was designed to have the shape depicted in Figure 8a to
maximize the internal spaces. However, during the early stages of the “Concept Defini-
tion”, the following usability requirement emerged from interviews with doctors and
physiotherapists: (u1) the device must allow patients to lean on its frontal part (circled in
green in the figure) during the rehabilitation therapy on the upper limbs. This necessity
Figure 7.
LEPRE Case Study: Application of the Usability-Oriented Model for the Design of Medical
Devices.
3.1. Carter Shape
The original LEPRE Carter was designed to have the shape depicted in Figure 8a
to maximize the internal spaces. However, during the early stages of the “Concept Defi-
nition”, the following usability requirement emerged from interviews with doctors and
Designs 2023,7, 9 13 of 27
physiotherapists: (u1) the device must allow patients to lean on its frontal part (circled in
green in the figure) during the rehabilitation therapy on the upper limbs. This necessity
does not arise from a specific pathology but might arise because of a particular physical
condition, prominent breasts, or because of the set law of motion, which involves reaching
the device workspace most distal from the patient’s trunk (a condition to be avoided in the
exercise setting).
Designs 2023, 7, x FOR PEER REVIEW 15 of 31
does not arise from a specific pathology but might arise because of a particular physical
condition, prominent breasts, or because of the set law of motion, which involves reaching
the device workspace most distal from the patient’s trunk (a condition to be avoided in
the exercise setting).
(a) (b)
Figure 8. From the left: (a) The Original LEPRE Carter; (b) the carter of the LEPRE Rehabilitation
Device in the CE marked version.
The newly designed LEPRE carter (Figure 8b) presents a radius of curvature of 90
mm, in contrast to the initially planned 8 mm. In this way, through simple geometric con-
siderations, it turns out that the reduction of the interior space (considering the four cor-
ners of the crankcase) results to be about 6900 mm
2
.
The carter usability requirement impacted aesthetics, safety, mechanical and elec-
tronic structure and device market share.
The first affected aspect is the aesthetics. For the carter to be useful, the radii of cur-
vature must be much more pronounced.
Instead, safety is strengthened on two fronts. First, mechanical safety improves, as
more pronounced radii of curvature allow for safer and more comfortable support of the
patient’s trunk (Figure 9). Above all, the new usability requirement necessitates the relo-
cation of the emergency push bottom (element in red in Figure 8), which would otherwise
be difficult to access by the operator (physician or physiotherapist) in the event of an
emergency, especially when the device is in the lower-limb rehabilitation therapy config-
uration.
Figure 8.
From the left: (
a
) The Original LEPRE Carter; (
b
) the carter of the LEPRE Rehabilitation
Device in the CE marked version.
The newly designed LEPRE carter (Figure 8b) presents a radius of curvature of 90 mm,
in contrast to the initially planned 8 mm. In this way, through simple geometric considera-
tions, it turns out that the reduction of the interior space (considering the four corners of
the crankcase) results to be about 6900 mm2.
The carter usability requirement impacted aesthetics, safety, mechanical and electronic
structure and device market share.
The first affected aspect is the aesthetics. For the carter to be useful, the radii of
curvature must be much more pronounced.
Instead, safety is strengthened on two fronts. First, mechanical safety improves,
as more pronounced radii of curvature allow for safer and more comfortable support
of the patient’s trunk (Figure 9). Above all, the new usability requirement necessitates
the relocation of the emergency push bottom (element in red in Figure 8), which would
otherwise be difficult to access by the operator (physician or physiotherapist) in the event
of an emergency, especially when the device is in the lower-limb rehabilitation therapy
configuration.
Designs 2023, 7, x FOR PEER REVIEW 16 of 31
Figure 9. Illustrative detail of the contact between the patient’s trunk and the carter in correspond-
ence with the radius of curvature.
Furthermore, the new usability requirement affected the mechanical and electronic
structure within the carter, particularly in terms of overall dimensions. More pronounced
curvature radii, the carter’s longitudinal dimensions being equal, imply a reduction in the
space available for the internal mechanical and electronic structure.
Finally, thanks to the possibility of the carter supporting the trunk, the inclusion cri-
teria of the device are expanded, allowing even patients who need frontal support to per-
form upper-limb rehabilitation therapies with the device.
3.2. Structural Shape
The Structural Shape of the LEPRE chassis was initially designed, as represented in
Figure 10a, to maximize the device stability. After the first prototype was built, the pri-
mary usability requirement from clinician interviews was: (u2) the ability to perform on-
oliteral rehabilitation therapies laterally with respect to the device's central body. This un-
expressed requirement was translated into the possibility of positioning the patient on the
wheelchair at the side of the device. For example, if the patient needed to perform reha-
bilitation exercises only on the right limb, they would position themselves to the left of
the device's central body. This functionality was not possible with the original structural
form. The structure currently implemented in the CE-marked device (Figure 10b) was de-
signed to fulfill the new requirement.
Figure 9.
Illustrative detail of the contact between the patient’s trunk and the carter in correspondence
with the radius of curvature.
Designs 2023,7, 9 14 of 27
Furthermore, the new usability requirement affected the mechanical and electronic
structure within the carter, particularly in terms of overall dimensions. More pronounced
curvature radii, the carter’s longitudinal dimensions being equal, imply a reduction in the
space available for the internal mechanical and electronic structure.
Finally, thanks to the possibility of the carter supporting the trunk, the inclusion
criteria of the device are expanded, allowing even patients who need frontal support to
perform upper-limb rehabilitation therapies with the device.
3.2. Structural Shape
The Structural Shape of the LEPRE chassis was initially designed, as represented in
Figure 10a, to maximize the device stability. After the first prototype was built, the primary
usability requirement from clinician interviews was: (u2) the ability to perform onoliteral
rehabilitation therapies laterally with respect to the device’s central body. This unexpressed
requirement was translated into the possibility of positioning the patient on the wheelchair
at the side of the device. For example, if the patient needed to perform rehabilitation
exercises only on the right limb, they would position themselves to the left of the device’s
central body. This functionality was not possible with the original structural form. The
structure currently implemented in the CE-marked device (Figure 10b) was designed to
fulfill the new requirement.
Designs 2023, 7, x FOR PEER REVIEW 17 of 31
(a) (b)
Figure 10. From the left: (a) the First LEPRE Structural Shape, (b) the Structural Shape of the LEPRE
Rehabilitation Device in the CE-marked version.
The new usability requirement for the LEPRE structural shape influenced aesthetics,
mechanical structure, safety, and electronic and software features.
Due to the revision of the structural shape of the chassis, the device aesthetics have
been drastically altered.
New load analyses and structural simulations were required from a mechanical
structural standpoint to develop a new reliable, and robust solution.
Furthermore, the device's stability must be tested to ensure safety according to stand-
ards [55]. As a result, new stability tests were required.
However, the new requirement had the greatest impact on electronics and software
functionality. In fact, it was necessary to implement the device and the accessories of new
electronic components capable of recognizing the working side of the device and imple-
menting all relevant software features. It was mandatory, for example, to prevent the set-
ting of a bilateral rehabilitation exercise when only one tool is mounted or to set a reha-
bilitation exercise on the incorrect side.
3.3. Mirror Function
During the “on-field evaluation”, the following new usability requirement emerged:
(u3) the clinical operator must be able to perform the device movements when setting the
trajectory foreseen by the rehabilitation exercise according to their position (relative ref-
erence system) rather than the device absolute reference system. The mirror function (Fig-
ure 11) was implemented to meet this requirement. This function allows users to mirror
motion commands, invert the clockwise and anticlockwise sense, and change the horizon-
tal translation direction on both the Mouse 3D and the virtual console.
Figure 10.
From the left: (
a
) the First LEPRE Structural Shape, (
b
) the Structural Shape of the LEPRE
Rehabilitation Device in the CE-marked version.
The new usability requirement for the LEPRE structural shape influenced aesthetics,
mechanical structure, safety, and electronic and software features.
Due to the revision of the structural shape of the chassis, the device aesthetics have
been drastically altered.
New load analyses and structural simulations were required from a mechanical struc-
tural standpoint to develop a new reliable, and robust solution.
Furthermore, the device’s stability must be tested to ensure safety according to stan-
dards [55]. As a result, new stability tests were required.
However, the new requirement had the greatest impact on electronics and software
functionality. In fact, it was necessary to implement the device and the accessories of
new electronic components capable of recognizing the working side of the device and
implementing all relevant software features. It was mandatory, for example, to prevent
Designs 2023,7, 9 15 of 27
the setting of a bilateral rehabilitation exercise when only one tool is mounted or to set a
rehabilitation exercise on the incorrect side.
3.3. Mirror Function
During the “on-field evaluation”, the following new usability requirement emerged:
(u3) the clinical operator must be able to perform the device movements when setting
the trajectory foreseen by the rehabilitation exercise according to their position (relative
reference system) rather than the device absolute reference system. The mirror function
(Figure 11) was implemented to meet this requirement. This function allows users to mirror
motion commands, invert the clockwise and anticlockwise sense, and change the horizontal
translation direction on both the Mouse 3D and the virtual console.
Designs 2023, 7, x FOR PEER REVIEW 18 of 31
Figure 11. Detail of the Mirror Function switch on the LEPRE virtual console.
The new usability requirement for the Mirror Function has changed how the software
operates and the device's safety.
The first domain affected by the new requirement was undoubtedly software, as it
was necessary to implement the previously unanticipated Mirror Feature.
However, the most significant impact was on device safety, particularly in the med-
ical field. The new function implies that the clinical operator better manages the funda-
mentals for the setting of robotic rehabilitation therapies. The new function prevents the
device from unwanted and potentially dangerous motions that could harm the patient.
3.4. Mouse 3D Height
Although the device was successfully operating at full capacity in the hospital after
receiving the CE mark, the periodic interviews revealed a new usability requirement: (u4)
Mouse 3D Height must be increased by a few centimeters. Therefore, a minor design re-
view was required (Figure 12), based on which the height of the Mouse 3D was raised by
40 mm.
Figure 12. Mouse 3D Height.
The new usability requirement for Mouse 3D Height affected both the mechanical
aspect and the ease of use.
A minor design review was performed to meet the requirement, resulting in mechan-
ical modifications to the monitor support.
Figure 11. Detail of the Mirror Function switch on the LEPRE virtual console.
The new usability requirement for the Mirror Function has changed how the software
operates and the device’s safety.
The first domain affected by the new requirement was undoubtedly software, as it
was necessary to implement the previously unanticipated Mirror Feature.
However, the most significant impact was on device safety, particularly in the medical
field. The new function implies that the clinical operator better manages the fundamentals
for the setting of robotic rehabilitation therapies. The new function prevents the device
from unwanted and potentially dangerous motions that could harm the patient.
3.4. Mouse 3D Height
Although the device was successfully operating at full capacity in the hospital after
receiving the CE mark, the periodic interviews revealed a new usability requirement: (u4)
Mouse 3D Height must be increased by a few centimeters. Therefore, a minor design review
was required (Figure 12), based on which the height of the Mouse 3D was raised by 40 mm.
The new usability requirement for Mouse 3D Height affected both the mechanical
aspect and the ease of use.
A minor design review was performed to meet the requirement, resulting in mechani-
cal modifications to the monitor support.
However, this new usability requirement’s major impact was improving the ease and
comfort of using the Mouse 3D.
Designs 2023,7, 9 16 of 27
Designs 2023, 7, x FOR PEER REVIEW 18 of 31
Figure 11. Detail of the Mirror Function switch on the LEPRE virtual console.
The new usability requirement for the Mirror Function has changed how the software
operates and the device's safety.
The first domain affected by the new requirement was undoubtedly software, as it
was necessary to implement the previously unanticipated Mirror Feature.
However, the most significant impact was on device safety, particularly in the med-
ical field. The new function implies that the clinical operator better manages the funda-
mentals for the setting of robotic rehabilitation therapies. The new function prevents the
device from unwanted and potentially dangerous motions that could harm the patient.
3.4. Mouse 3D Height
Although the device was successfully operating at full capacity in the hospital after
receiving the CE mark, the periodic interviews revealed a new usability requirement: (u4)
Mouse 3D Height must be increased by a few centimeters. Therefore, a minor design re-
view was required (Figure 12), based on which the height of the Mouse 3D was raised by
40 mm.
Figure 12. Mouse 3D Height.
The new usability requirement for Mouse 3D Height affected both the mechanical
aspect and the ease of use.
A minor design review was performed to meet the requirement, resulting in mechan-
ical modifications to the monitor support.
Figure 12. Mouse 3D Height.
3.5. Usability Evaluation
As expected, the performed interviews did not reveal quantitative values to be an-
alyzed but rather qualitative suggestions and requirements regarding functionality and
usability to be implemented in the device as modifications, improvements, or new func-
tions. In particular, it was possible to elicit requirements that involved: the modification of
the Carter Shape (u1) and the Structural Shape (u2); the improvement of the usability of
the Mouse 3D (u4); and the implementation of the new Mirror Function (u3). In addition,
the involved changes impact multiple areas of influence, from Aesthetics to Safety.
Concerning the quantitative analyses, Appendix Bpresents the collected data’s details.
Instead, the SUS results elaborated for groups of participants (physiotherapists, physiatrists,
and patients), i.e., the primary endpoints, are shown in Table 4.
Table 4. SUS results for groups of participants.
Category T0 T1 T2
Patients \
75.00 ±11.48 85.00 ±9.35 (p= 0.028)
min = 60.00 min = 75.00
Max = 100.00 Max = 100.00
Physiotherapists
64.38 ±12.30 78.13 ±9.98 (p= 0.028)
\
min = 40.00 min = 60.00
Max = 77.50 Max = 90.00
Physiatrists
85.00 ±3.54 86.25 ±1.77 (p= 0.667)
\
min = 82.50 min = 85.00
Max = 87.50 Max = 87.50
In addition, secondary endpoints are shown in Table 5: that is, the components of SUS,
Usability and Learning.
The mean SUS score incurs a statistically significant change (p= 0.028) from
75.00 ±11.48 to 85.00 ±
9.35 for patients and from 64.38
±
12.30 to 78.13
±
9.98 for physio-
therapists. In contrast, for physiatrists, the change in mean SUS score from 85.00
±
3.54 to
86.25 ±1.77 is not statistically significant (p= 0.667).
The mean value of the usability component SUS(U) always increased, but the change
is statistically significant only for patients and physiotherapists. Indeed, SUS(U) increased
from 77.34
±
10.92 to 86.56
±
8.87 (p= 0.039) for patients, from 63.67
±
11.80 to
78.91 ±8.64
(p= 0.010) for physiotherapists, and from 82.81
±
2.21 to 90.63
±
4.42 (p= 0.333) for
physiatrists. On the other hand, the mean value of the learning component SUS(L) increased
only for patients and physiotherapists, and in all cases, the changes were not statistically
significant. The SUS(L) increased from 65.63
±
17.78 to 79.17
±
15.39 (p= 0.114) for patients,
from 67.19
±
22.10 to 75.00
±
21.13 (p= 0.442) for physiotherapists and decreased from
93.75 ±8.84 to 68.75 ±8.84 (p= 0.333) for physiatrists.
Designs 2023,7, 9 17 of 27
Table 5. SUS(U) and SUS(L) results for groups of participants.
Category T0 T1 T2
Patients
SUS(U)
\
77.34 ±10.92 86.56 ±8.87 (p= 0.039)
min = 62.50 min = 75.00
Max = 100.00 Max = 100.00
SUS(L) \
65.63 ±17.78 79.17 ±15.39 (p= 0.114)
min = 37.50 min = 62.50
Max = 100 Max = 100.00
Physiotherapists
SUS(U)
63.67
±
11.80
78.91 ±8.64 (p= 0.010)
\
min = 43.75 min = 65.63
Max = 78.13 Max = 93.75
SUS(L)
67.19
±
22.10
75.00 ±21.13 (p= 0.442)
\
min = 25.00 min = 37.50
Max = 100.00 Max = 100.00
Physiatrists
SUS(U)
82.81 ±2.21 90.63 ±4.42 (p= 0.333)
\
min = 81.25 min = 87.50
Max = 84.38 Max = 93.75
SUS(L)
93.75 ±8.84 68.75 ±8.84 (p= 0.333)
\
min = 87.50 min = 62.50
Max = 100.00 Max = 75.00
In addition, the Mann–Whitney U-test and single-sample t-test comparing the physio-
therapists’ score to the average SUS score obtained by Sauro revealed a significant difference:
p_Mann = 0.007, p_Ttest = 0.005, with the LEPRE mean SUS 12.07 higher.
4. Discussion
The presented Usability-Oriented Model for the Design of Medical Devices addresses
usability methodologically, overcoming the limitations revealed by the analysis of scientific
literature and regulations. To apply this method, no specific conditions must be a priori
met, so we expect that the model could be easily extended to other contexts or, in general, to
medical product development, provided the proper fine adaptations. Indeed, the involved
stakeholders could greatly vary, depending on the product’s function and the main entailed
analysis aspects. For surgical devices, the focus will be, for instance, on the grip and/or
the movement ability, whereas the elements evaluated in the case of the LEPRE device
are expected to be shared among electromedical devices for rehabilitation. Nonetheless,
the continuous interaction provided by the proposed method allows for the integration
of the different visions of the various stakeholders, identifying solutions that represent a
trade-off between the technical needs indicated by designers and the demands of medical
stakeholders.
As the proposed model emphasizes, the usability aspect encompasses the entire
development process and regulation framework. The paradigm shift introduced by this
model corresponds to the extension of the notion of usability, intended as an integrated
methodological approach to be applied since the early stages of the project rather than
as an outcome or a simple attribute, as it is commonly seen in literature or in normative
contexts. Furthermore, within this model, the concept of usability has been broadened to
all elements of the product or service being designed rather than being limited to software
aspects or the human–machine interface.
Finally, the scientific literature and standards cover the topic of usability in a broad
sense. For example, ISO 9241-210:2019 proposes a general design approach based on
iterative repetition of four steps: (i) understanding and specifying the context of use;
(ii) defining user requirements; (iii) producing design solutions; and (iv) evaluating the
design. In contrast, the usability-oriented model proposed in this paper contextualizes and
specifies the influence of usability in the typical phases of a medical device design process.
Designs 2023,7, 9 18 of 27
Furthermore, the introduced paradigm shift requires that usability represents both
the goal to be achieved and the guideline as well of the whole design process and product
development. In these terms, compared to the models in the literature for both design
process, such as the model of Pahl and Beitz [
56
], and product development, like the one
described by Ulrich and Eppinger [
57
], the proposed usability-oriented model presents
two main peculiar characteristics: (a) details the phases of the design process and product
development for the medical field, showing the involved actors (Users, Physicians, Opera-
tors and Engineers), and (b) emphasizes the need to focus throughout the development
of the device, in addition to the regulations, on usability. Moreover, the application of
the usability-oriented model enables the identification of all the usability requirements in
reference to a particular functional group but does not exclude the possibility of applying
the approaches indicated by the literature in the process of design and development of its
constituent components. For example, once the shape of the Carter has been determined
thanks to the usability requirements that emerged, the design and development of its
components can follow the phases indicated by the method of Pahl and Beitz (Concept,
Embodiment and Detail) or those indicated by Ulrich and Eppinger (Planning, Concept
Development, System-Level Design, Detail Design, Testing and Refinement). However,
without the definition of the usability requirements resulting from the application of the
usability-oriented model, the approaches of the literature would generate products that
work and perform well but could reveal to be usability non-compliant.
Compared to other design support tools existing in the scientific literature, such as
Design Thinking and Codesign, the proposed model presents strong affinities and, in some
aspects, embodies them. In fact, the five stages of the design thinking defined by the Hasso
Plattner Institute of Design at Stanford [
58
] (empathize, define, ideate, prototype, and test)
or the four ones of the typical design thinking framework known as Double Diamond [
59
]
(Discover, Define, Develop and Deliver) reasonably reflect the macro phases of the proposed
usability-oriented model (“Concept Definition”, “Simulations”, “Prototype”, “On-Field
Evaluation”, and “Final Prototype”). At the same time, the principle of involving end
users in the design established by the Codesign paradigm [
60
] finds correspondence in the
usability-oriented model in making explicit the actors involved in the individual stages and
in dedicating extensive attention to constructive confrontations, e.g., through interviews
and questionnaires, involving various stakeholders throughout the design process.
However, both Design Thinking and Codesign are not contextualized to the various
stages of product design. Furthermore, even the integration of the two paradigms [
61
]
does not resolve the weaknesses of the two individual approaches since it results in the
introduction of co-design sessions in models composed of generic steps (seeing, knowing,
thinking, acting and reflecting) that are not related to the stages that occur in industrial
practice. In contrast, the presented usability-oriented model overcomes the two paradigms,
although it implements their principles. In fact, whereas the frameworks described by
Design Thinking are general, the proposed model details the specific stages; for example, by
distinguishing between functional and technical specifications in the “Concept Definition”
or by differentiating the “Simulations” from the “Prototype” in the detail stage.
Moreover, in contrast to Codesign, the usability-oriented model not only indicates
the importance of end-user involvement but also delineates the stakeholders involved
in the individual stages of the design process. For example, there are phases, such as in
the definition of “Functional Specifications”, where the involvement of patients (users),
operators, and physicians is essential. In contrast, other phases, such as “Simulations” or
“Prototype,” remain primarily engineering-related tasks. In addition, compared to the two
paradigms, the presented model explicitly introduces the influence of usability and regula-
tions affecting all stages of design, anticipating various critical issues and encouraging the
arising of unexpressed requirements.
Therefore, the novelty and usefulness of the presented usability-oriented model consist
of the extension of the concept of usability applied to a detailed design model in which
the various actors involved in the different stages are also made explicit. More specifically,
Designs 2023,7, 9 19 of 27
the novelty lies in the paradigm shift of considering usability not as an outcome but
systematically throughout the various stages of the design process. The usefulness lies in
the contextualization of this paradigm shift using a practical and specific design model that
presents the various phases pursued in the medical design practice. Table 6summarizes
the comparison with respect to the presented paradigms and methods.
Table 6.
Overview of the comparison between the usability-oriented model and the presented
paradigms/models (Y: Yes, N: No).
Paradigm/
Method
A Priori
Condition
Different
Vision
Different
Stakeholders
Define
Engineering
Tasks
Regulation
Oriented
Usability
in All Stages
Design Process Y N N Y N N
Product Development Y N N Y N N
Design Thinking N Y N N N N
Human-Centered Design N Y Y N Y Y
Codesign N Y Y N N N
Usability-Oriented Model N Y Y Y Y Y
With specific reference to the LEPRE medical device application case, the adoption
of the Usability-Oriented Model for the Design of Medical Devices highlighted several
benefits. The qualitative investigation performed through informal interviews with experts
emphasized suggestions for design improvement, as the analyzed usability requirements.
Quantitative and statistical analyses could be useful tools to assess the changes and
improvements that have been described. However, these evaluations cannot be applied if no
additional measurable data is introduced. In our case study, the Carter Shape improvement,
where the numerical values of the radii of curvature were given, and the Mouse 3D Heigh
modification, where the specific value of the uplift was given, were the only cases suitable
for analyses based on the numerical evaluation. In fact, the case of the Structural Shape
resolves in a radical change in shape; therefore, no values are meaningful for numerical
comparison, while the Mirror Function is a newly introduced function, which thus has no
comparison term. In addition, it would have been useful to punctually compare results
in terms of usability evaluation before and after each change to understand its impact
through differentials. However, this was not possible because the best design support and
usability assessment tool, according to the standard ISO/TR 16982:2002, was in the early
stages of the device development interviews. Nonetheless, these tools did not provide
quantitative results to be analyzed but rather qualitative suggestions and requirements
regarding functionality and usability to be implemented in the device as modifications,
improvements, or new functions.
Table 7summarizes the areas affected by these requirements, detailing the nature of
the modification required for each field.
Table 6shows for each usability requirement the impacted areas, specifying their
nature where appropriate. For example, focusing on the “Mechanical field”, the first
two usability requirements ((u1) and (u2)) affected the “Mechanical Structure”, while
the modification regarding the Mouse 3D (u4) introduced “Minor Design Review”. At
the same time, in terms of the “Electronic field”, the revision regarding the Carter (u1)
changed the “Electronic Structure”, while the Structural Shape (u2) concerned “Electronic
Features”. Similarly, in the “Other fields,” the usability requirements related to the Carter
(u1) expanded the “Market Share”, while changing the height of the Mouse 3D improved
the “Ease of Use”. For “Aesthetic field”, “Safety field” and “Software field”, no further
detail about the impact area was provided as it would not carry further relevant information.
In addition to specifying the impacted areas, the table also provides an overview of which
specific fields were impacted. Table 6also depicts the broadening of the concept of usability
Designs 2023,7, 9 20 of 27
since the presented usability requirements are not limited to influencing the software area
or the human–machine interface
Table 7. Overview of fields affected by usability requirements that emerged in the illustrative case.
Impact Areas Usability Requirements
(u1) Carter Shape (u2) Structural Shape (u3) Mirror Function (u4) Mouse 3D Height
Aesthetic field Aesthetics Aesthetics x x
Safety field Safety Safety Safety x
Mechanical field Mechanical Structure Mechanical Structure x Minor Design Review
Electronic field Electronic Structure Electronic Features x x
Software field x Software Features Software Features x
Other fields Market Share x x Ease of use
As a general outcome, time-to-market and project costs are significantly reduced, as
critical design issues can be anticipated and emerging usability requirements promoted
thanks to the usability paradigm shift. Table 6reveals that the earlier emerging usability
requirements are also the most impactful in terms of the number of affected fields and
relative impact on the overall development process.
As a result, the number of pre-validation prototypes is reduced, and the design process
follows a linear path, as evidenced by the increase in SUS score recorded for all categories
of participants involved. The analysis of SUS results shows, as summarized in Table 4
that for physiotherapists, an increase from 64.38
±
12.30 to 78.13
±
9.98 (from T0 to T1),
for physiatrists from 85.00
±
3.54 to 86.25
±
1.77 (from T0 to T1), and for patients from
75.00 ±11.48
to 85.00
±
9.35 (from T1 to T2). In particular, for physiotherapists and patients,
there is statistical consistency as the p-values recorded, in both cases, p= 0.028, are such as
to reject the null hypothesis. On the contrary, there is no statistical consistency in the case
of physiatrists as the p-value is found to be p= 0.667 and, therefore, not such as to reject the
null hypothesis.
These last statistical results are besides strongly affected by the small sample size
available for physiatrists, which corresponds to only two participants. In addition, Table 4
also reveals that for all three categories of participants, standard deviation decreases and
minimum recorded values increase between consecutive measures of SUS. These are two
very significant results since a reduction in standard deviation indicates greater agreement
in usability assessment. The increase in the minimum recorded values means that an
increasing number of participants consider the device to be at least acceptable in terms of
usability. In fact, for physiotherapists, the standard deviation decreased from 12.30 to 9.98
(from T0 to T1), for physiatrists from 3.54 to 1.77 (from T0 to T1), and patients from 11.48 to
9.35 (from T1 to T2).
On the other hand, in terms of minimum value, it increased from 40.00 to 60.00 for
physiotherapists (from T0 to T1), shifting from a usability between “Poor” and “OK” to a
usability between “OK” and “Good”, increased from 82.50 to 85.00 for physiatrists (from
T0 to T1), but still reg at an almost “Excellent” level of usability, and increased from 60.00
to 75.00 for patients (from T1 to T2), upgrading from a usability between “OK” and “Good”
to a usability between “OK” to “Excellent”.
After implementing all the improvements that emerged from the application of the
presented usability model, the CE-marked device (at instant T1) shows a usability level
between “Good” and “Excellent” for the three categories of participants, which is definiable.
Moreover, the SUS value emerging for physiotherapists and patients is rather close, whereas
it is significantly higher in the case of physiatrists—this is the reason for the different uses of
the device by the categories. Indeed, different users have different roles; thus, their usability
scores provide different indications. In the case of the LEPRE device, physiatrists are
primarily responsible for setting up the rehabilitation protocol, physiotherapists deal with
Designs 2023,7, 9 21 of 27
setting up the exercise (defining the trajectories and the remaining parameters), whereas
the patients perform the exercises guided by the on-screen instructions and feedback
from the device. Therefore, SUS values at T1 indicate that the usability of the CE-marked
device is acceptable with respect to rehabilitation protocol and exercise setting and exercise
execution.
The results of the SUS components always increase their value, except in the case of
physiatrists. In fact, the SUS(U) increased from 82.81
±
2.21 to 90.63
±
4.42; in contrast, the
SUS(L) decreased from a value of 93.75
±
8.84 to 68.75
±
8.84 in the case of physiatrists.
Therefore, the device is perceived as more complex to use. Nevertheless, the SUS score
measured for physiatrists grew due to the increase of the SUS(U) component.
Ultimately, the SUS measures involving clinical stakeholders (physiotherapists and
physiatrists) indicate the influence on the device usability evaluation of the changes intro-
duced through the application of the usability-oriented model, whereas the SUS measures
on patients reveal how much using a device multiple times affects the overall device us-
ability evaluation. Given the numerical results, the introduced changes have improved the
device’s usability, which grows further at a second session.
Furthermore, the results of the Mann–Whitney U-test and single-sample t-test compar-
ing the physiotherapists’ score to the average SUS score obtained by Sauro indicate that the
usability of the LEPRE device is statistically different from those of most market products,
with LEPRE mean SUS higher of 12.07. Moreover, implementing the usability-oriented
model has allowed for improvements or the introduction of new functions, influencing
the mechanical, electronic, and software aspects and enabling the increase of the potential
device market share.
More specifically, at the beginning of the project, three prototypes were planned
to be realized before achieving the final one. However, a single, final prototype was
produced thanks to the application of this model, whose changes impacted economically
as one prototype. Therefore, the estimated economic saving compared to the forecast
is 33%, corresponding to the saving of an intermediate prototype. On the other hand,
the greatest advantage is in the product development timeline since this method saves
time. Although no specific evaluation has been performed to quantify the overall time
saving, this aspect can be concretely assessed, considering that only one prototype was
made thanks to the early modifications. Half of the time required to design and make
a prototype is estimated to be the impact of the early modifications. Consequently, the
estimated time saving compared to the forecast is 50%, corresponding to the saving of
one-and-a-half intermediate prototypes. The reported percentages are estimations, as there
is no comparison term; that is, no comparison device is designed without the aid of the
Usability-Oriented Model. Furthermore, the proposed estimates are for the specific case
under consideration. However, economic savings of 10% to 40% and time-saving of 20%
to 60% are estimated through the application of the model. These estimations could be
demonstrated and further investigated in a specific study on the design of different devices
and possibly in different fields of application.
5. Conclusions
The Usability-Oriented Model for the Design of Medical Devices presented in the
current paper reduces project time and cost but also leads to linear product development.
The novelty introduced by the model consists in the extension of the notion of usability,
intended as an integrated methodological approach to be applied since the early stages of
the project rather than as an outcome or a simple attribute, and in its practicality and broad
applicability.
The LEPRE design applicative case demonstrates some of its potentiality. In fact,
through the usability paradigm shift, many criticalities of the project have been anticipated,
and several unexpressed requirements have emerged. For the application case, the effi-
ciency of the model emerges from the economic-temporal saving values (33% in economic
terms and 50% in temporal ones).
Designs 2023,7, 9 22 of 27
Future research could focus on analyzing the influence of usability in the post-CE-
marking phases; in this work, it was only mentioned as the focus was on the design process.
Specific studies could also be developed to better investigate and validate the reported
values of time-economic saving resulting from the application of the proposed model with
different application cases and in different application areas.
The various components that constitute SUS could also be investigated in future
research, introducing and investigating new metrics, even more specifically, such as mea-
suring usefulness, perceived benefit, comprehensiveness, interpretability and ease of use of
the presented usability-oriented methodology.
Finally, although the presented model is centered on medical device design, its applica-
bility could be easily extended to other fields, or further investigations could be performed
within the same sector by broadening the involved stakeholders.
Author Contributions:
Conceptualization, A.B., C.A. and R.F.; methodology, A.B., C.A. and R.F.;
validation, L.B., M.M. and R.F.; formal analysis, A.B., C.A. and R.F.; investigation, M.M., L.B. and R.F.;
resources, M.M., L.B. and R.F.; data curation, A.B., M.M. and R.F.; writing—original draft preparation,
A.B., C.A. and R.F.; writing—review and editing, A.B. and C.A.; project administration, C.A., M.M.
and L.B. All authors have read and agreed to the published version of the manuscript.
Funding: This research received no external funding.
Institutional Review Board Statement:
The study was conducted according to the guidelines of
the Declaration of Helsinki and approved by the Ethics Committee of the Brescia province (np
3123–Studio LEPRE).
Data Availability Statement: Not applicable.
Acknowledgments: RF acknowledges Polibrixia Srl for the partial funding of the PhD scholarship.
Conflicts of Interest: A.B. and M.M. declare themselves shareholders of Polibrixia s.r.l.
Appendix A
This appendix contains the statements of the SUS questionnaire in the English version
and the Italian translation (Table A1). The Italian translation complies with previous
application examples and the suggestions provided by the local authority for the public
health system, Regione Lombardia. The subject must answer each question with an integer
number between 1 and 5.
The formula for calculating the amount of SUS from the values of individual responses
to the statements is presented in relation (A1).
SUS = 4
∑
n=0
(S2n+1−1)+
5
∑
n=1
(5−S2n)!·2.5 (A1)
In this relation, nis the counter for the question number, and S
n
is the value assigned
by the subject as answer to the specific question. The formula does not merely sum the
results of all the questions in the same way, since the numerical values assume a different
meaning for odd and even questions: for odd statements, the higher the value, the better
the feedback, and the other way around for the even questions.
The formulas for calculating the amount of the Usability component, SUS(U), and
of the Learning one, SUS(L), of the SUS from the values of individual responses to the
statements, are presented in relations (A2) and (A3), respectively.
SUS(U) = 4
∑
n=0
(S2n+1−1)+
5
∑
n=1
(5−S2n)−((5−S4)+(5−S10 ))!·2.5·10
8(A2)
SUS(L) = ((5−S4)+(5−S10 ))·2.5·10
2(A3)
Designs 2023,7, 9 23 of 27
Table A1. Statements of the SUS questionnaire.
Number Language Statement
S1 English I think that I would like to use this system frequently
Italian Penso che mi piacerebbe utilizzare questo Sistema frequentemente
S2 English I found the system unnecessarily complex
Italian Ho trovato il sistema complesso senza che ce ne fosse bisogno
S3 English I thought the system was easy to use
Italian Ho trovato il sistema molto semplice da usare
S4
English
I think that I would need the support of a technical person to be able
to use this system
Italian Penso che avrei bisogno del supporto di una persona giàin grado di
utilizzare il sistema
S5
English I found the various functions in this system were well integrated
Italian Ho trovato le varie funzionalitàdel sistema bene integrate
S6 English I thought there was too much inconsistency in this system
Italian Ho trovato incoerenze tra le varie funzionalitàdel sistema
S7
English I would imagine that most people would learn to use this system
very quickly
Italian
Penso che la maggior parte delle persone potrebbero imparare ad utilizzare il
sistema facilmente
S8 English I found the system very cumbersome to use
Italian Ho trovato il sistema molto macchinoso da utilizzare
S9 English I felt very confident using the system
Italian Ho avuto molta confidenza con il sistema durante l’uso
S10
English I needed to learn a lot of things before I could get going with this
system
Italian Ho avuto bisogno di imparare molti processi prima di riuscire ad utilizzare
al meglio il sistema
Appendix B
In this appendix, the detail of the results of the applications of the SUS questionnaire
to the LEPRE device are presented.
In particular, they collect the evaluations from 8 physiotherapists at T0 (Table A2) and
T1 (Table A3), 2 physiatrists at T0 (Table A4) and T1 (Table A5), 12 post-stroke patients at T1
(Table A6) and T2 (Table A7). Color scales are added to the original values to ease the data
interpretation. Legend for the adopted color scales is provided at the end of each table.
Table A2. Results of the application of the SUS questionnaire to eight physiotherapists at T0.
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 SUS SUS(U) SUS(L)
P01 3 3 2 5 2 3 4 3 2 3 40 43.75 25
P02 3 1 1 3 2 1 1 1 2 1 55 50 75
P03 3 2 4 3 4 1 3 2 4 2 70 71.875 62.5
P04 4 2 3 3 4 2 4 2 3 2 67.5 68.75 62.5
P05 4 1 4 1 3 2 3 1 3 1 77.5 71.875 100
P06 4 2 3 2 1 1 4 1 3 1 70 65.625 87.5
P07 3 1 3 2 3 2 2 2 3 3 60 59.375 62.5
P08 5 1 4 3 4 3 5 2 3 2 75 78.125 62.5
Legend for Color Scales:
S2n 12345
S2n+1 12345
SUS 0 10 20 30 40 50 60 70 80 90 100
Designs 2023,7, 9 24 of 27
Table A3. Results of the application of the SUS questionnaire to eight physiotherapists at T1.
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 SUS SUS(U) SUS(L)
P01 4 2 2 4 3 2 4 2 4 3 60 65.625 37.5
P02 4 2 4 3 4 1 2 2 4 1 72.5 71.875 75
P03 4 2 3 2 4 1 5 1 3 3 75 78.125 62.5
P04 3 1 5 2 4 1 4 2 5 1 85 84.375 87.5
P05 5 1 5 1 5 1 4 1 4 3 90 93.75 75
P06 5 2 5 1 4 2 4 2 5 1 87.5 84.375 100
P07 4 1 4 3 3 2 4 2 4 2 72.5 75 62.5
P08 4 2 4 1 3 2 5 2 5 1 82.5 78.125 100
Legend for Color Scales:
S2n 12345
S2n+1 12345
SUS 0 10 20 30 40 50 60 70 80 90 100
Table A4. Results of the application of the SUS questionnaire to two physiatrists at T0.
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 SUS SUS(U) SUS(L)
P01 4 1 4 1 4 1 4 1 4 1 87.5 84.375 100
P02 4 1 3 2 4 2 5 2 5 1 82.5 81.25 87.5
Legend for Color Scales:
S2n 12345
S2n+1 12345
SUS 0 10 20 30 40 50 60 70 80 90 100
Table A5. Results of the application of the SUS questionnaire to two physiatrists at T1.
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 SUS SUS(U) SUS(L)
P01 5 1 5 1 5 1 4 1 4 4 87.5 93.75 62.5
P02 5 1 4 2 4 2 5 2 5 2 85 87.5 75
Legend for Color Scales:
S2n 12345
S2n+1 12345
SUS 0 10 20 30 40 50 60 70 80 90 100
Table A6. Results of the application of the SUS questionnaire to 12 post-stroke patients at T1.
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 SUS SUS(U) SUS(L)
P01 5 2 4 3 3 3 4 2 4 4 65 71.875 37.5
P02 4 2 3 3 4 2 3 3 3 3 60 62.5 50
P03 4 2 4 2 4 2 4 2 4 2 75 75 75
P04 5 1 5 1 5 1 5 1 5 1 100 100 100
P05 5 2 4 4 3 3 3 2 4 3 62.5 68.75 37.5
P06 4 2 4 2 4 2 4 2 4 2 75 75 75
P07 5 1 5 2 5 2 3 2 5 3 82.5 87.5 62.5
P08 5 1 4 3 4 3 5 2 4 2 77.5 81.25 62.5
P09 4 3 3 3 4 3 4 3 4 2 62.5 62.5 62.5
P10 5 1 4 3 4 2 4 1 5 1 85 87.5 75
P11 5 2 4 2 4 3 5 2 5 2 80 81.25 75
P12 4 2 4 2 4 2 4 2 4 2 75 75 75
Legend for Color Scales:
S2n 12345
S2n+1 12345
SUS 0 10 20 30 40 50 60 70 80 90 100
Designs 2023,7, 9 25 of 27
Table A7. Results of the application of the SUS questionnaire to 12 post-stroke patients at T2.
S1 S2 S3 S4 S5 S6 S7 S8 S9 S10 SUS SUS(U) SUS(L)
P01 5 2 4 2 4 3 5 1 4 3 77.5 81.25 62.5
P02 5 1 5 3 4 2 5 2 4 2 82.5 87.5 62.5
P03 5 1 5 1 5 1 5 1 5 1 100 100 100
P04 5 1 5 1 5 1 5 1 5 1 100 100 100
P05 5 2 5 2 3 2 4 2 4 1 80 78.125 87.5
P06 4 2 4 2 4 2 4 2 4 2 75 75 75
P07 5 1 5 2 5 2 3 2 5 3 82.5 87.5 62.5
P08 5 1 4 3 4 3 5 2 4 2 77.5 81.25 62.5
P09 4 2 5 1 4 2 4 1 4 2 82.5 81.25 87.5
P10 5 1 5 1 5 1 5 1 5 1 100 100 100
P11 5 2 4 2 4 3 5 2 5 2 80 81.25 75
P12 4 2 5 2 4 1 4 1 4 2 82.5 84.375 75
Legend for Color Scales:
S2n 12345
S2n+1 12345
SUS 0 10 20 30 40 50 60 70 80 90 100
References
1.
Vincent, C. Product development—Safety and usability of medical devices. In Contemporary Ergonomics and Human Factors 2013;
Taylor & Francis: London, UK, 2013; pp. 127–130.
2. Shackel, B. Usability—Context, framework, definition, design and evaluation. Interact. Comput. 2009,21, 339–346. [CrossRef]
3.
ISO 9241-11:2018; Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts. ISO: Geneva,
Switzerland, 2018.
4.
Becerril, L.; Stahlmann, J.-T.; Beck, J.; Lindemann, U. Usability of processes in engineering design. In Proceedings of the 21st
International Conference on Engineering Design (ICED17), Design Processes, Design Organisation and Management, Vancouver,
BC, Canada, 21–25 August 2017; Volume 2.
5.
Roma, M.S.G.; de Vilhena Garcia, E. Medical device usability: Literature review, current status, and challenges. Res. Biomed. Eng.
2020,36, 163–170. [CrossRef]
6.
Bitkina, O.V.; Kim, H.K.; Park, J. Usability and user experience of medical devices: An overview of the current state, analysis
methodologies, and future challenges. Int. J. Ind. Ergon. 2020,76, 102932. [CrossRef]
7.
Braun, S. Usability for medical devices. In Proceedings of the IEEE Symposium on Product Safety Engineering, 2005, Schaumburg,
IL, USA, 3–4 October 2005; pp. 16–22. [CrossRef]
8.
Ravizza, A.; Lantada, A.D.; Sánchez, L.I.B.; Sternini, F.; Bignardi, C. Techniques for usability risk assessment during medical
device design. In Proceedings of the BIODEVICES 2019—12th International Conference on Biomedical Electronics and Devices,
Proceedings; Part of 12th International Joint Conference on Biomedical Engineering Systems and Technologies, BIOSTEC 2019,
Prague, Czech, 22–24 February 2019; pp. 207–214. [CrossRef]
9.
Bevan, N.; Carter, J.; Earthy, J.; Geis, T.; Harker, S. New ISO standards for usability, usability reports and usability measures.
In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial