Copyright © 2022 by Author/s and Licensed by IADITI. This is an open access article distributed under the Creative Commons Attribution License which permits
unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Journal of Information Systems Engineering and Management
2022, 7(3), 15751
e-ISSN: 2468-4376
https://www.jisem- journal.com/
Research Article
Heuristic Evaluation and Usability Testing of G-MoMo
Applications
Guma Ali1 *, Mussa Ally Dida1 , Anael Elikana Sam2
1Department of Information Technology Development and Management (ITDM), Nelson Mandela African Institution of Science and Technology (NM-AIST), Tanzania
2Department of Communication Science and Engineering (CoSE), Nelson Mandela African Institution of Science and Technology (NM-AIST), Tanzania
*
Corresponding Author:
a.guma@muni.ac.ug
Citation: Ali, G., Dida, M. A., and Sam, A. E. (2022). Heuristic Evaluation and Usability Testing of G-MoMo Applications. Journal of Information
Systems Engineering and Management, 7(3), 15751. https://doi.org/10.55267/iadt.07.12296
ARTICLE INFO
ABSTRACT
Received: 12 May 2022
Accepted: 29 Jul. 2022
Financial technology (FinTech) has swiftly revolutionized mobile money as one of the ways of accessing financial
services in developing countries. Numerous mobile money applications were
developed to access mobile money
services but are hindered by severe authentication security challenges, thus, forcing the researchers to design a secure
multi-factor authentication (MFA) algorithm for mobile money applications. Three prototypes of native
mobile money
applications (G-
MoMo applications) were developed to confirm that the algorithm provides high security and is
feasible. This study, therefore, aimed to evaluate the usability of the G-
MoMo applications using heuristic evaluation
and usability
testing to identify potential usability issues and provide recommendations for improvement. Heuristic
evaluation and usability testing methods were used to evaluate the G-
MoMo applications. The heuristic evaluation
was carried out by five experts that used the 10 principles proposed by Jakob Nielsen with a five-
point severity rating
scale to identify the usability problems. While the usability testing was conducted with forty participants selected
using a purposive sampling method to validate the usability of the G-
MoMo applications by performing tasks and
filling out the post-test questionnaire. Data collected were analyzed in RStudio software. Sixty-
three usability issues
were identified during heuristic evaluation, where 33 were minor and 30 were major.
The most violated heuristic
items were “help and documentation”, and “user control and freedom”, while the least violated heuristic items were
“aesthetic and minimalist design” and “visibility of system status”. The usability testing findings revealed that the G-
MoMo applications’ performance proved good in learnability, effectiveness, efficiency, memorability, and errors. It
also provided user satisfaction, ease of use, aesthetics, usefulness, integration, and understandability. Therefore, it
was highly recommended that the developers of G-
MoMo applications fix the identified usability problems to make
the applications more reliable and increase overall user satisfaction.
Keywords: Mobile money, industry 4.0, digital transformation, G-MoMo applications, blockchain, heuristic
evaluation, usability testing, experts, participants.
INTRODUCTION
The evolution of the fourth industrial revolution has caused
a substantial transformation in business through FinTech, such
as mobile money. However, over a billion people in developing
countries do not have formal bank accounts, forcing the
unbanked population to incur high transaction costs and theft
since they resorted to informal financial networks.
Surprisingly, the unbanked population has access to mobile
phones, making it easy to transfer money digitally, thus, giving
rise to mobile money. As one of the essential financial
innovations in developing countries, mobile money fills this
gap by offering convenient digital financial services over
mobile phones and improving financial inclusions (Ali et al.,
2020a; Rwiza et al., 2020).
The mobile money subscribers perform mobile money
services using either a dedicated mobile application or the
unstructured supplementary service data (USSD) (Ali et al.,
2020b; Ayeb et al., 2022). The current mobile money applications
developed by the mobile money service providers only use a
personal identification number (PIN) and one-time password
(OTP) to authenticate mobile money subscribers. Though
promising, this two-factor authentication (2FA) scheme is
susceptible to many security attacks (Ali et al., 2020b). Much as
the developed applications helped ensure mobile money
security, more work is still needed to improve their algorithms.
This prompted the researchers to develop a secure MFA
algorithm for mobile money applications to overcome these
security issues (Ali et al., 2021).
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
2 / 14
Other emerging technologies such as blockchain are also
used in mobile money systems to (1) prevent fraud and ensure
the security of the mobile money transactions data; (2) ensure
interoperability among the different mobile money service
providers to permit cross-operators transactions; (3) create
transparent and traceable data; (4) ensure auditability by
allowing governments and regulators to access a
cryptographically verifiable copy of an immutable ledger; and
(5) address trusted third-party issues. These are achieved by
storing the mobile money user’s identity and the mobile money
transactions data in a distributed and immutable ledger
(Agbezoutsi et al., 2021, 2019).
Three prototypes of native G-MoMo were developed to
prove the algorithm’s feasibility and provide high security
against attacks and threats (Ali et al., 2021). This study,
therefore, validates the developed native G-MoMo
applications using heuristic evaluation and usability testing to
identify usability issues and provide feedback for
improvement to achieve usability and satisfaction. The
usability of applications due to user-friendly design enhances
user experience, attracts and retains customers, increases
profitability of services, allows users to attain their desired
objectives and contributes to user satisfaction and loyalty. The
critical task of mobile application developers requiring keen
attention is improving the applications’ design to appease user
satisfaction through heuristic evaluation and usability testing.
The heuristic evaluation uses the 10 heuristics established by
Jakob Nielsen to identify any usability problems with the
application interfaces and provide feedback for improving the
early designs (Hertzum, 2020; J. Nielsen, 1994a). At the same
time, usability testing is evaluated based on quality
components such as effectiveness, efficiency, satisfaction,
learnability, errors, memorability, understandability,
attractiveness, and accessibility (Ammar, 2019; Chipa and
Mwanza, 2021). Validating the native G-MoMo applications
using heuristic evaluation and usability testing is crucial
because it allows the application developers, experts, mobile
money service providers, and end-users to test the G-MoMo
applications’ user interfaces to ensure usability and
satisfaction.
The significant contribution of this study is the validation
of the G-MoMo applications using heuristic evaluation and
usability testing to identify usability issues and provide
feedback for improvements. Both heuristic evaluation and
usability testing were achieved through laboratory
experiments. Experts and a few selected participants used the
G-MoMo applications in the real environment to identify
usability problems and provide feedback by filling the post-
test questionnaire for improvements (Ammar, 2019).
The remaining contents of this article are structured as
follows: The background to mobile application usability,
usability evaluation, heuristic evaluation, usability testing, and
G-MoMo applications are provided in Section 2, and Section 3
describes the materials and methods used. The analysis of the
results is explained in Section 4, while discussions are
presented in Section 5. Finally, the conclusion and
recommendations are raised in Section 6.
BACKGROUND
Mobile Application Usability
(Weichbroth, 2020) defines a mobile application as a program
developed for smartphones and tablets to perform specific tasks
and functions. Mobile applications can either be installed on
mobile devices or accessed using web browsers (Lynn et al.,
2020). The installed mobile applications can be downloaded from
mobile App Stores such as Google Play Store, Apple App Store,
Amazon Appstore, Samsung Galaxy Apps, Huawei App Store,
and Sony Apps (Byun et al., 2020).
Mobile application usability has attracted the attention of
application users, software developers, and academics and is a
crucial area of research in human-computer interaction (HCI)
because it determines the application’s success and reduces
irritation during usage (Lynn et al., 2020; Weichbroth, 2020). The
usability of mobile applications also determines the successful
technology adoption depending on how users feel about the use
of the application in terms of improving work performance
(Byun et al., 2020). The user’s feelings, understanding, desires,
convenience, attitude, and achievement are assessed through
mobile application usage (Bajcar et al., 2020). The successful
adoption and usage of mobile applications depend on how users
easily and efficiently perform services. When mobile
applications are poorly designed, it results in low usage.
Nevertheless, an application can only be used when it performs
its functions effectively, efficiently, and satisfactorily. The
usability of mobile applications is evaluated using expert-based
methods and a standard questionnaire designed and
administered to end-users to ascertain their satisfaction based on
ease of usage of the applications, how they enable users to
perform tasks efficiently and clearly, and so on (Salari et al.,
2021).
According to (Ammar, 2019), usability is how specific people
use a software product to attain stated goals effectively,
efficiently, and satisfactorily. A usability evaluation is a sequence
of a well-defined set of tasks for gathering relevant information
related to end-user interaction with the application to determine
how features of the software product add to earning a certain
level of usability by identifying the usability problems (Kous et
al., 2020; Lynn et al., 2020). Its main objective is to assess the
quality of the interface designs of the system and mobile
applications, ascertain the possible interaction issues with the
applications, develop aesthetic interface designs and check them
per the usability standards. Usability evaluation can be formative
or summative, depending on the assessment goals (Salman et al.,
2018). Formative evaluation collects feedback from users for
additional development, while summative evaluation evaluates
whether the set usability requirements are fulfilled (Hewett,
1986). Several factors such as screen size, storage capacity,
interface design, and context of use are considered in the
usability evaluation of mobile applications.
Usability Evaluation
The usability evaluation process is essential and includes
many evaluation methods such as heuristic evaluation, cognitive
walkthrough, interviews, log analysis, task analysis, eye
tracking, perspective-based inspection, think-aloud usability
testing, guideline reviews, focus group, questionnaires, and
remote testing (Islam et al., 2020; Wahyuningrum et al., 2020).
These usability evaluation methods are applied at any stage of
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
3 / 14
the software development process, and the evaluation process
can be performed on prototypes and the final product. It helps
reduce costs since changes are easy to implement and
determine if the application meets an appropriate level of
usability. They provide recommendations for improving the
interface designs of the applications (Eliseo et al., 2017).
(Jeong et al., 2020) and (Putri et al., 2021) further added that
the usability evaluation method measures software usability
by testing it with selected users to identify any usability
problems and give direct feedback and recommendations for
improvement to achieve effectiveness, efficiency, and user
satisfaction. Heuristic evaluation and usability testing are the
primary methods for validating application interfaces
(Ammar, 2019). By carrying out usability evaluation, it helps to
identify defects in mobile applications by checking and
inspecting the application interfaces, thus, allowing developers
to produce suitable application designs to achieve usability
evaluation goals (Lynn et al., 2020). It emphasizes the level
users interact with mobile devices, ensures the acceptability of
the application, provides ease of application usage, ensures
effective and efficient application, and acts as an essential
reference for improving the applications’ design and features
(Lynn et al., 2020). Heuristic evaluation is extensively used to
validate system and application interfaces during and after
development (Hussain and Omar, 2020). Jakob Nielsen
proposed the heuristic evaluation method in 1994 and defined
it as an evaluation method with between 3 to 5 trained HCI
expert evaluators to examine a system or prototype according
to established heuristic principles to check for problems and
deficiencies with the application interfaces and rectify possible
faults (J. Nielsen and Mack, 1994). It is performed by experts
who rely on 10 heuristics established by Jakob Nielsen to serve
as a framework for evaluating the user interface design of
applications, and the experts are selected carefully (J. Nielsen,
1994a). The experts include usability specialists, fellow
developers, and expert users who use heuristic rules,
subjective judgment, and task-based evaluation to rate the
severity of application interface issues. Then all problems and
ratings are analyzed collectively to identify the most critical
usability issues (Tremoulet et al., 2021).
Heurestic Evaluation
Heuristic evaluation’s primary goal is to identify usability
problems when some expert users operate the system or
application interface at a relatively low cost, resulting in
multiple enhancements to the mobile applications (Kumar et
al., 2020). When the heuristic principle is violated, the expert
determines the severity of the problem and suggests solutions
(Paramitha et al., 2018). It can be carried out at any software
development stage upon identifying the problem and usually
presents the best practical results(Kumar et al., 2020). The
expert goes through the system once to get familiar with it
during the heuristic evaluation and then thoroughly assesses
the specific heuristics (Ball and Bothma, 2018). A maximum of
five experts can identify usability problems (J. Nielsen and
Mack, 1994).
The 10 principles proposed by Jakob Nielsen to examine the
usability problems with the interface designs are “visibility of
system status”; “match between the system and the real
world”; “user control and freedom”; “consistency and
standards”; “error prevention”; “recognition rather than recall”;
“flexibility and efficiency of use”; “aesthetic and minimalist
design”; “help users recognize, diagnose, and recover from
errors”; and “help and documentation” (J. Nielsen, 1994a).
Heuristic evaluation offers numerous benefits, including simple,
fast & efficient identification of usability problems, a cheap
method to evaluate applications, easy to motivate people to do,
and suitable for every life-cycle software phase (Kumar et al.,
2020; Tremoulet et al., 2021) Due to these benefits, the heuristic
evaluation approach is widely used in HCI to evaluate the
effectiveness of mobile applications.
Usability Testing
The usability testing method defines the usability of a system
and mobile application. It has been accepted as an essential
activity in software design, implementation, testing, acceptance,
and revision because it aims to determine user perception about
the application by measuring convenience and efficiency and
ensuring user satisfaction with the software product (Ramayasa
and Candrawibawa, 2021; Wirasasmiata and Uska, 2019).
Software developers must perform usability tests on mobile
applications to achieve the quality of tasks and help improve
application designs for higher market and competency (Byun et
al., 2020; Lynn et al., 2020). It is also essential in developing an
application to ensure that the various end-users can access,
understand and identify gaps, and use it (Babatunde et al., 2020).
Usability testing can be conducted throughout the development
life cycle of the mobile application (Weichbroth, 2020). It consists
of multiple facets commonly known as usability attributes, used
to measure the quality of the applications (Putri et al., 2021;
Zakaria et al., 2020). Internal and external attributes are the two
types of usability attributes used to measure the quality of
applications (Ammar, 2019). Researchers have identified many
attributes for testing the usability of mobile applications, which
include (a) learnability; (b) effectiveness; (c) efficiency; (d)
memorability; (e) errors; (f) user satisfaction; (g) simplicity; (h)
comprehensibility; (i) cognitive load; (j) ease of use; (k)
understandability; (l) operability; (m) aesthetic; (n) accessibility;
and (o) learning performance (Byun et al., 2020; Hussain and
Omar, 2020; Kous et al., 2020; Lynn et al., 2020). Several usability
testing methods used techniques, such as performance
measurement, to evaluate effectiveness and efficiency.
Retrospective think-aloud and post-study system usability
questionnaires measure user satisfaction (Ammar, 2019).
Usability testing offers several benefits, such as identifying
usability problems via user interaction observation; determining
how easily a user uses an application’s interface. Furthermore,
usability testing reduces the cost of changes later in the software
development life cycle (Burkard, 2020; Jeong et al., 2020).
G-MoMo Application
(Ali et al., 2021) designed a secure MFA algorithm for mobile
money applications. The algorithm authenticates mobile money
subscribers using a novel method combining PIN, OTP, and
biometric fingerprint. The mobile money customer confirms
money withdrawal by scanning their biometric fingerprint and
the agent’s QR code. In addition, the PINs and OTPs are
protected by SHA-256; biometric fingerprints by FIDO, where
the RSA encryption secures the public/private key pair and the
fingerprint templates; and the QR codes, confidential
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
4 / 14
information in the databases, and all the data before
transmission using Fernet encryption (Ali et al., 2021).
G-MoMo IT support application, G-MoMo agent
application, and G-MoMo customer application prototypes
were developed to prove the algorithm’s feasibility and
provide high security against attacks and threats. The
applications’ front-end was developed using the Vue JS
framework, Python for the back-end, MySQL as a back-end
database, and Twilio SMS to receive the OTP (Ali et al., 2021).
The mobile money IT support staff uses the G-MoMo IT
support application to register new mobile money IT support
staff and agents, add new smartphones for registered mobile
money agents and customers, display the enrolled subscribers’
statistics, and manage their PIN and biometric fingerprint. G-
MoMo agent application allows mobile money agents to enrol
new customers, deposit money, display agent’s QR code, check
available float and manage their PIN and biometric fingerprint.
While G-MoMo customer application allows mobile money
customers to withdraw money, send money, pay bills, check
electronic balances and mini statements, and manage their PIN
and biometric fingerprint (Ali et al., 2021). The three G-MoMo
applications were used to explain the enrolment,
authentication, and transaction phases (Ali et al., 2021).
Mobile Money Agent Enrolment Phase
The enrolment phase involves the mobile money IT
support staff downloading and installing the G-MoMo IT
support application on their smartphones connected to the
Internet. The mobile money IT support staff must log into the
G-MoMo IT Support Application. Once they log in
successfully, they can register a new mobile money agent by
capturing their first name, last name, and phone number, and
then confirm their registration. A 5-digit OTP will be generated
and sent to the new mobile money agent’s smartphone, where
they are requested to read and tell the IT support staff. The
mobile money IT support staff will enter the OTP to complete the
registration. If the OTP matches, the information is saved in the
database and the new agent is successfully registered but
required to finish the enrolment process by installing and
running the G-MoMo agent application; else, required to attempt
three (3) times. Figure 1(a)–(f) illustrates the steps the mobile
money IT support staff follows to enrol the new mobile money
agent using the G-MoMo IT support application.
After installing the G-MoMo agent application, and when the
mobile money agent runs the G-MoMo agent application for the
first time, the system will require the agent’s smartphone and
phone number to be registered. The agent will be requested to
enter their phone number, through which a 5-digit OTP will be
sent to verify the phone number and smartphone. Once the agent
receives and enters the OTP, it will be compared with the copy
stored in the database, and if it matches, the smartphone and
phone number are registered, and a universally unique identifier
(UUID) is generated for the phone number and smartphone, and
the UUID is encrypted with Fernet and saved in the main
database. Figure 2(a)–(d) illustrates the steps the mobile money
agent follows to register their smartphone and phone number.
After successfully registering the phone number and the
smartphone, the agent must complete the enrolment process by
running the G-MoMo agent application, entering their five (5)-
digit PIN, and re-entering the 5-digit PIN again. If the PINs
match, the agent is requested to confirm the creation of the new
PIN. Once the new PIN is approved, the mobile money agent can
enroll their biometric fingerprint by scanning it using its
biometric fingerprint sensor, and if it is successfully captured, the
fingerprint template is saved, and the mobile money agent is
successfully registered. The application then sends a notification
to the mobile money agent for successful registration and is
ready to use the G-MoMo agent application. Figure 3(a)–(g)
illustrates the steps the mobile money agent must follow to
complete the enrolment using the G-MoMo agent application.
Figure 1(a)-(f). Illustrates the steps the mobile money IT support staff follows to enroll the mobile money agent using the G-
MoMo IT support application.
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
5 / 14
Mobile Money Agent Authentication Phase
After the successful enrolment, the agent can log in to the
G-MoMo agent application by entering the five-digit PIN. If the
PIN is correct, a 5-digit OTP is generated and sent to the agent
by SMS. Once the OTP is received, they must enter it, where
the system will compare it with the template stored in the
database. If it matches, the agent is requested to scan their
biometric fingerprint. If the scanned fingerprint matches, the
agent successfully logs in to the G-MoMo agent application
and is presented with the menu to choose service(s). Figure
4(a)–(d) illustrates the steps the mobile money agent follows
during authentication.
Mobile Money Agent Authentication Phase
After the successful enrolment, the agent can log in to the G-
MoMo agent application by entering the five-digit PIN. If the PIN
is correct, a 5-digit OTP is generated and sent to the agent by
SMS. Once the OTP is received, they must enter it, where the
system will compare it with the template stored in the database.
If it matches, the agent is requested to scan their biometric
fingerprint. If the scanned fingerprint matches, the agent
successfully logs in to the G-MoMo agent application and is
presented with the menu to choose service(s). Figure 4(a)–(d)
illustrates the steps the mobile money agent follows during
authentication.
Figure 2(a)-(g). Illustrates the steps the mobile money agent must follow to register the smartphone and phone number using the
G-MoMo agent application.
Figure 3(a)-(d). Illustrates the process of completing mobile money agent enrolment using the G-MoMo agent application.
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
6 / 14
Figure 4(a)-(d). Illustrates the steps the mobile money agent follows during authentication using the G-MoMo agent application.
Transaction Phase
The transaction phase involves the mobile money agent
using the G-MoMo agent application to enrol new customers,
deposit money, and confirm money withdrawal through a QR
code scan.
(a) Depositing Money
The mobile money agent begins depositing money into
customers’ accounts by running the G-MoMo agent
application and signing in by entering their PIN and OTP and
scanning their fingerprint. If the agent logs in successfully,
their electronic balance is displayed. If they have enough float,
they can select the deposit menu, enter the recipient’s phone
number, and search to confirm whether it is registered with the
G-MoMo customer application. If the phone number is
enrolled, the application will display the phone number and
the name of the person who registered the phone number. The
application will request the mobile money agent to enter the
amount they want to deposit. The system will only accept the
deposit amount less than the available float. The application
will then request the agent to confirm whether they want to
deposit the money into the recipient’s phone number. If the
agent clicks the deposit button, the system will deposit the
funds into the recipient’s account and display the successful
money deposited notification. Figure 5(a)–(d) illustrates the
steps followed by the mobile money agent to deposit money
into customers’ accounts using the G-MoMo agent application.
(b) Money Withdrawal
To withdraw money from their mobile money wallet, the
customer must first log into the G-MoMo customer application
by entering their PINs, OTP, and biometric fingerprints. The
system will verify the PINs, OTP, and biometric fingerprints
entered, and if they do not match, they are required to try again;
else, they are successfully logged in. The mobile money customer
must check their available electronic balance and ensure that it is
withdrawable. The system only accepts the customer to enter an
amount less than the available balance. After entering the
amount, they can click the withdraw button, where the system
will request them to scan their biometric fingerprint for
authorization. The scanned biometric fingerprint is matched
with the fingerprint template stored in the database, and if it
matches, the customer is requested to scan the secure QR code of
the mobile money agent using the customer’s smartphone smart
scanner for final confirmation. The system will then verify the
scanned secure QR code, and if it is correct, money is withdrawn
from the customer’s account, and the electronic balance is
updated. A notification for successful money withdrawal is
displayed, authorizing the customer to collect money from the
mobile money agent (Ali et al., 2021). Figure 6(a)–(g) illustrates
the steps followed by the mobile money customer to withdraw
money from their account using the G-MoMo customer
application.
Figure 5(a)-(d). Illustrates the steps the mobile money agent follows during authentication using the G-MoMo agent application.
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
7 / 14
Figure 6(a)-(g). Illustrates the steps followed by the mobile money customer to withdraw money from their account using the G-
MoMo customer application.
MATERIALS AND METHODS
Heuristic Evaluation
Heuristic evaluation and usability testing methods were
used to evaluate the usability of the three G-MoMo
applications, identify usability issues with their interface
designs, and suggest recommendations for improvement. They
were conducted for five months from December 2021 to April
2022. The application of the two methods is briefly described
below.
The Heuristic evaluation method adopted the 10 heuristic
guidelines established by Jakob Nielsen to serve as a
framework for evaluating the user interface design of G-MoMo
applications. Five experts conducted the heuristic evaluation of
the interface designs of G-MoMo applications to identify
usability issues and suggest recommendations for their
improvements. The experts were selected based on the
recommendation by Nielsen and Mack that requires 3-5
evaluators (J. Nielsen and Mack, 1994). Two of the selected
experts have expertise in usability evaluation and knowledge
base and three with the knowledge base. They consist of four
males and one female between 28 to 40 years old. Three experts
were web and mobile application developers who had used the
G-MoMo applications for some days. The experts were chosen
based on their profiles. The two experts were teaching staff who
had experience in HCI and conducted research in usability
testing and mobile applications. They had significant expertise in
conducting heuristic evaluations on systems and applications
and had used the G-MoMo applications for three weeks before
analyzing them. The three experts in the second group included
web and mobile application developers focusing on developing
mobile-friendly web and mobile applications and had experience
developing special-purpose mobile applications. The experts
acquired two-hour training on using Nielsen’s heuristics to
assess the G-MoMo applications before conducting the
evaluation. Each heuristic item was thoroughly explained with
examples to help them accurately identify usability problems.
The profiles of the experts are summarized in Table 1.
Table 1.
The profile of the usability evaluation experts.
S/No Participant Gender Age
Profession
Years of
Experience
1
Expert 1
Female
37
PhD in Information Technology
8
2
Expert 2
Male
39
PhD in Computer Science
10
3
Expert 3
Male
30
Web and Mobile Application Developer
6
4
Expert 4
Male
31
Web and Mobile Application Developer
6
5
Expert 5
Male
29
Mobile Application Developer
5
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
8 / 14
Table 2.
Participants’ Social Demography Characteristics.
S/No Variable Frequency Percentage (%)
1
Gender
Male 25 62.5
Female 15 37.5
2
Age
Less than 20 years 0 0.0
Between 20–29 years 19 47.5
Between 30–39 years 16 40.0
More than 39 Years 5 12.5
3
Level of Education
Bachelors 32 80.0
Masters 6 15.0
PhD 2 5.0
4
Category of Evaluation
Mobile Money IT Support Staff 4 10.0
Mobile Money Agent 10 25.0
Mobile Money customers 26 65.0
The heuristic evaluation post-test questionnaire contained
five-point Likert scale statements developed based on the 10
heuristic guidelines to evaluate the prototypes of the G-MoMo
applications (J. Nielsen, 1994a).
The experts were asked to install and test the three
prototypes of G-MoMo applications on Android-based
smartphones like Tecno Camon 16 Pro running Android 10
with a touchscreen having 720 x 1640 pixels and a rear-
mounted fingerprint sensor. It was also tested on Samsung
Galaxy S7 Edge running Android 7.0, with a touchscreen
having a 2560 x 1440 pixels resolution and a front-mounted
fingerprint sensor.
During the evaluation, the experts used the heuristic
evaluation post-test questionnaire to evaluate the prototypes of
the G-MoMo applications interfaces by giving their opinion
about the usability issues with the interfaces after performing
some tasks such as registration, authentication, transaction,
and system logout. The usability issues’ severity was rated
based on applying the Jakob Nielsen scale of (0) no problem,
(1) cosmetic problem only, (2) minor usability problem, (3)
major usability problem, and (4) usability catastrophe (J.
Nielsen, 1994b), which provided the experts with a better
insight into the usability issues with their degrees of severity
which the application developers can consider as a priority and
make the essential corrections (Nabovati et al., 2014). The
experts were also requested to give additional suggestions
after the evaluation. After the evaluation, the results obtained
using the post-test questionnaires were compiled and a
consensus was generated for the ratings, and recommendations
were provided. Descriptive statistics were calculated from the
collected data about the usability issues and analyzed using the
RStudio software.
This study also employed a usability testing method to
ascertain the ease of use of the G-MoMo applications. The
method was used to obtain quantitative data from the selected
participants about the G-MoMo applications. Forty participants
were chosen using a purposive sampling method to validate the
usability of the G-MoMo applications. The selected sample size
is enough to carry out usability testing because, according to
Jakob Nielsen, the number of respondents to participate in
usability testing is at least 20 people (Nielsen, 2021). Of the 40
participants, 25 were male (62.5%), and 15 (37.5%) were female.
The participants were between the ages of 20 to 40. However, 19
(47.5%) of the participants were between 20-29 years, 16 (40.0%)
between 30-39 years, while the remaining 5 (12.5%) were above
39 years. Among the participants, 32 (80%) had Bachelor’s
degrees, 6 (15%) master’s degrees, and 2 (5%) had PhD. The
selected participants were computer literate. Nevertheless, not
all participants were familiar with the functioning of the G-
MoMo applications. The participants were further divided into
mobile money IT support staff, agents, and customers. 4 (10%) of
the participants were grouped as mobile money IT support staff,
10 (25%) as mobile money agents, and 26 (65%) as mobile money
customers. Table 2 Summarizes the social demography
characteristics of the participants.
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
9 / 14
Table 3.
The frequency of severity of usability issues with the interfaces of G-MoMo applications.
ID Heuristic Principles Severity Total Average
Severity
Cosmetic Minor Major Catastrophe Frequency %
H1 Visibility of system status 0 1 0 0 1 1.6 2
H2 Match between system and the real
world 0 4 0 0 4 6.3 2
H3 User control and freedom 0 8 3 0 11 17.5 2.2
H4 Consistency and standards 0 5 0 0 5 7.9 2
H5 Error prevention 0 3 0 0 3 4.8 2
H6 Recognition rather than recall 0 4 0 0 4 6.3 2
H7 Flexibility and efficiency of use 0 3 3 0 6 9.5 2.5
H8 Aesthetic and minimalist design 0 1 0 0 1 1.6 2
H9 Help users recognize, diagnose,
and recover from errors 0 3 0 0 3 4.8 2
H10 Help and documentation 0 1 24 0 25 39.7 3
TOTAL 0 33 30 0 63 100.0 2.2
0.0 52.4 47.6 0.0 100%
Before the selected participants began performing tasks, the
researchers conducted a quick pilot study to ensure a smooth
usability testing session. The researchers began conducting
usability testing by briefing and checking the participants’
smartphones to ensure that they were connected to the Internet
and that the fingerprint sensors functioned well. A functioning
version of G-MoMo applications was downloaded and
installed based on the participants’ category. The participants
were introduced to G-MoMo applications, their features,
functionalities, and workflow. The G-MoMo applications were
demonstrated to each category of the participants. Each
participant was allowed to carry out tasks using the G-MoMo
applications to learn what they were doing. Three moderators
supervised the process to ensure the smooth running of the
session. After completing tasks, the participants were required
to validate the applications by filling out the post-test
questionnaire. The questionnaire contained five-point Likert
scale statements developed based on the usability testing
attributes used to validate the usability of the G-MoMo
applications. The agreement scale used in the post-test
questionnaire was (1) strongly disagree, (2) disagree, (3)
neutral, (4) agree, and (5) strongly agree. After completing the
tasks, the selected participants appraised their satisfaction with
the applications and shared their experiences and
recommendations with the moderators. Data collected using
post-test questionnaires in the usability testing was analyzed
in RStudio software. Percentages, means, standard deviations,
and graphs were computed and analyzed to understand the
general usability of the assessed G-MoMo applications. The
results for the mean (M) ≥ 3.41 were considered statistically
significant (Pimentel, 2010).
RESULTS
Heuristic Evaluation Results
Usability experts conducted the heuristic evaluation to
ascertain the usability issues with G-MoMo applications using
Jakob Nielsen’s 10 principles of heuristic evaluation. Each of
the five usability experts independently evaluated the G-
MoMo applications using heuristic evaluation principles to
find usability issues. The usability issues encountered by each
usability expert were compiled to produce the report on the
heuristic evaluation. Sixty-three (63) usability issues were
identified, where 33 (52.4%) were minor and 30 (47.6%) were
major. Table 3 shows the frequency of severity of usability issues
with the interface designs of G-MoMo applications based on the
10 principles of heuristics evaluation.
G-MoMo IT support application had 10 minor and 8 major
usability issues compared to the G-MoMo agent application,
with 10 minor and 10 major and the G-MoMo customer
application with 13 minor and 12 major usability issues. Figure 7
shows the distribution of usability issues across the three G-
MoMo applications.
Table 3 and Figure 8 show the regularity of severity of
usability issues with the interfaces of G-MoMo applications. The
severity rating results showed that the "help and documentation
(H10)" principle was mentioned 25 times (39.7%) with a mean
severity score of 3.0 and had the most frequency, where it is
grouped as a major problem. This was followed by the "user
control and freedom (H3)" principle which had a mean severity
score of 2.2 and frequency of 17.5% where it was grouped as a
minor problem. The principles of "visibility of system status
(H1)" and "aesthetic and minimalist design (H8)" were each
mentioned once (1.6%), with mean severity scores of 2.0, and
they had the least frequency and were grouped as minor
problems.
The G-MoMo IT support, G-MoMo agent, and G-MoMo
customer applications had more usability issues related to “help
and documentation (H10)”, and “user control and freedom
(H3)”. However, they had few usability issues related to
“aesthetic and minimalist design (H8)” and “visibility of system
status (H1)”. The results of usability issues are presented in
Figure 9.
Five significant themes related to usability issues were
identified through the qualitative analysis of the three G-MoMo
applications. The identified usability problems were based on
heuristics principles and are clearly explained.
Lack of forward navigation button: The three G-MoMo
applications have a backward navigation button for mobile
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
10 / 14
money subscribers but lack a forward button. This has
made it difficult for the subscribers to navigate between
the different pages of the applications, thus affecting user
control and freedom.
Lack of search field options: The G-MoMo applications do
not have search field options, making finding the
required services difficult. This lack of search field
options affects the applications’ flexibility & efficiency
and user control & freedom.
Lack of actions needed for recovery: The G-MoMo
applications lack detailed steps essential for recovery in
case the applications crush, thus affecting the
applications’ error diagnosis and recovery.
Lack of uniformity in the G-MoMo applications menu titles:
Some menu titles of the applications are aligned to the left,
center, and justified. This has caused inconsistency in the
applications’ menu titles, thus affecting consistency and
standards.
Lack of help and documentation: The G-MoMo applications
lack help and documentation components for the
subscribers. The instructions on using the applications are
not visible or clear. They also lack a panel of tips and tricks
for the application, and it is not easy for new users to
understand the navigation menu. This usability issue thus
results in errors during the applications’ usage and makes
it difficult for the users to recall the steps involved in the
application’s usage.
Figure 7. The distribution of usability issues across the G-MoMo applications.
Figure 8. The frequency of severity of usability issues with the interfaces of the three G-MoMo applications.
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
11 / 14
Figure 9. The usability issues with the user interfaces of the three G-MoMo applications.
Table 4.
Opinion of participants about the usability of G-Momo applications.
1
S/No Usability Testing
Attributes SD D N A SA M Std Dev
1 Learnability 0.0 5.0 15.0 42.5 37.5 4.13 .853
2 Effectiveness 0.0 0.0 32.5 62.5 5.0 3.73 .554
3 Efficiency 0.0 0.0 7.5 42.5 50.0 4.43 .636
4 Memorability 0.0 0.0 17.5 70.0 12.5 3.95 .552
5 Errors 0.0 0.0 7.5 57.5 35.0 4.28 .599
6 User satisfaction 0.0 0.0 17.5 65.0 17.5 4.00 .599
7 Ease of use 0.0 0.0 15.0 67.5 17.5 4.03 .577
8 Aesthetic 0.0 0.0 17.5 62.5 20.0 4.03 .620
9 Usefulness 0.0 0.0 22.5 60.0 17.5 3.95 .639
10 Integration 0.0 0.0 10.0 57.5 32.5 4.23 .620
Usability Testing
The usability testing results presented are from the 40
participants selected to validate the G-MoMo applications by
performing various tasks discussed in the materials and
methods. The participants managed to submit 40 (100%) post-
test questionnaires, which were analyzed. The percentages,
mean, and standard deviations were calculated to assist in
making decisions. Table 4 depicts the participants’ opinions
regarding the usability of G-MoMo applications.
As shown in Table 4, majority of the participants agreed
that, learnability (M = 4.13, Std Dev = 0.853), effectiveness (M =
3.73, Std Dev = 0.554), efficiency (M = 4.43, Std Dev = 0.636),
memorability (M = 3.95, Std Dev = 0.552), errors (M = 4.28, Std
Dev = 0.599), user satisfaction (M = 4.00, Std Dev = 0.599), ease
of use (M = 4.03, Std Dev = 0.577), aesthetic (M = 4.03, Std Dev
= 0.620), usefulness (M = 3.95, Std Dev = 0.639), integration (M
= 4.23, Std Dev = 0.620), and understandability (M = 4.10, Std
Dev = 0.496) were the usability testing attributes achieved
while using the G-MoMo applications. Therefore, it was
statistically significant to conclude that the above-mentioned
attributes were achieved, and no issues were got with their
usability because their means are greater than 3.41.
1 SD = Strongly Disagree, D = Disagree, N = Neutral, A = Agree, SA = Strongly Agree, M = Means, and Std Dev = Standard Deviation.
DISCUSSION
Before fully deploying the developed G-MoMo applications,
the researchers needed to perform heuristic evaluation and
usability testing to identify usability issues and suggest
recommendations. Five experts and 40 participants were selected
to participate in the validation. They were allowed to use the
three G-MoMo applications to perform various tasks to identify
usability issues, give recommendations for improvement, and
assess their usability. The user interfaces design of G-MoMo
applications was evaluated using Jakob Nielsen's proposed 10
usability heuristics, and usability testing was measured using
essential attributes. The heuristic evaluation results revealed that
many usability issues exist with the interface designs of G-MoMo
applications, and most of the problems were ranked as minor
and major. The usability issues identified are discussed as
follows.
The G-MoMo applications lack forward navigation button,
thus, forcing the users to either go back to the home page or select
any service provided by the applications and then proceed to the
required pages. This affected the user’s control and freedom.
This finding is similar to the studies by (Höhn and Bongard-
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
12 / 14
Blanchy, 2020; Jeddi et al., 2020), where it was found that
applications without backward and forward navigation
buttons make it difficult for users to navigate between the
pages. Most of the menu titles of the G-MoMo applications are
not uniformly aligned. Some menu titles are aligned to the left,
centre and justify, which shows inconsistency in the alignment.
This has affected the consistency and standards in the design
of the G-MoMo applications. This reaffirms the results of
earlier studies by (Othman et al., 2018; Vingen et al., 2020), who
observed inconsistency in heading and a lack of adherence to
conventions, design principles, and application patterns with
the applications.
The G-MoMo applications do not have search field options,
making finding the required services difficult and affecting the
applications’ user control & freedom, flexibility & efficiency.
This is in line with (Paramitha et al., 2018), who identified a
lack of search features in the application. (Eliseo et al., 2017)
recommended that search features be strategically placed on
the application to search for required services. The G-MoMo
applications lack the necessary steps vital for recovery in case
the applications crush. This affected the applications’ error
diagnosis and recovery. It was well confirmed by other studies
(Caro-Alvaro et al., 2018; Höhn and Bongard-Blanchy, 2020),
who observed that the applications do not indicate the error
occurrence and the error was not explained in plain language.
(Eliseo et al., 2017) recommended that error messages be
displayed clearly. The G-MoMo applications also lack help and
documentation components for novice users. The instructions
on using the applications are not clear, and they also lack a
panel of tips and tricks for the application, thus making it
difficult for novice users to understand the navigation menu.
This usability issue then results in errors during G-MoMo
applications’ usage and makes it difficult for the subscribers to
recall the steps involved in the application’s usage. This
outcome is consistent with the earlier studies by (Jeddi et al.,
2020; Kekkonen and Oinas-Kukkonen, 2019), who mentioned
that the applications lacked any features related to the help and
documentation and tooltips/instructions that provide helpful
guidance to users. In addition, (Abidin et al., 2019) suggested
that applications should have a help and documentation
feature to access help easily and be explained in plain language
so that new users can understand.
The discussion of the results for the usability testing of the
G-MoMo applications is: The users found it easy to learn how
to use the G-MoMo applications, thus, enhancing their
performance. This finding is also reported in earlier studies by
(A’bas et al., 2021; Al-Gayar et al., 2021; Byun et al., 2020),
where they found that the systems were easy to learn. The G-
MoMo applications were highly effective because they allowed
mobile money subscribers to complete the tasks accurately to
achieve their specified goals. This finding is logical to the
studies by (A’bas et al., 2021; Byun et al., 2020; Lowe et al.,
2021). The overall usability testing result is positive for the
effectiveness of applications. The G-MoMo applications are
efficient since users take less time to complete tasks accurately.
This reconfirmed the results of earlier studies by (A’bas et al.,
2021; Hussain and Omar, 2020; Lowe et al., 2021; Zakaria et al.,
2020), where they found the applications efficient for the users.
The users also agreed that they would remember navigating
between G-MoMo applications pages the next time they use it.
This is supported by (Alturki et al., 2020; Putri et al., 2021;
Sukmasetya et al., 2020; Zakaria et al., 2020), who reported that
the users easily remember the steps followed while using the
applications. The participants reported having encountered
fewer errors while using the G-MoMo applications. This finding
is consistent with the earlier studies by (Alturki et al., 2020; Putri
et al., 2021; Zakaria et al., 2020), where it was mentioned that the
total number of errors decreased while using the applications.
The users achieved satisfaction with the features, functionalities,
design, information and display quality of the G-MoMo
applications. This finding was also reported in earlier studies by
(A’bas et al., 2021; Al-Gayar et al., 2021; Lowe et al., 2021; Putri et
al., 2021), where they observed that the users were satisfied with
the applications’ functionalities and design. The participants
mentioned that the G-MoMo applications were easy to use,
which helped them achieve satisfaction. This outcome is reported
in studies by (Al-Gayar et al., 2021; Byun et al., 2020), where it
was noticed that the applications had excellent content
knowledge and were pretty easy to use. The G-MoMo
applications have a high aesthetic rating because they are
attractive to the participants. It is also affirmed by (Santesteban-
Echarri et al., 2020), who observed that the applications are
aesthetically designed. The G-MoMo applications were helpful
for mobile money subscribers since they can perform the
required services. It is further supported by (Al-Gayar et al.,
2021; Kumar et al., 2020), who reported that the applications were
useful in achieving their intended goals. Other usability
attributes identified included integration and understandability.
CONCLUSION
With mobile money’s wide adoption and usage in
developing countries to foster financial inclusion, the current
2FA schemes suffer severe security issues. The researchers
developed secure G-MoMo applications to resolve the security
issues encountered. This paper, therefore, validated the
developed secure G-MoMo applications using heuristic
evaluation and usability testing. The three prototypes of the
native G-MoMo applications were validated to identify usability
issues and defects with the applications’ user interface. The
usability issues identified in the heuristic evaluation included
lack of forward navigation buttons, lack of uniformity in the
application’s menu title, lack of search field options, lack of
actions needed for recovery, and lack of help & documentation.
While, the usability testing results confirm that the three G-
MoMo applications’ performance proved good in learnability,
effectiveness, efficiency, memorability, and errors. It also
provided user satisfaction, ease of use, aesthetics, usefulness,
integration, and understandability. These validations were
carried out independently with no biases that could have
influenced the study’s outcome. Therefore, the developers must
improve the interface designs of G-MoMo applications by fixing
the usability problems identified to make them more reliable and
increase users’ overall satisfaction.
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
13 / 14
REFERENCES
A’bas, N.N., Rahim, S.S., Dolhalit, M.L., Saifudin, W.S.N.,
Abdullasim, N., Parumo, S., Omar, R.N.R., Khair,
S.Z.M., Kalaichelvam, K., Izhar, S.I.N., 2021.
Development and Usability Testing of a Consultation
System for Diabetic Retinopathy Screening.
International Journal of Advanced Computer Science
and Applications 12.
https://doi.org/10.14569/IJACSA.2021.0120522
Abidin, S.R.Z., Fadzilah, S., Sahari, N., 2019. Heuristic
Evaluation of Serious Game Application for Slow-
reading Students. International Journal of Advanced
Computer Science and Applications.
Agbezoutsi, K.E., Urien, P., Dandjinou, T.M., 2021. Mobile
money traceability and federation using blockchain
services. Annals of Telecommunications 76, 223–233.
https://doi.org/10.1007/s12243-021-00840-4
Agbezoutsi, K.E., Urien, P., Dandjinou, T.M., 2019. Towards
Blockchain Services For Mobile Money Traceability
And Federation, in: 2019 3rd Cyber Security in
Networking Conference (CSNet). pp. 14–20.
https://doi.org/10.1109/CSNet47905.2019.9108970
Al-Gayar, S., Goga, N., Al-Habeeb, N., Ali, H., Marin, I.,
Shubber, M., 2021. Testing the Usability of the
MediCare System. Turkish Journal of Computer and
Mathematics Education (TURCOMAT) 12, 3227–3237.
https://doi.org/10.17762/turcomat.v12i3.1569
Ali, G., Dida, M., Sam, A., 2021. A Secure and Efficient Multi-
Factor Authentication Algorithm for Mobile Money
Applications. Future Internet 13, 1–31.
https://doi.org/10.3390/fi13120299
Ali, G., Dida, M., Sam, A., 2020a. Two-Factor Authentication
Scheme for Mobile Money: A Review of Threat Models
and Countermeasures. Future Internet 12, 1–27.
https://doi.org/10.3390/fi12100160
Ali, G., Dida, M., Sam, A., 2020b. Evaluation of Key Security
Issues Associated with Mobile Money Systems in
Uganda. Information 11, 1–24.
https://doi.org/10.3390/info11060309
Alturki, R., AlGhamdi, M., Gay, V., Awan, N., Kundi, M.,
Alshehri, M., 2020. Analysis of an ehealth app: Privacy,
security and usability. International Journal of
Advanced Computer Science and Applications 11, 209–
214. https://doi.org/10.14569/IJACSA.2020.0110428
Ammar, L.B., 2019. A Usability Model for Mobile Applications
Generated with a Model-Driven Approach.
International Journal of Advanced Computer Science
and Applications 10.
https://doi.org/10.14569/IJACSA.2019.0100218
Ayeb, S., Hemery, B., Jeanne, F., Cherrier, E., 2022. Community
detection for mobile money fraud detection. IEEE,
Paris, France, pp. 1–6.
https://doi.org/10.1109/SNAMS52053.2020.9336578
Babatunde, F.O., Macdermid, J., Grewal, R., Macedo, L.G.,
Szekeres, M., 2020. Development and Usability Testing of
a Web-Based and Therapist-Assisted Coping Skills
Program for Managing Psychosocial Problems in
Individuals With Hand and Upper Limb Injuries: Mixed
Methods Study. JMIR Human Factors 7.
Bajcar, B., Borkowska, A., Jach, K., 2020. Asymmetry in Usability
Evaluation of the Assistive Technology among Users
With and Without Disabilities. null 36, 1849–1866.
https://doi.org/10.1080/10447318.2020.1798084
Ball, L.H., Bothma, T.J.D., 2018. Heuristic evaluation of e-
dictionaries. Library Hi Tech 36, 319–338.
https://doi.org/10.1108/LHT-07-2017-0144
Burkard, E.C., 2020. Usability Testing within a Devsecops
Environment, in: 2020 Integrated Communications
Navigation and Surveillance Conference (ICNS). pp. 1C1-
1. https://doi.org/10.1109/ICNS50378.2020.9222919
Byun, D.-H., Yang, H.-N., Chung, D.-S., 2020. Evaluation of
Mobile Applications Usability of Logistics in Life
Startups. Sustainability 12.
https://doi.org/10.3390/su12219023
Caro-Alvaro, S., García, E., García-Cabot, A., Marcos, L. de,
Martínez-Herráiz, J.J., 2018. Identifying Usability Issues
in Instant Messaging Apps on iOS and Android
Platforms. Mob. Inf. Syst. 2018, 2056290:1-2056290:19.
Chipa, N., Mwanza, B., 2021. Factors Impeding Mobile Money
Expansion in Zambia. International Journal of
Engineering and Management Research 11, 178–186.
https://doi.org/10.31033/ijemr.11.1.24
Eliseo, M.A., Casac, B.S., Gentil, G.R., 2017. A comparative study
of video content user interfaces based on heuristic
evaluation, in: 2017 12th Iberian Conference on
Information Systems and Technologies (CISTI). pp. 1–6.
https://doi.org/10.23919/CISTI.2017.7975820
Hertzum, M., 2020. Usability Testing: A Practitioner’s Guide to
Evaluating the User Experience. Synthesis Lectures on
Human-Centered Informatics 13, i–105.
Hewett, T.T., 1986. The role of iterative evaluation in designing
systems for usability, in: People and Computers II:
Designing for Usability. Cambridge University Press,
Cambridge, pp. 196–214.
Höhn, S., Bongard-Blanchy, K., 2020. Heuristic evaluation of
COVID-19 chatbots. Springer, pp. 131–144.
Hussain, A., Omar, A.M., 2020. Usability evaluation model for
mobile visually impaired applications. International
Journal of Interactive Mobile Technologies 14, 95–107.
https://doi.org/10.3991/ijim.v14i05.13349
Islam, M.N., Bouwman, H., Islam, A.N., 2020. Evaluating web
and mobile user interfaces with semiotics: An empirical
study. IEEE Access 8, 84396–84414.
Jeddi, F.R., Nabovati, E., Bigham, R., Farrahi, R., 2020. Usability
evaluation of a comprehensive national health
information system: A heuristic evaluation. Informatics in
Medicine Unlocked 19, 100332.
Ali G. et al. / J INFORM SYSTEMS ENG, 7(3), 15751
14 / 14
Jeong, J., Kim, N., In, H.P., 2020. GUI information-based
interaction logging and visualization for asynchronous
usability testing. Expert Systems with Applications 151,
113289.
Kekkonen, M., Oinas-Kukkonen, H., 2019. Social Comparison
in Behavior Change Support Systems: Heuristic
Evaluation of a System’s Usability.
Kous, K., Pušnik, M., Heričko, M., Polančič, G., 2020. Usability
evaluation of a library website with different end user
groups. Journal of Librarianship and Information
Science 52, 75–90.
Kumar, B.A., Goundar, M.S., Chand, S.S., 2020. A framework
for heuristic evaluation of mobile learning applications.
Education and Information Technologies 25, 3189–3204.
Lowe, C., Sing, H.H., Browne, M., Alwashmi, M.F., Marsh, W.,
Morrissey, D., 2021. Usability testing of a digital
assessment routing tool: protocol for an iterative
convergent mixed methods study. JMIR research
protocols 10, e27205.
Lynn, N.D., Sourav, A.I., Setyohadi, D.B., 2020. Increasing User
Satisfaction of Mobile Commerce using Usability.
International Journal of Advanced Computer Science
and Applications 11.
https://doi.org/10.14569/IJACSA.2020.0110839
Nabovati, E., Vakili-Arki, H., Eslami, S., Khajouei, R., 2014.
Usability evaluation of Laboratory and Radiology
Information Systems integrated into a hospital
information system. J Med Syst 38, 35.
https://doi.org/10.1007/s10916-014-0035-z
Nielsen, J., 2021. How many test users in a usability study?
[WWW Document]. URL
https://www.nngroup.com/articles/how-many-test-
users/
Nielsen, J., 1994a. 10 Usability heuristics for user interface
design [WWW Document]. URL
https://www.nngroup.com/articles/ten-usability-
heuristics/ (accessed 6.22.21).
Nielsen, J., 1994b. Severity ratings for usability problems.
[WWW Document]. URL
https://www.nngroup.com/articles/how-to-rate-the-
severity-of-usability-problems/ (accessed 7.23.21).
Nielsen, J., Mack, R.L., 1994. Heuristic evaluation. Usability
inspection methods.
Othman, M.K., Sulaiman, M.N.S., Aman, S., 2018. Heuristic
Evaluation: Comparing Generic and Specific Usability
Heuristics for Identification of Usability Problems in a
Living Museum Mobile Guide App. Advances in
Human-Computer Interaction 2018, 1518682.
https://doi.org/10.1155/2018/1518682
Paramitha, A.A.I.I., Dantes, G.R., Indrawan, G., 2018. The
Evaluation of Web Based Academic Progress
Information System Using Heuristic Evaluation and
User Experience Questionnaire (UEQ). 2018 Third
International Conference on Informatics and
Computing (ICIC) 1–6.
Pimentel, J.L., 2010. A note on the usage of Likert Scaling for
research data analysis. USM R&D Journal 18, 109–112.
Putri, N.L., Maluana, A., Maryam, S.D., Juraida, A., 2021.
Improving online course based on the result of usability
testing methods. Psychology and Education 1, 6373–6382.
https://doi.org/10.17762/pae.v58i1.3795
Ramayasa, I.P., Candrawibawa, I.G.A., 2021. Usability
Evaluation of Lecturer Information Systems Using Sirius
Framework and Moscow Technique. Scientific Journal of
Informatics 8, 16–23.
Rwiza, S., Kissaka, M., Kapis, K., 2020. A methodology for
evaluating security in MNO financial service model.
IEEE, pp. 1–10.
Salari, R., Kalhori, S.R.N., GhaziSaeedi, M., Jeddi, M., Nazari, M.,
Fatehi, F., 2021. Mobile-based and cloud-based system for
self-management of people with type 2 diabetes:
Development and usability evaluation. Journal of medical
Internet research 23, e18167.
Salman, H.M., Ahmad, W.F.W., Sulaiman, S., 2018. Usability
evaluation of the smartphone user interface in supporting
elderly users from experts’ perspective. Ieee Access 6,
22578–22591.
Santesteban-Echarri, O., Tang, J., Fernandes, J., Addington, J.,
2020. Development and usability testing of SOMO, a
mobile-based application to monitor social functioning
for youth at clinical high-risk for psychosis. Digital
Psychology 1, 4–19.
Sukmasetya, P., Setiawan, A., Arumi, E., 2020. Usability
evaluation of university website: a case study. IOP
Publishing, p. 012071.
Tremoulet, P.D., Shah, P.D., Acosta, A.A., Grant, C.W., Kurtz,
J.T., Mounas, P., Kirchhoff, M., Wade, E., 2021. Usability
of Electronic Health Record–Generated Discharge
Summaries: Heuristic Evaluation. Journal of medical
Internet research 23, e25657.
Vingen, D., Andrews, E.J., Ferati, M., 2020. Usability in Patient-
Oriented Drug Interaction Checkers—A Scandinavian
Sampling and Heuristic Evaluation. MDPI, p. 42.
Wahyuningrum, T., Kartiko, C., Wardhana, A.C., 2020. Exploring
e-Commerce Usability by Heuristic Evaluation as a
Compelement of System Usability Scale. IEEE, pp. 1–5.
Weichbroth, P., 2020. Usability of mobile applications: a
systematic literature study. IEEE Access 8, 55563–55577.
Wirasasmiata, R., Uska, M., 2019. Evaluation of E-Rapor
Usability using Usability Testing Method. Atlantis Press,
pp. 343–346.
Zakaria, N., Wahabi, H., Al Qahtani, M., 2020. Development and
usability testing of Riyadh Mother and Baby Multi-center
cohort study registry. Journal of Infection and Public
Health 13, 1473–1480.
