ChapterPDF Available

Usability Evaluation Methods: How Usable Are They?

Authors:

Abstract

Technological advancements have occurred one after another since the industrial revolution. Moreover, thefourth industrial revolution, aka Industry 4.0, has led to a new era where human-machine coordination is inevitable.Technological advances, increasing competition in the market, and more sophisticated customer expectations havemade usability, which was once a luxury, a necessity in high demand. Therefore, user experience is more importantthan ever, and usability is essential to this experience. Usability is an umbrella term to define the learnability,efficiency and memorability of a product, errors a user makes while using the product and the satisfaction a user getsfrom using the product. In other words, it is a term that covers the efficiency, effectiveness and satisfiability of aproduct or a system. Considering the expectations, needs and limitations of the potential user during and after thedesign process is of vital importance for the success of a product. As usability issues have become prominent, themethods available to evaluate the usability of a product or a system are worth addressing. This chapter covers theterm usability and the evaluation methods with the strength and weaknesses of each technique
Global Studies On Management
Information Systems
Editors
Elif KARTAL, Emre AKADAL, Gökhan ÖVENÇ, Saeed TABAR
100th ANNIVERSARY OF THE REPUBLIC BOOKS
GLOBAL STUDIES ON
MANAGEMENT INFORMATION
SYSTEMS
EDITORS
Assoc. Prof. Elif KARTAL
Istanbul University, Faculty of Economics, Department of Management Information Systems, Istanbul, Turkiye
Assoc. Prof. Emre AKADAL
Istanbul University, Faculty of Economics, Department of Management Information Systems, Istanbul, Turkiye
Assoc. Prof. Gökhan ÖVENÇ
Istanbul University, Faculty of Economics, Department of Management Information Systems, Istanbul, Turkiye
Assist. Prof. Saeed TABAR
Ball State University, Department of Information Systems and Operations Management, Muncie, U.S.A.
iv




















This book was published as part of the Istanbul University Press’ 100th Anniversary
of the Republic book project.
v
CONTENTS
 ....................................................................................................................................................vii
CHAPTER 1 / BÖLÜM 1




 ........................................................................................................................................
CHAPTER 2 / BÖLÜM 2


 ...........................................................................................................................................53
CHAPTER 3 / BÖLÜM 3



 .............................................................................................................................
CHAPTER 4 / BÖLÜM 4


 .................................................................................................................
CHAPTER 5 / BÖLÜM 5

THE POTENTIAL AND DISADVANTAGED GROUPS


 ..................................................................................
CHAPTER 6 / BÖLÜM 6


 .............................................................................................................................
CHAPTER 7 / BÖLÜM 7




Gökhan ÖVENÇ ......................................................................................................................................................
CHAPTER 8 / BÖLÜM 8


Elif KARTAL ........................................................................................................................................................... 
CHAPTER 9 / BÖLÜM 9


Zeki ÖZEN ...............................................................................................................................................................
vi
CHAPTER 10 / BÖLÜM 10

DIMENSIONAL DATASETS


 .............................................................................................................................................................
CHAPTER 11 / BÖLÜM 11


Mehmet Hakan SATMAN, Emre AKADAL ...........................................................................................................
CHAPTER 12 / BÖLÜM 12


 ...................................................................................................................
CHAPTER 13 / BÖLÜM 13


 .................................................................................................................................333
CHAPTER 14 / BÖLÜM 14


 .............................................................................................................................................................
CHAPTER 15 / BÖLÜM 15


 ............................................................................................................................
CHAPTER 16 / BÖLÜM 16


 ........................................................................................................................................................
CONTENTS
vii
PREFACE
th

  th          
           

                

 
   
st
          

   


 
               
    


               





to failure to set their strategies based on evidence or to adjust to technological changes.
           
 
 

  


         


       



viii
    

      
       
        
missions.



      





     


     






           
  
              


PREFACE
CHAPTER 16 / BÖLÜM 16
USABILITY EVALUATION METHODS:
HOW USABLE ARE THEY?


MUTLU*




ABSTRACT
        



                 

    
  
 
  

Keywords:
ÖZ
      


             


 




USABILITY EVALUATION METHODS: HOW USABLE ARE THEY?
            


Tuğba MUTLU

































USABILITY EVALUATION METHODS: HOW USABLE ARE THEY?
























        









Tuğba MUTLU





         









 











        





USABILITY EVALUATION METHODS: HOW USABLE ARE THEY?


























think aloud protocol


Remote testing



Tuğba MUTLU

coaching method


Performance measurement



Co-discovery learning technique


question asking protocol



retrospective testing technique


teaching method



shadowing method









USABILITY EVALUATION METHODS: HOW USABLE ARE THEY?

















cognitive walkthrough



Heuristic evaluation



feature inspection method



pluralistic walkthrough

Perspective-based inspection 

Tuğba MUTLU
Formal usability inspection


consistency inspection method


standards inspection,















Field observation



focus group method




interviewing the user

USABILITY EVALUATION METHODS: HOW USABLE ARE THEY?
system logs





Questionnaires and surveys




 
 





 





 


















Tuğba MUTLU







          
























USABILITY EVALUATION METHODS: HOW USABLE ARE THEY?











Tuğba MUTLU


Journal of
Web Librarianship, 16
Formative Evaluation

Summative Evaluation

Cost-Justifying Usability: An Update for an Internet Age

International Journal of Medical Informatics,128

Usability Evaluation


The human-computer interaction handbook: fundamentals, evolving technologies, and emerging app-
lications 
Usability Evaluation Of Mobile And Desktop Websites: A Study Of Comparing Usability
Evaluation Methodologies

A literature review about usability evaluation methods for
e-learning platforms

  International Journal of Compu-
ter Trends and Technology, 22    

Communications and Infor-
mation Technology (ICCIT)

Proceedings of the 5th International Conference on Sustainable In-
formation Engineering and Technology

International Journal of Human-Computer Interaction, 13 

Usability Testing

Communicationf of the ACM,
48(
Ergonomics of human-system interaction — Part 210: Humancentred design for intera-
ctive systems. Technical report, International Organization for Standardization

 Usability Evaluation in Industry

Formative vs. Summative Evaluations 

USABILITY EVALUATION METHODS: HOW USABLE ARE THEY?
          
Journal of Librarianship and Information Science52 

Usability testing and system evaluation

Human-Computer Interaction. Applications and Services, 85  

Defining Usability

Usability Engineering
Usability 101: Introduction to Usability. 


  

PC/Computing, 5
Handbook Of Usability Testing: How to Plan, Design, and Conduct Effective
Tests 
Usability First — Why Usability Design Matters to UI/UX Designers







 Usability Evaluation Basics.


Behaviour & Information Technology, 22


Symbiosis Of Human And Artifact: Future Computing And Design For Human-Com-
puter Interaction, 20B


International Journal of Medi-
cal Informatics, 83

2014 3rd International Conference on User Science and En-
gineering (i-USEr)

The Communications of the Association for Information Systems, 9



Tuğba MUTLU

Proceedings of the Human Factors and Ergonomics Society Annual Meeting42

Article
Full-text available
We have long been committed to improving Usability Evaluation and one of the proposals we have been working on the most is the use and improvement of the Heuristic Evaluation (HE) technique. With this in mind, we proposed an improvement which was tested in an experiment. This article describes an experiment carried out in a real business professional context. Fifteen usability experts from a reputable company evaluated eight websites (four supermarket and four bank platforms) using our HE proposal for the first time in real-world scenarios. This experimentation analyzed two main aspects: firstly, whether individual or group evaluations affect the final result, and secondly, whether the heuristic evaluation technique is effective in a real business and professional context. Regarding the Usability Percentage (UP) event, the results indicate that there was little difference between group and individual evaluations. The mean UP for the group was 57.88%, while the mean UP for individuals was 56.66%. The experiment provided sufficient information to suggest a new version of our HE methodology, specifically designed to improve results in real-life contexts. Furthermore, the experiment’s findings support the proposal of this new methodology, which is better suited to the business environment.
Conference Paper
Full-text available
The term Industry 4.0 derives from the new (fourth) industrial revolution enabling suppliers and manufacturers to leverage new technological concepts like Internet of Things, Big Data, and Cloud Computing: New or enhanced products and services can be created, costs be reduced, and productivity be increased. Similar terms are Smart Factory or Smart Manufacturing. The ideas, concepts, and technologies are not hyped anymore-they have at least partly become a reality, but many software specifications and development aspects are still not sufficiently covered, e.g., standardization, specification, and modeling languages. This paper presents an Industry 4.0 process modeling language (I4PML) that is an extension (UML profile with stereotypes) of OMG's BPMN (Business Process Model and Notation) standard. We also describe a method for the specification of Industry 4.0 applications using UML and I4PML.
Article
Full-text available
To ensure a certain degree of usability, a library website should be carefully designed, especially since end users constitute a multitude of people with different needs and demands. The focal objective of this research was to investigate how different types of end users (i.e. pupils, students, the working population, seniors and researchers) respond to a library website in terms of its effectiveness, efficiency and satisfaction, which together represent its usability. The answers were obtained by performing formal usability testing, including think-aloud protocol, log analysis and questionnaires. The results of the statistical analysis show that different groups of end users achieve different levels of effectiveness and efficiency, while there is no significant difference between groups in satisfaction level. The results also indicate that participants did not achieve the threshold for a usable website. Based on the identified weaknesses, researchers present recommendations for improving a website’s usefulness, especially for non-experienced users. This research has two main contributions: (1) the connection between the theoretical definition and practical use of ISO 9241-11 attributes and (2) a usability testing procedure with a measurement framework applicable for different types of users in a specific domain, which could be applied to other domains.
Article
National library websites are essential vehicles for the general public to access national information and resources. The purpose of this study is to examine the usability, user experience (UX), and information architecture (IA) of a sample of national library websites. Through a three-phased multi-methods investigation featuring (1) a content inventory of 28 websites, (2) a web IA and usability evaluation of five websites, and (3) usability tests for three websites, the study results revealed statistically significant gender differences in participants’ task performance. Significant correlations were found between participants’ satisfaction and the content, visual design, and IA, as well as between heuristic evaluation scores and participants’ ratings. Several factors affected participants’ satisfaction: quality of information, the trustworthiness of the content, the perceived credibility of the content producers/providers, the website’s overall structure, and the visual design. The findings of the research also helped to identify common content elements on website homepages and to present a practical procedure for assessing national library websites.
Article
Background: It is generally assumed that usability benchmarking instruments are technology agnostic. The same methods for usability evaluations are used for digital commercial, educational, governmental and healthcare systems. However, eHealth technologies have unique characteristics. They need to support patients' health, provide treatment or monitor progress. Little research is done on the effectiveness of different benchmarks (qualitative and quantitative) within the eHealth context. Objectives: In this study, we compared three usability benchmarking instruments (logging task performance, think aloud and the SUS, the System Usability Scale) to assess which metric is most indicative of usability in an eHealth technology. Also, we analyzed how these outcome variables (task completion, system usability score, serious and critical usability issues) interacted with the acceptance factors Perceived benefits, Usefulness and Intention to use. Methods: A usability evaluation protocol was set up that incorporated all three benchmarking methods. This protocol was deployed among 36 Dutch participants and across three different eHealth technologies: a gamified application for older adults (N = 19), an online tele-rehabilitation portal for healthcare professionals (N = 9), and a mobile health app for adolescents (N = 8). Results: The main finding was that task completion, compared to the SUS, had stronger correlations with usability benchmarks. Also, serious and critical issues were stronger correlated to task metrics than the SUS. With regard to acceptance factors, there were no significant differences between the three usability benchmarking instruments. Conclusions: With this study, we took a first step in examining how to improve usability evaluations for eHealth. The results show that listing usability issues from think aloud protocols remains one of the most effective tools to explain the usability for eHealth. Using the SUS as a stand-alone usability metric for eHealth is not recommended. Preferably, the SUS should be combined with task metrics, especially task completion. We recommend to develop a usability benchmarking instrument specifically for eHealth.
Conference Paper
This research evaluated the usability of educational websites. In particular, it focused on nine Jordanian university websites. Evaluation criteria for assessing the usability of educational websites were developed; these consisted of five categories: navigation, architecture / organisation, ease of use and communication, design and content. Then 237 students were asked to provide ratings for nine Jordanian university websites using the developed usability criteria and their categories. Students were also asked to report qualitatively what they liked and disliked regarding the design of the tested websites. The results provided detailed information for each tested website regarding their conformance with the developed usability criteria. The results showed that the majority of the students were satisfied with the usability of the Jordanian university websites. Specifically, the results showed that the students were satisfied with the content and navigation (ease of use) issues of the tested websites, but were dissatisfied with the design of the websites. Common and unique strengths and weaknesses related to design issues of the investigated websites were identified based on the quantitative results and the qualitative data obtained from the open-ended questions.