Content uploaded by A A Istri Ita Paramitha
Author content
All content in this area was uploaded by A A Istri Ita Paramitha on Apr 28, 2020
Content may be subject to copyright.
A preview of the PDF is not available
The Evaluation of Web Based Academic Progress Information System Using Heuristic Evaluation and User Experience Questionnaire (UEQ)
Abstract and Figures
This research is aimed to evaluate the interface and user experience of SIsKa, and then to give recommendation of the system interface. SIsKA is a web-based information system which is developed to thesis management computer science department of post graduate students Ganesha University of education. This evaluation is aimed to identify the problems faced by the users when using the system and measure the respond of the user when interacting through the system. The interface evaluation is done by means of heuristic evaluation technic; meanwhile the evaluation of user experience is done by means of User Experience Questionnaire. In the heuristic evaluation done by three evaluators, it is found that there are 40 violations over the 10 Nielsen's principal. Based on the UEQ result, it is found that in 20 SIsKA users, the average score for each aspect are: attractiveness aspect is 1,59, perspicuity aspect is 1,75, efficiency aspect is 1,64, dependability aspect is 1,40, stimulation aspect is 1,56, and novelty aspect is 1,10. After comparing the average score in dataset product of UEQ Data Analysis Tool, it is found that the attractiveness, perspicuity, efficiency, and novelty aspects of SIsKA are categorized as good. The dependability aspect is categorized as above average, and stimulation aspect is categorized as excellent. The recommendation of interface of SIsKA users is focused on 10 most severe problems found in heuristic evaluation. This recommendation is done in the form of wireframe, so it will be easier to draw the real framework of the website. The result of this research can be used as the suggestion and reference for the SIsKA developers to improve the quality of SIsKA particularly on the system interface.
Figures - uploaded by A A Istri Ita Paramitha
Author content
All figure content in this area was uploaded by A A Istri Ita Paramitha
Content may be subject to copyright.
... Moreover, testimonies of workable procedures for improving mobile apps based on user experience are lacking. Mobile app user experience has been surveyed only in other disciplines and service industries (Paramitha et al. 2018;Heshmati et al. 2021;Sabukunze and Arakaza 2021;Setiyawati and Bangkalang 2022). Our study employed a multifaceted approach to assess user experiences with mobile software applications by employing three distinct questionnaires: i) a User Experience Questionnaire (UEQ), ii) a System Usability Scale (SUS), and iii) a qualitative questionnaire (open-ended). ...
... Permanasari et al. (2021) utilized both the UEQ and SUS methods to assess an augmented reality mobile application designed as an educational tool for children. Paramitha et al. (2018) employed the UEQ to evaluate SIsKA, a web-based information system developed for the management of theses of postgraduate students in the Computer Science department at Ganesha University. The review work of published studies revealed a lack of comprehensive methodology. ...
Mobile applications (apps) have become widely available as decision
support tools in agriculture and forestry worldwide. Owing to the
above-mentioned development trend, identifying testing methods
that can be easily implemented for evaluating mobile app development
targeting rural livelihood-related topics is urgently required. This
study examined user experiences with a previously launched and
published mobile app (EVO) that small-scale eucalyptus growers can
utilize to estimate their gross income and net profit from timber sales.
The survey incorporated three user experience assessment methods.
The study involved two rounds of data collection regarding the mobile
app: the first round concerned version 1.0, whereas the second round
referred to version 2.0. The aim was to identify the user experience of
the EVO app for guiding further software development to improve its
suitability among potential user groups, as well as to estimate the
feasibility and usability of survey methods for evaluating mobile apps
among smallholders and related stakeholders. The results indicated
a remarkable improvement in user satisfaction between the two
rounds. Using a combination of system usability scale, user acceptance
testing, and quality questions is recommended for collecting valuable
and useful feedback for the development of mobile apps.
... The UEQ has been shown to be a reliable and valid measure of user experience, with high levels of consistency and correlation with other measures of user satisfaction (Schrepp et al. 2014a;Schrepp 2015;Paramitha et al. 2018). Additionally, the questionnaire has been used in a variety of contexts, including web design (Hinderks et al. 2019), software development (Zhu et al. 2022), and AR and VR applications (Su et al. 2020;Somrak et al. 2019), making it a versatile and adaptable tool for researchers and practitioners alike (Somrak et al. 2021). ...
... Unlike other measures of user experience, which often rely on objective metrics such as task completion time or error rates, the UEQ asks users to self-report their feelings and impressions about the product or service being evaluated (Laugwitz et al. 2008). This allows researchers to capture the full range of user experiences, including both positive and negative emotions, and to identify areas where the software or system design may be lacking from the user's perspective (Schrepp et al. 2014a;Schrepp 2015;Paramitha et al. 2018). ...
Police investigation in real-life crime scenes is an essential aspect of forensic science education. However, the practicality of bringing young investigators to actual crime scenes is often hindered by the costs and challenges involved. In order to overcome these obstacles, new technologies such as mixed reality (MR) are being explored as potential solutions. MR technology offers an interactive and cost-effective way to simulate real-life crime scenes, providing a valuable training experience for young investigators. This paper presents a novel design of a MR system using Microsoft HoloLens 2.0, which is tailored to work in a spatial 3D scanned and reconstructed crime scene using FARO point cloud 3D scanner X130 blended with photogrammetry techniques. The system was developed through the lens of Experiential Learning Theory and designed using a participatory approach, providing a cost-effective solution to help trained Kuwaiti police officers enhance their investigative skills. In order to evaluate the system’s user experience and user interaction, the Questionnaire of User Interaction Satisfaction and User Experience Questionnaire were utilised. Forty-four young police officers evaluated the system. Police students showed positive levels of satisfaction with user interaction and overall user experience with minimal negative feedback. Female students showed higher satisfaction with the overall impression compared to male students. Based on the positive feedback regarding the system expansion, the system will be taken into the commercialisation stage in the future to be provided as an essential tool for crime scene education and investigation practices.
... The challenge is compounded in an online learning environment. Previous studies have examined the interplay between achievement emotions and academic performance (Pekrun et al. 2017), user experience and academic performance (Paramitha, Dantes, and Indrawan 2018), and the relationship between achievement emotions and user experience (Schrader and Grassinger 2021) in both face-to-face and online contexts. However, studies have yet to consider all three factors simultaneously in an online environment, particularly in computer science. ...
This study investigates the interplay between achievement emotions, user experience, and academic performance of online university assignments. Drawing on Pekrun’s Control-Value Theory (CVT), User Experience Theory (UXT), Self-Determination Theory (SDT), Cognitive Load Theory (CLT), and Expectancy-Value Theory (EVT), the research explores how these factors impact first-year students’ engagement and performance in a computer programming module. The sample comprised 350 responses across three assignments, with 292 responses for Assignment 1, 30 for Assignment 2, and 27 for Assignment 3. Achievement emotions were assessed using the Achievement Emotions Questionnaire (AEQ) across eight scales: enjoyment, hope, pride, anger, anxiety, shame, hopelessness, and boredom. The User Experience Questionnaire (UX) evaluated user experience across four scales: ease of use, interest, alignment, and appeal. Additionally, academic performance was measured as a percentage. Descriptive statistics revealed the prevalence of both positive and negative emotions, as well as varied user experiences. Correlation analysis indicated significant relationships between positive emotions (enjoyment, hope, pride) and higher academic performance, while negative emotions (anger, anxiety, hopelessness) were negatively correlated with performance. Positive user experiences (ease of use, interest, alignment, appeal) were also positively correlated with academic success. Reliability analysis confirmed the internal consistency of the scales, and regression analyses demonstrated that both achievement emotions and user experiences significantly predict academic performance. Recommendations for educators are to create assignments that minimise cognitive load, maximise positive user experiences, and support students’ intrinsic motivations.
... Evaluasi pengalaman pengguna dapat memberikan gambaran kepada pihak pengembang tentang respon pengguna ketika berinteraksi dengan antarmuka sistem dan juga tingkat penerimaan pengguna terhadap sistem. Evaluasi adalah tahap untuk mengamati keefektifan dan efisiensi produk serta mengukur bagaimana antarmuka pengguna dapat dengan mudah dipelajari baik dari sisi usability maupun user experience sehingga dapat mengidentifikasi masalah yang nantinya mungkin terjadi pada produk [1]. Untuk melakukan evaluasi pada sistem informasi maka teknikteknik dalam usability dapat diterapkan. ...
This study aims to determine the usability analysis results of the Teampal.id application in terms of users and find out interface improvement recommendations. Teampal.id is one of the start-ups under the guidance of the Primakara STMIK business incubator located in Denpasar-Bali. Start-up is a company that has just been established and is in the development and research phase to find the right market. Teampal.id is a start-up that includes a unique e-commerce that was formed in 2018. The product offered is a service for online farming communities. In this study using a combination of the Restrocpective Think Aloud (RTA) method and Post Study System Usability Questionnaire (PSSUQ). RTA is used to get results qualitatively while PSSUQ is to get quantitative results. This study used 30 respondents who came from active users of Teampal.id. Quantitative research results show that Teampal.id shows positive user acceptance results. This can be seen from the overall satisfaction score (OVERALL), which is 5.46 from a scale of 7. For system use (SYSUSE) scores 5.47, information quality (INFOQUAL) scores 5.47 and interface quality (INTERQUAL) scores 5.43. However, based on the results of research using the RTA method there are a number of things that must be corrected at the Teampal.id interface. Recommendations for improvements are made in high-fidelity wireframes, so they can be input in improving Teampal.id's interface. With this recommendation, it is expected to improve the quality of Teampal.id's user experience.Keywords : usability, RTA, PSSUABSTRAKPenelitian ini bertujuan untuk mengetahui hasil analisis kebergunaan (usability) dari aplikasi Teampal.id ditinjau dari sisi pengguna serta mengetahui rekomendasi perbaikan antarmuka. Teampal.id merupakan salah satu start-up binaan inkubator bisnis STMIK Primakara yang terletak di Denpasar-Bali. Start-up merupakan perusahaan yang baru saja didirikan dan berada dalam fase pengembangan dan penelitian untuk menemukan pasar yang tepat. Teampal.id adalah sebuah start-up yang termasuk unique e-commerce yang dibentuk pada tahun 2018. Produk yang ditawarkan adalah layanan untuk masyarakat berternak secara online. Dalam penelitian ini menggunakan metode kombinasi yaitu Restrocpective Think Aloud (RTA) dan Post Study System Usability Questionnaire (PSSUQ). RTA digunakan untuk mendapatkan hasil secara kualitatif sedangkan PSSUQ untuk mendapatkan hasil secara kuantitatif. Penelitian ini menggunakan 30 responden yang berasal dari pengguna aktif Teampal.id. Hasil penelitian secara kuantitatif bahwa Teampal.id menunjukkan hasil penerimaan pengguna yang positif. Hal ini dapat dilihat dari skor kepuasan secara keseluruhan (OVERALL) yaitu 5.46 dari skala 7. Untuk kegunaan sistem (SYSUSE)337 Jurnal Teknologi Informasi dan Komputer, Volume 6, Nomor 3, Oktober 2020memperoleh skor 5.47, kualitas informasi (INFOQUAL) memperoleh skor 5.47 dan kualitas antarmuka (INTERQUAL) memperoleh skor 5.43. Namun, berdasarkan hasil penelitian menggunakan metode RTA terdapat beberapa hal yang harus diperbaiki pada antarmuka Teampal.id. Rekomendasi perbaikan dibuat dalam high fidelity wireframe, sehingga dapat menjadi masukan dalam perbaikan antarmuka Teampal.id. Dengan rekomendasi ini diharapkan mampu meningkatkan kualitas pengalaman pengguna Teampal.id.Kata kunci: usability, RTA, PSSUQ
... Metode UEQ memberikan keunggulan yang tidak dapat diberikan oleh metode-metode lainnya seperti SUS, QUIS, dan SUPR-Q. Hasil analisis yang dilakukan dengan UEQ akan diberikan secara komprehensif, meliputi aspek kegunaan (usability) dan aspek pengalaman pengguna (user experience) [11]. Hasil pengukuran pengalaman pengguna pada website Pusat Pengembangan Karir dapat membantu dalam mengidentifikasi tren dan perubahan dalam kebutuhan pengguna sehingga dapat mengadaptasi dan mengembangkan fitur-fitur yang relevan dan inovatif. ...
Pembangunan National “Veteran” University of East Java has developed the Career Development Center website as a job vacancy information center to prepare graduates who are tough, innovative, and creative in the world of work. The Career Development Center website makes it easier to search for and find job opportunities that suit the field of study, skills, and interests of college graduates. A positive user experience in using the Career Development Center website can increase satisfaction and optimize career searches, so it is important to carry out measurements systematically and objectively. This study aims to measure the level of experience of users of the Career Development Center website using the User Experience Questionnaire (UEQ). UEQ measurements on the Career Development Center website focus on gaining an understanding of the Attractiveness, Pragmatic quality, and Hedonic quality aspects. UEQ provides measurements of technical and non-technical aspects related to the user's emotion or perception of pleasure. The results of measuring user experience on the Career Development Center website can help identify trends and changes in user needs so that they can adapt and develop relevant and innovative features. This study uses quantitative data by distributing 26 items of the UEQ questionnaire online to 350 alumni. The research results stated that the attractiveness aspect had the highest average value of 1.94. These results prove that the Career Development Center website has a high attractiveness value for users, thereby increasing the opportunity for continuous use. Apart from that, based on the high average stimulation value of 1.930, it can be proven that graduates who use the Career Development Center website get large benefits related to alumni career development.
Many curriculum guides are created to support teachers' adoption of digital games for learning. However, their impact on teachers' acceptance of games has not been studied. We investigate how the experience of theory-based curriculum guides for the educational game Discovery Tour Ancient Egypt affects teachers' confidence and acceptance of game-based learning. Teachers (n = 100) reported positive experiences with the guide. Analysis revealed that the guide's pragmatic qualities predict teachers' acceptance of DTAE. This effect is mediated by their digital self-efficacy. Results suggest well-designed curriculum guides can increase teachers' acceptance of GBL, regardless of their prior knowledge of games.
The rapid growth of mobile banking technology requires special attention to user experience so that applications can be used efficiently and satisfactorily. App users perform a series of tasks designed to cover the app's key features, while researchers observe and record responses, difficulty, and time required to complete the tasks. The results of usability testing provide in-depth insight into the specific problems users face when using the Hijra Bank mobile banking application. Next, the User Experience Questionnaire (UEQ) is used to measure qualitative aspects of user experience, such as satisfaction, fear, stimulation, and concentration. Data from UEQ is used to identify design elements that contribute positively or negatively to the user experience. The results of this research provide an in-depth view of the strengths and weaknesses of the mobile banking application by Hijra Bank in Indonesia. This information can be used by application developers to make necessary improvements and improvements, ensuring that users have a satisfactory and efficient experience when using mobile banking services by Hijra Bank. This research also contributes to increasing awareness and understanding in the development of user-oriented mobile banking applications by Hijra Bank in Indonesia.
Sejalan dengan UU No. 14 Tahun 2008 tentang Keterbukaan Informasi Publik, Universitas Pendidikan Ganesha sebagai lembaga terbuka telah membangun situs PPID untuk memenuhi kebutuhan data publik. Situs PPID dibangun untuk memudahkan akses berbagai dokumen atau informasi yang tersedia dan telah dikategorikan sesuai dengan kebutuhan pada halaman Daftar Informasi Publik, sehingga memudahkan masyarakat untuk mengaksesnya. Situs PPID belum pernah dilakukan analisis secara menyeluruh, terutama dalam aspek usabilitynya, Evaluasi usability memiliki banyak teknik dan tahapan untuk dapat menghasilkan rekomendasi perbaikan yang valid. Sehingga, langkah awal yang dapat dilakukan adalah meninjau tingkat pengalaman penggunaan situs PPID sebelum melakukan evaluasi lanjutan. Evaluasi berbasis kuesioner dengan menggunakan user experience questionnaire digunakan dalam penelitian ini. Pada kuesioner ini terdapat 6 indikator yang dinilai dan dibagi ke dalam 26 butir pernyataan. Dari 40 responden, dapat disimpulkan bahwa secara umum tingkat keterlibatan klien terhadap aplikasi PPID Undiksha dinilai positif dilihat dari aspek attractiveness (rata-rata: 1.963), efficiency (rata-rata: 1.906), dependability (rata-rata: 1.775), dan stimulation (rata-rata: 1.713) termasuk dalam kategori sangat baik, sementara aspek perspicuity (rata-rata: 1.869) dan novelty (rata-rata: 1.356) termasuk dalam kategori baik. Penelitian selanjutnya dapat melakukan kombinasi dengan open-ended questions atau wawancara untuk mendapatkan pendapat pengguna mengenai sistem dan masukan pengembangannya.
Introduction: Simultaneously with the COVID -19 epidemic, many efforts were made to manage and treat the disease. One of the main activities was the development of health information systems to help monitor this disease, but due to the speed of the disease’s spread, there was not enough time to evaluate the usability of these systems. Therefore, the present research was conducted to evaluate the usability of the Medical Care Monitoring Center (MCMC) system . Method: In this cross -sectional descriptive study (2022), the usability of the MCMS system was evaluated using a heuristic evaluation method. The system’s usability was investigated based on the Nielsen’s usability principles. All the problems identified by evaluators were collected in a single list and the mean severity of each problem was calculated. Results: There were 41 usability problems. The lowest mismatch with usability principles was related to "Help and documentation" (n=1) and the highest one was related to "user control and freedom" (n=8) and "Match with real world" (n=7). In addition, most of the identified problems had a relatively high degree of severity and were placed in the category of major problems. Conclusion: Using the usability evaluation, a significant number of usability problems of the system were identified which, if not resolved, can cause increased errors, dissatisfaction, and decreased quality of information; therefore, it is necessary to solve these problems by the designers and developers of this system.
Tujuan dari artikel ini adalah melakukan sebuah review terhadap desain antar muka (interface) pada aplikasi yang bernama System Online Payment Point PT Pos Indonesia atau disingkat SOPPPOS. Review menggunakan evaluasi interaksi manusia komputer yang bernama evaluasi heuristik yang diperkenalkan oleh Molich dan Nielsen. Berdasarkan hasil evaluasi menyatakan bahwa secara umum desain interface dan kemudahan pengguna (usability) dari aplikasi SOPPPOS cukup baik, namun masih ada beberapa hal yang perlu diperbaiki terutama dalam karakteristik kemudahan penggunaan.
Kata kunci: review, interface, aplikasi SOPPPOS, evaluasi, heuristik.
Learning Management System (LMS) becomes more accessible to provide a virtual learning environment. The growth of pervasive and ubiquitous computing promotes development and improvement LMS rapidly. Shifting paradigm in designing LMS from technology-centered to user-centered design drives the developer to consider user experience as the primary concern to improve LMS. Therefore, evaluation and measurement of user experience for LMS become crucial. There were some studies have been conducted to evaluate and measure an LMS product in to gain feedback from the user. However, the studies implement a partial view of LMS product. They have conducted researches that limited to comparing LMS and getting perception rather than observing the quality of the LMS. This paper proposed an evaluation and measurement of user experience (UX) for LMS by implementing user experience questionnaire tools. The study aims to get feedback, benchmark and identify UX while interacting with an LMS. The data are taken from 38 students who have experience using two LMS in a different semester. The first LMS based on Moodle was implemented in the first semester (LMS 1), on the other hand, the LMS using Schoology was applied in the second semester (LMS 2). As the results, LMS 1 and LMS 2 got the relative different impression in user perception. Similarly, they also achieved a different level of relative quality. Generally, evaluation and measurement LMS using user experience questionnaire not only provide comprehensive user perception but also provide relative quality level about LMS. The results give the feedback and support when implementing the LMS.
The aim of the current study is to develop an adapted version of User Experience Questionnaire (UEQ) and evaluate a learning management system. Although there is a growing interest on User Experience, there are still limited resources (i.e. measurement tools or questionnaires) available to measure user experience of any products, especially learning management systems. Two hundreds and thirteen computer science students participated and completed the adapted version of UEQ. In the study, the researchers used a learning management system named Student Centered e-Learning Environment (SCELE). Several types of learning materials are posted in SCELE such as audio files, simulations, PowerPoint slides, multimedia contents, and webpage links. Most of the lecturers use discussion forums in their courses to encourage students to participate in active learning setting. Staff and lecturers sometimes post academic-related announcements on the SCELE homepage. Two hundred thirteen students enrolled in Computer Science program were invited to evaluate the SCELE. This study will benefit UX practitioners, HCI educators, program and center of learning resources administrators, and learning management system developers. Findings of the present study may also be valuable for universities and high schools which are using computer-based learning environments.
An end-user questionnaire to measure user experience quickly in a simple and immediate way while covering a preferably comprehensive impression of the product user experience was the goal of the reported construction process. An empirical approach for the item selection was used to ensure practical relevance of items. Usability experts collected terms and statements on user experience and usability, including ‘hard’ as well as ‘soft’ aspects. These statements were consolidated and transformed into a first questionnaire version containing 80 bipolar items. It was used to measure the user experience of software products in several empirical studies. Data were subjected to a factor analysis which resulted in the construction of a 26 item questionnaire including the six factors Attractiveness, Perspicuity, Efficiency, Dependability, Stimulation, and Novelty. Studies conducted for the original German questionnaire and an English version indicate a satisfactory level of reliability and construct validity.
Nowadays, application store become major channel of content and software solution for modern operating system. It enriches the operating system with a wide range choice of the applications through several category and classification. One of the consideration when people purchase or install an application is based on the application rating. Application rating represents the accumulation experience of the users. However, it doesn't provide and additional information in term of user experience. This research makes an effort to improve the value of rating assessment by giving additional variables through user experience questionnaire. As a result, the proposed method can be used for anyone who need to deep evaluate the application by combining the user rating and the user experience.
Usability testing is one of the most used methods to define the level of usability of a software product. However, there is always uncertainty to determine the best method that complements user testing in a depth usability assessment. Nowadays, the concern of many software developers is to identify an appropriate methodology, whose evaluation methods could be capable of measuring all usability aspects of a user interface. For this reason, we conducted a heuristic evaluation as a preliminary step to the implementation of a usability test, in order to determine in which extent these methods complement each other, and establish the gaps which are covered by each of them. The heuristic evaluation was performed by five specialists in the field of Human-Computer Interaction, who identified a total of fifty-nine usability problems in a transactional WEB site. Subsequently, a usability test was conducted with the participation of eight postgraduate students of a master's program in Informatics Engineering. The results show that most of the usability problems which were detected during the usability testing, had already been identified by the heuristic evaluation. Nevertheless, there were significant differences in the importance that was given to each problem. Usability experts emphasized in aspects that were not relevant to end users.
This study investigates and critiques existing Type 2 Diabetes Mellitus (T2DM) websites for usability concerns and provides design recommendations to improve it. A web search for T2DM and diabetes education based on Alexa and Google provided the basis for the study. A discounted usability evaluation technique, heuristic evaluation was applied. The heuristic evaluation resulted in 12 usability issues for American Diabetes Association (ADA) website; 11 usability issues for WebMD; and 15 usability issues for National Diabetes Education Program (NDEP) website. The top five usability problems were presented and discussed in detail. The site that had the best usability was ADA. Findings from this study will be essential for future research to assure that users have a great user experience with T2DM information websites and gain meaningful and relevant information on how to prevent the disease.
Usability specialists were better than non-specialists at performing heuristic evaluation, and “double experts” with specific expertise in the kind of interface being evaluated performed even better. Major usability problems have a higher probability than minor problems of being found in a heuristic evaluation, but more minor problems are found in absolute numbers. Usability heuristics relating to exits and user errors were more difficult to apply than the rest, and additional measures should be taken to find problems relating to these heuristics. Usability problems that relate to missing interface elements that ought to be introduced were more difficult to find by heuristic evaluation in interfaces implemented as paper prototypes but were as easy as other problems to find in running systems.