ArticlePDF Available

Usefulness of ubiquitous-based testing for evaluations in medical education

Authors:
Oh Young Kwon
Oh Young Kwon
Oh Young Kwon
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Sang Youl Rhee at Kyung Hee University
Sang Youl Rhee
Joong-Myung Choi at Kyung Hee University
Joong-Myung Choi
Young Seol Kim
Young Seol Kim
Young Seol Kim
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Download full-text PDFDownload full-text PDF
Read full-text
Download citation

Abstract and Figures

The purpose of this study was to evaluate the suitability (convenience, objectiveness, and satisfaction) of ubiquitous-based testing (UBT) as a medical education evaluation tool. UBT was administered using a smart pad in our medical school in May 2012. A questionnaire was given twice. The pre-UBT questionnaire examined possession of a tablet computer, skillfulness of smart devices, the convenience of UBT, and the usefulness of a medical educational assessment tool. The post-UBT questionnaire evaluated the satisfaction, convenience, and preference of UBT and the usefulness of a medical educational assessment tool, as in the pre-UBT test. The survey was measured on a 4-point scale: 1 is "strongly disagree" and 4 is "strongly agree." One hundred three students (male, 55.3%) participated in the UBT. The mean age was 29.2±2.4 years. In the pre-UBT questionnaire analysis, students responded affirmatively to the items about the skillfulness of smart devices, clinical skill assessment, and achievement of educational objectives. The responses to the items on the convenience and satisfaction with the UBT were positive in the post-UBT. The factors that affected the post-UBT questionnaire were as follows: knowledge assessment (p=0.041) and achievement of educational objectives (p=0.015) were significant, based on gender, and satisfaction with the UBT (p=0.002) was significant, based on possession of a tablet computer. The relationship between the ranks of this UBT and the average ranks of the three previous semesters was statistically significant (p<0.001). Convenience, objectiveness, knowledge assessment, and composition and completion were useful items in the UBT.
Figures - uploaded by Joong-Myung Choi
Author content
All figure content in this area was uploaded by Joong-Myung Choi
Content may be subject to copyright.
562ec37a08aef25a244452
49.pdf
Content uploaded by Joong-Myung Choi
Author content
All content in this area was uploaded by Joong-Myung Choi on Oct 27, 2015
Content may be subject to copyright.
Usefulness of ubiquitous-based testing for evaluations in medical educati
on.pdf
Available via license: CC BY-NC 3.0
Content may be subject to copyright.
A preview of the PDF is not available
... A lot of research is being carried out on this. In Korea, studies were conducted on tests using a tablet computer for medical [5,[16][17][18] and emergency rescue students [19]. These studies showed high satisfaction among the examinees due to the convenience in the administration of tests using a tablet computer. ...
... Computer-Based Testing (CBT): Computer-based testing (CBT) involves reading questions, responding to them on individual computers, and submitting them [17]. • Ubiquitous-Based Testing (UBT): In ubiquitous-based testing (UBT), testing, scoring, and grade management are performed using tablets and smartphones [17]. ...
... Computer-Based Testing (CBT): Computer-based testing (CBT) involves reading questions, responding to them on individual computers, and submitting them [17]. • Ubiquitous-Based Testing (UBT): In ubiquitous-based testing (UBT), testing, scoring, and grade management are performed using tablets and smartphones [17]. ...
Evaluation of Student Satisfaction with Ubiquitous-Based Tests in Women’s Health Nursing Course
Article
Full-text available
  • Nov 2021
Learning evaluation using ubiquitous-based tests may be essential during a public health crisis, such as the COVID-19 pandemic, during which theoretical classes and clinical practice are conducted online. However, students may not be as familiar with ubiquitous-based tests as they are with paper-based tests. This survey study aimed to evaluate students’ satisfaction with ubiquitous-based tests and compare the evaluation results of a paper-based test with that of a ubiquitous-based test in nursing education. For the midterm exam of the Women’s Health Nursing course, a paper-based test was conducted, while a ubiquitous-based test using a tablet computer was used for the final exam. The Ubiquitous-Based Test Usefulness and Satisfaction tool, which has a five-point Likert-type response scale, was employed to evaluate the post-test usefulness and satisfaction scores of the ubiquitous-based test. The mean score of the ubiquitous-based test usefulness was 4.01 ± 0.67. There was a significant difference in satisfaction levels between the ubiquitous-based and the paper-based test (t = −3.36, p = 0.001). Specifically, the evaluation scores were not affected by different evaluation methods. Study participants deemed the ubiquitous-based test highly useful and satisfactory, suggesting that such tests may be a future-oriented evaluation method, potentially replacing paper-based tests.
View
... In order for SBT to be recognized as a useful and efficient assessment tool in the field of medical education and to establish its status, various kinds of studies are required, and the actual characteristics of SBT must be identified. Studies on SBT as an assessment tool are gradually increasing and domestic and international studies have been published on smart device ownership, proficiency, preference, and so forth, or its relation to existing pencil-paper tests [10][11][12][13]. However, previous studies in Korea limited either the length of use or the number of people involved. ...
... In the previous study, it was found that there was no significant relationship between SBT exam scores and ownership of smart devices, and that SBT exam scores were neither affected by gender nor smart device proficiency. In addition, the association between accumulative rank in the pencil-paper test and SBT scores was found to be significant [11], and a study by Kim and Hwang [12], proved that the difficulty of pencil-paper test and the multimedia test using tablets was not significantly different [12]. Therefore, it is believed that SBT will not have any special effect on the results even if it replaces the existing pencil-paper test. ...
Evaluation of usefulness of smart device-based testing: a survey study of Korean medical students
Article
Full-text available
  • Sep 2020
Purpose: This study aims to understand the characteristics of smart device-based testing (SBT) by comparing the typical characteristics of students' satisfaction with SBT, its usefulness, advantages, and disadvantages when compared with existing testing methods. Methods: A total of 250 students from the first to third year were selected as the final targets of the study and the questionnaire was developed by faculty members who participated in the survey from the start of the SBT. The total number of questions is 12, and the questionnaire used a 4-point scale. The data obtained were analyzed using the IBM SPSS ver. 23.0 (IBM Corp., Armonk, USA). Results: Answers to the "satisfaction with SBT" were generally negative, while answers to the "usefulness of SBT" were generally positive. There was no difference in satisfaction across gender and smart device ownership, whereas there were significant differences across years. With reference to the usefulness of SBT, students responded positively, while about the overall configuration and completeness of SBT, students responded negatively. Students also seemed to show a greater preference toward the pencil-paper test. Conclusion: On the other hand, students generally thought that SBT helped to assess medical knowledge better and was a more objective method of knowledge assessment than a pencil-paper test. We believe that students preferred the traditional paper-pencil test due to their unfamiliarity with SBT. We believe that an appropriate and careful remedy for drawbacks of the SBT will have a significant impact in the accumulation of actual clinical knowledge and in the improvement of practical skills for medical students.
View
... SBT has also been administered in several other medical schools in Korea, where it has been known as ubiquitous-based testing (UBT). UBT was established to be a satisfactory and convenient testing tool in one of these medical schools [2]. In another medical school, students had a positive attitude towards the UBT interface. ...
Smart device-based testing for medical students in Korea: satisfaction, convenience, and advantages
Article
Full-text available
  • Apr 2017
The aim of this study was to investigate respondents’ satisfaction with smart device-based testing (SBT), as well as its convenience and advantages, in order to improve its implementation. The survey was conducted among 108 junior medical students at Kyungpook National University School of Medicine, Korea, who took a practice licensing examination using SBT in September 2015. The survey contained 28 items scored using a 5-point Likert scale. The items were divided into the following three categories: satisfaction with SBT administration, convenience of SBT features, and advantages of SBT compared to paper-and-pencil testing or computer-based testing. The reliability of the survey was 0.95. Of the three categories, the convenience of the SBT features received the highest mean (M) score (M= 3.75, standard deviation [SD]= 0.69), while the category of satisfaction with SBT received the lowest (M= 3.13, SD= 1.07). No statistically significant differences across these categories with respect to sex, age, or experience were observed. These results indicate that SBT was practical and effective to take and to administer.
View
View
Analysis of the Satisfaction with Computer Based Test Program and Test Environment in Medical School
Article
Full-text available
  • Oct 2020
This study aimed to identify needed improvements to current evaluation methods in medical school computer-based test (CBT) programs and test environments. To that end, an analysis of the importance and satisfaction was conducted through a survey of 3rd and 4th year medical students who had sufficient experience with CBT programs. Importance performance analysis methodology using the correlation coefficient was applied to assess average satisfaction and importance. The first quadrant (keep up the good work) was a factor of review and time management and test facilities among the conveniences of the CBT program. The second quadrant (concentrate here) was a factor of the convenience of the CBT program and computer monitor and chair factor within the test facilities. The third quadrant (low priority) was a factor of cheating and computer failure. The fourth quadrant (possible overkill) was the location, spacing, and temperature factors of the test facilities. Improvements are needed to reduce ‘eye fatigue’ and help students focus and understand the questions in the CBT programs. It is necessary to improve computer monitors, desks and chairs, and consider the subject’s body type and manager in order to cope with computer breakdown and peripheral failures. Spare computers are needed. These findings are meaningful in that they have been able to identify factors that require improvement in the CBT program and test environment resulting from changes in assessment tools.
View
Evaluation research in Korean medical education: a systematic review
Article
Full-text available
  • Dec 2022
The purpose of this study aims to analyze research trends related to 'evaluation' in Korean medical education through a systematic review. This study used a systematic review method, which is a research methodology for research trends and 'literature analysis.' Researchers searched the Korean journal literature published until the end of December 2020 in the Korean research database with keywords related to medicine and evaluation. Thus, 5,205 cases were identified. Based on these data, 143 papers were selected through a logical screening process, requiring 1 month to complete the data search and analysis process. In terms of publications, medical journals overwhelmingly outnumbered nonmedical journals until 2015; however, after 2016, the number of papers published in nonmedical journals increased, and the number of published papers was similar to that of medical journals. In terms of evaluation-related research, research on student and program evaluations has been very active compared to that on accreditation. As the number of evaluation studies has gradually decreased over the past 10 years, preparing a plan to revitalize them in Korean medical education is necessary. Considering that the role of evaluation in education has been emphasized in recent years, research on reestablishing the concept of evaluation; developing evaluation indicators; analyzing the status of student evaluation, program evaluation, and accreditation; and deriving measures to improve medical education through evaluation is required.
View
Development and application of a mobile-based multimedia nursing competency evaluation system for nursing students: A mixed-method randomized controlled study
Article
  • Sep 2022
  • Nurse Educ Pract
Aim To develop a mobile-based multimedia Nursing Competency Evaluation (NCE) system based on the Attention, Relevance, Confidence, Satisfaction model and verify its effectiveness. Background In education, mobile devices can enable the delivery of learning content without time and spatial constraints. Mobile-based test is emerging as a novel method using technologies to appraise students’ performance on practicum. This mobile-based test go beyond the simple evaluation of memorised knowledge, a limitation common to paper-based tests. They are useful because they can include multimedia items such as videos, animations and pictures to comprehensively evaluate students’ clinical competencies. Methods This study was conducted in a nursing university in Seoul, South Korea in September 2021. A mixed method randomised controlled design was employed to evaluate its usability. The participants in the experimental group used the Nursing Competency Evaluation system and joined in focus group interviews for verifying the effects of the Nursing Competency Evaluation system qualitatively. Those in the control group responded to the mobile-based test, but which has only text-based test items. The system usability, effectiveness and learning satisfaction in both groups were measured after the mobile-based test experience. Quantitative and qualitative data were analysed using t-tests and thematic analysis using focus group interviews, respectively. Results Sixty nursing students participated, with 30 each in the experimental and control groups. There were no significant differences in nursing competency scores between the two groups. However, average scores for effectiveness and learning satisfaction were significantly higher in the experimental than in the control group. Nineteen experimental group participants partook in the interviews, with many describing that the Nursing Competency Evaluation system allowed them to experience new learning contents and efficiently learn practical nursing skills that can be useful in clinical settings. Conclusions The Nursing Competency Evaluation system is a promising method because it used mobile technologies and multimedia to appraise students’ performance on nursing practicum. It was found that the Nursing Competency Evaluation system with multimedia items is more realistic, interactive and satisfactory compared with text-based mobile test. Thus, we expect it to be used in future nursing curriculums to improve students’ nursing competencies.
View
Development, Application, and Effectiveness of a Smart Device–based Nursing Competency Evaluation Test: A Mixed-Method Study
Article
  • Apr 2021
  • CIN-COMPUT INFORM NU
We aimed to develop and evaluate the effectiveness of a smart device-based test to assess Korean undergraduate students' clinical nursing competency, named SBT-NURS. The 65-item SBT-NURS comprises questions that simulate clinical situations, are problem solving-oriented, use multimedia (ie, videos/photos/animations), and involve the following topics: medical-surgical nursing, fundamentals of nursing, pediatrics, maternity, management, and psychiatric. We utilized a quantitative method to analyze the effects of the SBT-NURS (ie, via a single-group, post-experimental survey design) and a qualitative method to analyze students' experiences of using the SBT-NURS (ie, via seven focus group interviews [FGIs]). Students' overall adult health nursing paper-based test scores (ie, combining their scores in group activity, presentation, attendance, and attitude toward the midterm and final tests on adult health nursing) (r = 0.552, P < .001) and clinical practicum scores (r = 0.268, P = .040) in the last semester showed a statistically significant positive correlation with their SBT-NURS scores. Their paper-based testing practice average scores (ie, combination between paper-based tests and clinical practicum scores) showed a similar significant correlation (r = 0.506, P < .001). Students deemed the SBT-NURS advantageous, satisfactory, convenient, and useful. The SBT-NURS may be an effective learning and evaluation method for nursing education that help improve students' clinical competency and learning outcomes.
View
View
Evaluation in medical education
Article
Full-text available
  • Mar 2015
View
View
View
Human–computer interaction issues for mobile computing in a variable work context
Article
  • May 2004
The current paper takes an introspective look at the human–computer interaction (HCI) issues for mobile computing in a variable work context. We catalogue the current research in four major categories. The major findings of our study are following. (1) A majority of HCI issues, about 58%, fall under the category of computer systems and interface architecture implications. (2) 23% of the articles focus on development and implementation issues. (3) 13% of the articles focus on use and context of computer issues. (4) 6% of the articles focus on human characteristics issues. Further, the literature indicates that the field services is a main application of mobile computing (46%) followed by sales force (21%), health care (17%), fieldwork (8%), insurance claims (4%) and journalism (4%).
View
Hot Topics-Ubiquitous Computing
Article
  • Nov 1993
The author suggests that, due to the trends of unobtrusive technology and more intrusive information, the next phase of computing technology will develop nonlinearly. He states that, in the long run, the personal computer and the workstation will become practically obsolete because computing access will be everywhere: in the walls, on your wrist, and in `scrap' computers (i.e., like scrap paper) lying about to be used as needed. The current research on ubiquitous computing is reviewed
View
Computerized adaptive test
  • Jan 1998
  • Sg Baek
  • Sh Chae
Baek SG, Chae SH. Computerized adaptive test. Seoul, Korea: Wonmisa; 1998.
Beyond the comfort zone
  • Sep 2004
  • 468-474
  • R Atkinson
  • C Mcbeath
  • D Phillips
In: Atkinson R, McBeath C, Jonas-Dwyer D, Phillips R, editors. Beyond the comfort zone. Proceedings of the 21st Australasian Society for Computers in Learning in Tertiary Education Conference; 2004 December 5-8; Perth, USA; 2004. p 468-474.
Medical Council of Canada
  • Oct 2014
  • Medical Council
  • Canada
Medical Council of Canada [Internet]. Medical Council of Canada; c2014 [cited 2014 September 16]. Available from: http://www.mcc.ca.
Ubiquitous computing
  • Jan 1993
  • 71-72
  • M Weiser
Weiser M. Ubiquitous computing. Computer 1993; 26: Oh Young Kwon, et al : Ubiquitous-based testing in medical education 10 Korean J Med Educ 2015 Mar; 27(1): 3-10. 71-72.
Survey on the mobile internet usage. Seoul, Korea: Korea Internet & Security Agency; 2013. 10. Smith A. Smartphone adoption and usage
  • Jan 2011
  • Kj Lee
Lee KJ. Survey on the mobile internet usage. Seoul, Korea: Korea Internet & Security Agency; 2013. 10. Smith A. Smartphone adoption and usage. Washington, DC, USA: Pew Research Center; 2011.
Computerized adaptive test. Seoul, Korea: Wonmisa
  • Jan 1998
  • S G Baek
  • S H Chae
Baek SG, Chae SH. Computerized adaptive test. Seoul, Korea: Wonmisa; 1998.