Conference PaperPDF Available

Assessing the Quality of Online Learning for Secondary School Students: The Online Learning Evaluation Scale

Authors:

Abstract and Figures

Considerable research has been conducted in the domain of K-12 online learning, however, a limited number of studies have been conducted on the quality learning using this format and best practices [1, 2]. While standards have been created for developing online learning environments [3], no formally developed, evidence-based, reliable and valid scales for evaluating online learning for K-12 students were found. The purpose of this study, then, was to develop and evaluate a scale to assess the quality of K-12 online learning. The Online Learning Evaluation Scale (OLES), based on Garrison's Community of Inquiry Framework [4], consisted of three constructs: social presence (n=4 items), cognitive presence (n=5 items), and teacher presence (n=5 items). The OLES was tested on 315 secondary school students (females = 237, males = 78, other =3) from 28 courses covering a wide range of subject areas. The analysis revealed that the OLES demonstrated good internal reliability for each of the three constructs, convergent validity, construct validity, and predictive validity.
Content may be subject to copyright.
ASSESSING THE QUALITY OF ONLINE LEARNING FOR
SECONDARY SCHOOL STUDENTS: THE ONLINE LEARNING
EVALUATION SCALE
R. Kay, J. Li
University of Ontario Institute of Technology (CANADA)
Abstract
Considerable research has been conducted in the domain of K-12 online learning, however, a limited
number of studies have been conducted on the quality learning using this format and best practices [1,
2]. While standards have been created for developing online learning environments [3], no formally
developed, evidence-based, reliable and valid scales for evaluating online learning for K-12 students
were found. The purpose of this study, then, was to develop and evaluate a scale to assess the quality
of K-12 online learning. The Online Learning Evaluation Scale (OLES), based on Garrison’s Community
of Inquiry Framework [4], consisted of three constructs: social presence (n=4 items), cognitive presence
(n=5 items), and teacher presence (n=5 items). The OLES was tested on 315 secondary school students
(females = 237, males = 78, other =3) from 28 courses covering a wide range of subject areas. The
analysis revealed that the OLES demonstrated good internal reliability for each of the three constructs,
convergent validity, construct validity, and predictive validity.
Keywords: online learning, secondary school students, scale, evaluation, assessment measures, social
presence, cognitive presence, teacher presence.
1 INTRODUCTION
Online learning in K-12 environments has grown exponentially in the past decade [1, 2]. Considerable
research has been directed toward comparing student performance between the online and face-to-face
environments, and it is well established that there are no significant differences [5]. The availability of
published research that guides and informs teaching and learning in K-12 online classrooms is
noticeably absent [1, 5, 6]. According to Cavanaugh [7] the quality of K-12 online learning is primarily
assessed through general opinions and/or personal experience. To date, a reliable and valid scale to
assess the quality of online learning for K-12 students as yet to be developed and evaluated.
Garrison’s [4] Community of Inquiry Framework (COI) has been used extensively to assess the quality
of online learning in higher education. The COI model focusses on three distinct areas: teaching
presence, cognitive presence, and social presence. Teaching presence is the purposeful facilitation by
the teacher of online course activities and structure to support meaningful learning objectives [4, 8, 9].
Social presence refers to the extent to which learners progressively identify with the larger online course
community, communicate with purpose, and develop interpersonal relationships [4, 8, 9]. Cognitive
presence represents the degree to which students construct meaning and cognitively engage in an
online course [4, 8, 9]. The purpose of the current study was to develop a metric (Online Learning
Evaluation Scale) for assessing the quality of online learning for secondary school students based on
Garrison’s [4] COI model.
2 METHODOLOGY
2.1 Participants
Upon obtaining consent from their parents or guardians, 318 (237 females, 78 males. 3 other) secondary
students in grades 9 (n=7, 2%), 10 (n=36, 11%), 11 (n=58, 18%), and 12 (n=215, 68%) participated in
this study. Most students agreed (n=103, 32%) or strongly agreed (n=192, 61%) that they were using
computers, however, 77% (n=245) had never participated in an online course prior to the study. Data
was collected from 28 courses focusing on a wide range of subject areas including social studies (n=146,
46%), English (n=57, 18%), business (n=37, 18%), technology (n=24, 8%), science (n=19, 6%), health
(n=13, 4%), and mathematics (n=5, 2%).
Proceedings of ICERI2019 Conference
11th-13th November 2019, Seville, Spain
ISBN: 978-84-09-14755-7
2363
2.2 Context, Procedure and Data Collection
Students enrolled in an online course at the secondary school level were asked to provide feedback on
their experience. They were asked 14, seven-point, Likert-scale questions based on Garrison’s
Community of Inquiry Framework [4] including social presence (n=4 items), cognitive presence (n=5
items) and teacher presence (n=5 items). These three constructs comprised the Online Learning
Evaluation Scale (OLES). Students were also asked six questions about their overall stratification with
the course. The internal reliability for the overall satisfaction score was .91. Course performance was
assessed based on each student’s self-report of their final grade.
2.3 Data Analysis
The following tests were conducted to assess the reliability and validity of the OLES:
- reliability for each of the three OLES constructs (Pearson internal reliability coefficient);
- construct validity (factor analysis of the OLES scale items and correlations among constructs)
- convergent validity (correlations between OLES and overall course satisfaction)
- predictive validity (correlation between the three OLES constructs and course performance)
3 RESULTS
3.1 Overview
The purpose of this study was to develop and assess the reliability and validity of the Online Learning
Evaluation Scale (OLES). The reliability and validity tests are outlined below.
3.2 Internal Reliability
The internal reliability metrics for the OLES constructs based on Cronbach’s alpha were r=.90 for
teaching presence, r=.87 for cognitive presence, and r=.84 for social presence (Table 1). These values
are considered acceptable for constructs developed in the field of social sciences (Kline, 1999; Nunnally,
1978).
Table 1. Descriptions of OLES Scale (n=319)
Construct
No. of Items
Possible
Range
Mean Score
(SD)
Internal Reliability
Teaching Presence
5
7 to 35
24.6 (7.6)
r=.90
Cognitive Presence
5
7 to 35
24.8 (7.0)
r=.87
Social Presence
4
7 to 28
14.1 (5.7)
r=.84
3.3 Construct Validity
3.3.1 Principal Components Analysis
A principal components analysis was conducted to explore whether the OLES constructs (teaching,
cognitive and social presence) were three distinct factors. The KaiserMeyerOlkin measure of
sampling adequacy (.919) and Bartlett’s test of sphericity (p < .001) indicated an acceptable sample
size. The principal components analysis was set to extract three factors (Table 2). The resulting rotation
supported the assumption that the three OELS constructs were distinct.
2364
Table 2. Varimax rotated loading on OLES Scale
Scale Item
Factor 1
Factor 2
Factor 3
TP1Received regular feedback
.834
TP2Received clear assignment feedback
.814
TP3Got help when I needed it
.760
TP4Had regular contact with teacher
.711
TP5 Overall satisfied with quality of assessment
.671
CP1Overall, participated regularly in online course
.875
CP2Overall, involved in my online course
.847
CP3Overall, when taking online course
.723
CP4I was able to keep up online course material
.681
CP5Overall, my online course was interesting
.475
SP1Able to get to know other students
.867
SP2Felt a stronger sense of community in online course
.761
SP3 Able to get help from other students
.753
SP4Satisfied with online course community
.660
3.3.2 Correlations Among OLES Constructs
The correlations between the teaching, cognitive and social presence constructs ranged from r=.47 to
r=.60 (Table 3). Consequently, shared variances, ranging from 22% to 36%, indicated that the
constructs were related but distinct.
Table 3. Correlations Among OLES Constructs
Construct
Teaching
Presence
Cognitive
Presence
Social Presence
Teaching Presence
*
0.60
0.47
Cognitive Presence
*
0.48
Social Presence
*
3.4 Convergent Validity
Correlation among overall course satisfaction and the three OLES were used to assess convergent
validity. Overall course satisfaction was significantly and positively correlated with student assessment
of teaching presence (r=.66, n=318, p < .001), cognitive presence (r=.60, n=318, p < .001) and social
presence (r=.51, n=318, p < .001).
3.5 Predictive Validity
Course performance, as measured by the final self-reported grade in the course was significantly and
positively correlated with teaching presence (r=.20, n=318, p < .001), cognitive presence (r=.47, n=318,
p < .001) and social presence (r=.17, n=318, p < .001). A stepwise linear regression revealed that
cognitive and teaching presence (but not social presence) were significant predictors of course
performance.
4 CONCLUSIONS
This study assessed the reliability and validity of a scale created to evaluate the quality of online learning
in K-12 online learning environments. The Online Learning Evaluation Scale (OLES) was based on
Garrison’s [4] Community of Inquiry Framework (COI) was used, focussing on three constructs: teaching
presence, cognitive presence and social presence. Each of the constructs appeared to be internal
2365
reliable with alpha coefficients ranging from .84 and .90. Furthermore, based on the principal
components factor analysis, each construct appeared to measure a unique quality. Finally, while
constructs were significantly correlated with each other, as one might expect in the complex interactions
of an online course, the shared variance was small enough to support the distinctiveness of each COI
constructs.
A reasonably strong case could be made for convergent validity of the OLES. Overall course satisfaction
was significantly correlated with all three OLES constructs. Furthermore, the magnitude of these
correlations was relatively high ranging from .51 to .60. In other words, high scores on teaching,
cognitive and social presence were strongly related to student satisfaction with the overall learning
experience of an online course
Finally, the OLES demonstrated predictive validity. All three constructs were significantly and positively
correlated with course performance as measured by student self-reported grades. A regression analysis
revealed that cognitive and teaching presence were key predictors of final grades. In other words, social
presence at the K-12 level did not appear to predict success. It is worth noting, though, that using actual
student grades would increase the precision of this test. In addition, interviews or focus groups would
be particularly useful in future research studies to determine how teaching, cognitive and social
presence influence the quality of online learning.
Creating and assessing the OLES is the first step to developing a reliable and valid metric for assessing
the quality of K-12 online learning. Future studies need to focus on refining the scale items, exploring
the dynamics of how COI constructs interact with each other and impact learning. and expanding the
target populations to elementary and middle school environments.
REFERENCES
[1] M. K. Barbour, The Landscape of K-12 Online Learning: Examining What is Known” in Handbook
of Distance Education (4th Ed.) (M. G. Moore & W. C. Dieh), pp. 521-542, New York: Routledge,
2019.
[2] J. Linton, “Electronic Learning Communities as a Support for Building Relationships with Students
in a Statewide Virtual High School,” Journal of Online Learning Research, vol. 2, pp. 419-445, 2016.
[3] International Council for K-12 Online Learning, National Standards for Quality Online Courses.
Vienna, VA, 2011. Retrieved from https://www.inacol.org/wp-content/uploads/2015/02/national-
standards-for-quality-online-courses-v2.pdf
[4] D. R. Garrison, ELearning in the 21st Century. New York: Routledge, 2016.
[5] S. Lowes, “A Brief Look at the Methodologies Used in the Research on Online Teaching and
Learning” in Handbook of Research on K-12 Online and Blended Learning (R. Ferdig & K. Kennedy
eds.), pp. 83104, Pittsburgh, PA: ETC Press, 2016.
[6] K. Aresen, J. Hveem, C. R. Short, R. E. West & M. K. Barbour, K-12 Online Learning Journal
Articles: Trends from Two Decades of Scholarship,” Distance Education, vol. 40, no. 1., pp. 32-53,
2018.
[7] C. Cavanaugh, C. Barbour & T. Clark, “Research and Practice in K-12 Online Learning: A review of
literature,” International Review of Research in Open and Distance Learning, vol. 10, no. 1,
Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/607
[8] D. R. Garrison, E-Learning in The 21st Century: A Framework for Research and Practice. New York,
NY: Routledge, 2011
[9] D. R. Garrison, T. Anderson & W. Archer, “The first decade of the community of inquiry framework:
A retrospective,” The Internet and Higher Education, vol 13, no. 1, pp. 5-9, 2010.
[10] P. Kline, The Handbook of Psychological Testing (2nd ed.): London: Routledge, 1999
[11] J. C. Nunally, Psychometric Theory. New York: McGraw-Hill, 1978
2366
ResearchGate has not been able to resolve any citations for this publication.
Chapter
Full-text available
While the use of distance and online learning at the K-12 level of growing exponentially, the availability of empirical evidence to help guide this growth is severely lacking. The author of this chapter provides an overview of the nature of K-12 distance and online learning today and a critical examination of the literature and-lack of research-supporting its use. The author further describes some of the methodological issues surround the limited among of existing research, and calls for researchers in this field to adopt practices that have been employed by their colleagues investigating adult populations over the past decades. Biography: Michael Barbour is an Associate Professor of Instructional Design for the College of Education and Health Sciences at Touro University California in Vallejo, California. He has been involved with K-12 online learning in a variety of countries for well over a decade as a researcher, teacher, course designer and administrator. Michael's research interests focus on the effective design, delivery, and support of online learning to K-12 students in virtual school environments, particularly those in rural jurisdictions.
Article
Full-text available
In this study, we examined the research literature in the field of K-12 online learning to identify the leading scholars, journals, top cited articles, research methods, and topics in this field of inquiry. Our research process involved collecting a corpus of journal articles focused on K-12 online and distance learning; categorizing these articles according to their research methodologies; analyzing trends not only in methodologies employed but also in authorship, citations, journals, and topics addressed. We found the field of K-12 online learning to be growing rapidly in recent years with acceleration not only of new articles but especially of new authors. We also found the field began primarily with emphasizing theoretical articles but is now maturing and emphasizing increasingly more data-based articles. We found K-12 online learning scholarship is scattered among many journals, providing rich opportunities for scholars while also making it more difficult to discern trends across the discipline. Available online at https://www.tandfonline.com/doi/full/10.1080/01587919.2018.1553566
Article
Full-text available
This qualitative case study used Wenger's (1998) communities of practice (CoP) framework to analyze how the ongoing electronic learning community (eLC) process at an established state virtual high school (SVHS) supported online teachers in building relationships with online students. Lave and Wenger's (1991) concept of legitimate peripheral participation (LPP), which describes the participation of new CoP members as they move toward full membership, was used to examine the participation and perspective of new eLC members at SVHS. Elements of LPP were evident in case study data, particularly in the way the eLC process granted new members access to resources and to the practice of other members. Other elements of LPP were less visible in the eLC process, such as becoming and conferring legitimacy. Findings from this study suggest that online instructors should be given opportunities to build community and develop relationships with one another through repeated, ongoing collaboration. To overcome barriers due to separations in distance and time, community-building must be an intentional component of the eLC process. The eLC process provided teachers with opportunities to improve teaching in order to increase support for student learning. 420
Article
Full-text available
The second edition of E-Learning in the 21st Century provides a coherent, comprehensive, and empirically-based framework for understanding e-learning in higher education. Garrison draws on his decades of experience and extensive research in the field to explore the technological, pedagogical, and organizational implications of e-learning. Most importantly, he provides practical models that educators can use to realize the full potential of e-learning. This book is unique in that it focuses less on the long list of ever-evolving technologies and more on the search for an understanding of these technologies from an educational perspective.
Article
Full-text available
This article provides a personal perspective about the development of the seminal papers associated with the Community of Inquiry Framework. The framework and its construction are described. The main part of the paper explores the evolution of the framework and its associated methodology. Finally, research associated with the validation of the framework and new research directions are reviewed.
Article
Full-text available
The literature related to online learning programs for K-12 students dates to the mid-1990s and builds upon a century of research and practice from K-12 distance education. While K-12 online learning programs have evolved and grown over the past decade, the amount of published research on virtual schooling practice and policy is limited. The current literature includes practitioner reports and experimental and quasi-experimental studies, both published and unpublished. This paper reviews open access literature in K-12 online learning and reports on a structured content analysis of the documents. Themes in the literature include steady growth and a focus on the benefits, challenges, and broad effectiveness of K-12 online learning. In addition, newly developed standards for K-12 online learning are emerging in descriptions of effective practices.
ELearning in the 21st Century
  • D R Garrison
D. R. Garrison, ELearning in the 21st Century. New York: Routledge, 2016.
A Brief Look at the Methodologies Used in the Research on Online Teaching and Learning
  • S Lowes
S. Lowes, "A Brief Look at the Methodologies Used in the Research on Online Teaching and Learning" in Handbook of Research on K-12 Online and Blended Learning (R. Ferdig & K. Kennedy eds.), pp. 83-104, Pittsburgh, PA: ETC Press, 2016.