Content uploaded by Robin Holding Kay
Author content
All content in this area was uploaded by Robin Holding Kay on Nov 21, 2019
Content may be subject to copyright.
ASSESSING THE QUALITY OF ONLINE LEARNING FOR
SECONDARY SCHOOL STUDENTS: THE ONLINE LEARNING
EVALUATION SCALE
R. Kay, J. Li
University of Ontario Institute of Technology (CANADA)
Abstract
Considerable research has been conducted in the domain of K-12 online learning, however, a limited
number of studies have been conducted on the quality learning using this format and best practices [1,
2]. While standards have been created for developing online learning environments [3], no formally
developed, evidence-based, reliable and valid scales for evaluating online learning for K-12 students
were found. The purpose of this study, then, was to develop and evaluate a scale to assess the quality
of K-12 online learning. The Online Learning Evaluation Scale (OLES), based on Garrison’s Community
of Inquiry Framework [4], consisted of three constructs: social presence (n=4 items), cognitive presence
(n=5 items), and teacher presence (n=5 items). The OLES was tested on 315 secondary school students
(females = 237, males = 78, other =3) from 28 courses covering a wide range of subject areas. The
analysis revealed that the OLES demonstrated good internal reliability for each of the three constructs,
convergent validity, construct validity, and predictive validity.
Keywords: online learning, secondary school students, scale, evaluation, assessment measures, social
presence, cognitive presence, teacher presence.
1 INTRODUCTION
Online learning in K-12 environments has grown exponentially in the past decade [1, 2]. Considerable
research has been directed toward comparing student performance between the online and face-to-face
environments, and it is well established that there are no significant differences [5]. The availability of
published research that guides and informs teaching and learning in K-12 online classrooms is
noticeably absent [1, 5, 6]. According to Cavanaugh [7] the quality of K-12 online learning is primarily
assessed through general opinions and/or personal experience. To date, a reliable and valid scale to
assess the quality of online learning for K-12 students as yet to be developed and evaluated.
Garrison’s [4] Community of Inquiry Framework (COI) has been used extensively to assess the quality
of online learning in higher education. The COI model focusses on three distinct areas: teaching
presence, cognitive presence, and social presence. Teaching presence is the purposeful facilitation by
the teacher of online course activities and structure to support meaningful learning objectives [4, 8, 9].
Social presence refers to the extent to which learners progressively identify with the larger online course
community, communicate with purpose, and develop interpersonal relationships [4, 8, 9]. Cognitive
presence represents the degree to which students construct meaning and cognitively engage in an
online course [4, 8, 9]. The purpose of the current study was to develop a metric (Online Learning
Evaluation Scale) for assessing the quality of online learning for secondary school students based on
Garrison’s [4] COI model.
2 METHODOLOGY
2.1 Participants
Upon obtaining consent from their parents or guardians, 318 (237 females, 78 males. 3 other) secondary
students in grades 9 (n=7, 2%), 10 (n=36, 11%), 11 (n=58, 18%), and 12 (n=215, 68%) participated in
this study. Most students agreed (n=103, 32%) or strongly agreed (n=192, 61%) that they were using
computers, however, 77% (n=245) had never participated in an online course prior to the study. Data
was collected from 28 courses focusing on a wide range of subject areas including social studies (n=146,
46%), English (n=57, 18%), business (n=37, 18%), technology (n=24, 8%), science (n=19, 6%), health
(n=13, 4%), and mathematics (n=5, 2%).
Proceedings of ICERI2019 Conference
11th-13th November 2019, Seville, Spain
ISBN: 978-84-09-14755-7
2363
2.2 Context, Procedure and Data Collection
Students enrolled in an online course at the secondary school level were asked to provide feedback on
their experience. They were asked 14, seven-point, Likert-scale questions based on Garrison’s
Community of Inquiry Framework [4] including social presence (n=4 items), cognitive presence (n=5
items) and teacher presence (n=5 items). These three constructs comprised the Online Learning
Evaluation Scale (OLES). Students were also asked six questions about their overall stratification with
the course. The internal reliability for the overall satisfaction score was .91. Course performance was
assessed based on each student’s self-report of their final grade.
2.3 Data Analysis
The following tests were conducted to assess the reliability and validity of the OLES:
- reliability for each of the three OLES constructs (Pearson internal reliability coefficient);
- construct validity (factor analysis of the OLES scale items and correlations among constructs)
- convergent validity (correlations between OLES and overall course satisfaction)
- predictive validity (correlation between the three OLES constructs and course performance)
3 RESULTS
3.1 Overview
The purpose of this study was to develop and assess the reliability and validity of the Online Learning
Evaluation Scale (OLES). The reliability and validity tests are outlined below.
3.2 Internal Reliability
The internal reliability metrics for the OLES constructs based on Cronbach’s alpha were r=.90 for
teaching presence, r=.87 for cognitive presence, and r=.84 for social presence (Table 1). These values
are considered acceptable for constructs developed in the field of social sciences (Kline, 1999; Nunnally,
1978).
Table 1. Descriptions of OLES Scale (n=319)
Construct
No. of Items
Possible
Range
Mean Score
(SD)
Internal Reliability
Teaching Presence
5
7 to 35
24.6 (7.6)
r=.90
Cognitive Presence
5
7 to 35
24.8 (7.0)
r=.87
Social Presence
4
7 to 28
14.1 (5.7)
r=.84
3.3 Construct Validity
3.3.1 Principal Components Analysis
A principal components analysis was conducted to explore whether the OLES constructs (teaching,
cognitive and social presence) were three distinct factors. The Kaiser–Meyer–Olkin measure of
sampling adequacy (.919) and Bartlett’s test of sphericity (p < .001) indicated an acceptable sample
size. The principal components analysis was set to extract three factors (Table 2). The resulting rotation
supported the assumption that the three OELS constructs were distinct.
2364
Table 2. Varimax rotated loading on OLES Scale
Scale Item
Factor 1
Factor 2
Factor 3
TP1 – Received regular feedback
.834
TP2 – Received clear assignment feedback
.814
TP3 – Got help when I needed it
.760
TP4 – Had regular contact with teacher
.711
TP5 – Overall satisfied with quality of assessment
.671
CP1 – Overall, participated regularly in online course
.875
CP2 – Overall, involved in my online course
.847
CP3 – Overall, when taking online course
.723
CP4 – I was able to keep up online course material
.681
CP5 – Overall, my online course was interesting
.475
SP1 – Able to get to know other students
.867
SP2 – Felt a stronger sense of community in online course
.761
SP3 – Able to get help from other students
.753
SP4 – Satisfied with online course community
.660
3.3.2 Correlations Among OLES Constructs
The correlations between the teaching, cognitive and social presence constructs ranged from r=.47 to
r=.60 (Table 3). Consequently, shared variances, ranging from 22% to 36%, indicated that the
constructs were related but distinct.
Table 3. Correlations Among OLES Constructs
Construct
Teaching
Presence
Cognitive
Presence
Social Presence
Teaching Presence
*
0.60
0.47
Cognitive Presence
*
0.48
Social Presence
*
3.4 Convergent Validity
Correlation among overall course satisfaction and the three OLES were used to assess convergent
validity. Overall course satisfaction was significantly and positively correlated with student assessment
of teaching presence (r=.66, n=318, p < .001), cognitive presence (r=.60, n=318, p < .001) and social
presence (r=.51, n=318, p < .001).
3.5 Predictive Validity
Course performance, as measured by the final self-reported grade in the course was significantly and
positively correlated with teaching presence (r=.20, n=318, p < .001), cognitive presence (r=.47, n=318,
p < .001) and social presence (r=.17, n=318, p < .001). A stepwise linear regression revealed that
cognitive and teaching presence (but not social presence) were significant predictors of course
performance.
4 CONCLUSIONS
This study assessed the reliability and validity of a scale created to evaluate the quality of online learning
in K-12 online learning environments. The Online Learning Evaluation Scale (OLES) was based on
Garrison’s [4] Community of Inquiry Framework (COI) was used, focussing on three constructs: teaching
presence, cognitive presence and social presence. Each of the constructs appeared to be internal
2365
reliable with alpha coefficients ranging from .84 and .90. Furthermore, based on the principal
components factor analysis, each construct appeared to measure a unique quality. Finally, while
constructs were significantly correlated with each other, as one might expect in the complex interactions
of an online course, the shared variance was small enough to support the distinctiveness of each COI
constructs.
A reasonably strong case could be made for convergent validity of the OLES. Overall course satisfaction
was significantly correlated with all three OLES constructs. Furthermore, the magnitude of these
correlations was relatively high ranging from .51 to .60. In other words, high scores on teaching,
cognitive and social presence were strongly related to student satisfaction with the overall learning
experience of an online course
Finally, the OLES demonstrated predictive validity. All three constructs were significantly and positively
correlated with course performance as measured by student self-reported grades. A regression analysis
revealed that cognitive and teaching presence were key predictors of final grades. In other words, social
presence at the K-12 level did not appear to predict success. It is worth noting, though, that using actual
student grades would increase the precision of this test. In addition, interviews or focus groups would
be particularly useful in future research studies to determine how teaching, cognitive and social
presence influence the quality of online learning.
Creating and assessing the OLES is the first step to developing a reliable and valid metric for assessing
the quality of K-12 online learning. Future studies need to focus on refining the scale items, exploring
the dynamics of how COI constructs interact with each other and impact learning. and expanding the
target populations to elementary and middle school environments.
REFERENCES
[1] M. K. Barbour, “The Landscape of K-12 Online Learning: Examining What is Known” in Handbook
of Distance Education (4th Ed.) (M. G. Moore & W. C. Dieh), pp. 521-542, New York: Routledge,
2019.
[2] J. Linton, “Electronic Learning Communities as a Support for Building Relationships with Students
in a Statewide Virtual High School,” Journal of Online Learning Research, vol. 2, pp. 419-445, 2016.
[3] International Council for K-12 Online Learning, National Standards for Quality Online Courses.
Vienna, VA, 2011. Retrieved from https://www.inacol.org/wp-content/uploads/2015/02/national-
standards-for-quality-online-courses-v2.pdf
[4] D. R. Garrison, ELearning in the 21st Century. New York: Routledge, 2016.
[5] S. Lowes, “A Brief Look at the Methodologies Used in the Research on Online Teaching and
Learning” in Handbook of Research on K-12 Online and Blended Learning (R. Ferdig & K. Kennedy
eds.), pp. 83–104, Pittsburgh, PA: ETC Press, 2016.
[6] K. Aresen, J. Hveem, C. R. Short, R. E. West & M. K. Barbour, “K-12 Online Learning Journal
Articles: Trends from Two Decades of Scholarship,” Distance Education, vol. 40, no. 1., pp. 32-53,
2018.
[7] C. Cavanaugh, C. Barbour & T. Clark, “Research and Practice in K-12 Online Learning: A review of
literature,” International Review of Research in Open and Distance Learning, vol. 10, no. 1,
Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/607
[8] D. R. Garrison, E-Learning in The 21st Century: A Framework for Research and Practice. New York,
NY: Routledge, 2011
[9] D. R. Garrison, T. Anderson & W. Archer, “The first decade of the community of inquiry framework:
A retrospective,” The Internet and Higher Education, vol 13, no. 1, pp. 5-9, 2010.
[10] P. Kline, The Handbook of Psychological Testing (2nd ed.): London: Routledge, 1999
[11] J. C. Nunally, Psychometric Theory. New York: McGraw-Hill, 1978
2366