Content uploaded by Renée S. Jansen
Author content
All content in this area was uploaded by Renée S. Jansen on Oct 02, 2018
Content may be subject to copyright.
Validation of the revised Self-regulated Online Learning
Questionnaire
Renée S. Jansen, Anouschka van Leeuwen, Jeroen Janssen, and Liesbeth Kester
Utrecht University, Heidelberglaan 1, 3584 CS, Utrecht, The Netherlands
Abstract. Self-regulated learning (SRL) is essential for students in online edu-
cation to be successful. The Self-Regulated Online Learning Questionnaire was
developed to measure SRL in online educational contexts. In this paper, a re-
vised version of the questionnaire is presented and tested with three datasets.
The scale ‘metacognitive skills’ is split into three subscales: metacognitive ac-
tivities before, during, and after a learning task. Next to the three scales measur-
ing metacognitive activity, the questionnaire contains scales measuring time
management, environmental structuring, persistence, and help seeking. The re-
vised questionnaire was found to have improved validity, usability, and reliabil-
ity.
Keywords: Questionnaire, online education, blended learning, MOOCs.
1 Introduction
In online and blended learning, learners have more autonomy than in face-to-face
education [1, 2]. This increase in autonomy makes it essential for learners to be ac-
tively involved in their own learning process, meaning that they self-regulate their
learning [3, 4]. To accurately measure learners’ self-regulated learning in online edu-
cation, the Self-regulated Online Learning Questionnaire (SOL-Q; [5]) was devel-
oped. While this questionnaire is a useful instrument, its validity, reliability, and usa-
bility could be improved. In the current paper, a revised version of the questionnaire
is presented tested with three datasets.
1.1 Self-regulated learning
Self-regulated learners are actively involved in their own learning process, not only
during learning (performance phase), but also before (preparatory phase), and after
learning (appraisal phase) [6, 7]. In the preparatory phase, learners think about what
and how they will learn and the goals they have for the current learning session; they
engage in (strategic) planning and goal setting. In the performance phase, learners
engage in comprehension monitoring and strategy regulation. They furthermore man-
age their ‘resources’, including their time and study environment, as well as find help
when needed and persist when motivation drops. During the appraisal phase, learners
reflect on their learning progress and their learning strategies [6].
2
1.2 Self-regulated Online Learning Questionnaire (SOL-Q)
To improve students’ SRL in online education, it is important that students’ SRL can
be measured. The SOL-Q [5] was a first attempt at developing a questionnaire suita-
ble to measure students’ SRL in online learning environments. The developed ques-
tionnaire was based on several existing well-established SRL questionnaires (such as
the Motivated Strategies for Learning Questionnaire; [8]): items from these question-
naires were selected and adapted to fit the context of online education. Based on ex-
ploratory and confirmatory factor analysis, an initial version of the SOL-Q was pub-
lished. The SOL-Q consists of five scales: metacognitive skills (18 items, α = .90,
time management (3 items, α = .71), environmental structuring (5 items, α = .67),
persistence (5 items, α = .79), and help seeking (5 items, α = .83).
1.3 Further development of the SOL-Q
Although a satisfactory, initial version of the SOL-Q was created, the scale ‘metacog-
nitive skills’ proved to be large and diverse. It consisted of items from a range of
metacognitive self-regulation activities (e.g., goal setting, comprehension monitoring,
reflection) and covering all SRL phases (preparatory, performance, and appraisal
phase). The clustering of metacognitive items into a single metacognitive scale is not
unexpected. In the SRL model presented by Zimmerman [9], significant correlations
between the variables within a SRL phase are described, and Sitzmann and Ely [10]
indeed found strong correlations between SRL constructs. While learners may not be
able to distinguish among all the metacognitive activities, learners may be able to
distinguish among the SRL phases. We therefore propose to split the scale ‘metacog-
nitive skills’ into three separate subscales: activities before, during, and after a learn-
ing task. Not only would a separation into these three scales lead to an improvement
of the face validity of the questionnaire, but it would also allow for more specific use
of the questionnaire’s (sub)scales, and for conclusions to be drawn about specific
phases in the SRL process.
Based on the possible methodological and theoretical improvements on the scale
‘metacognitive skills’ outlined above, the aim of the current study is to create and test
a revised version of the SOL-Q to improve its validity, reliability, and usability.
2 Method
2.1 SOL-Q revised (SOL-Q-R)
The scale metacognitive skills within the SOL-Q was expanded and revised to gener-
ate three subscales. The existing 18 items in the scale were divided over the three
subscales (i.e., before, during and after learning) based on the meaning of the item
and on words signaling the timing of the activity. For instance, the item ‘I am aware
of what strategies I use when I study for this online course’ was placed into the sub-
scale ‘metacognitive activity during learning’. Second, the subscales were comple-
mented to make sure all relevant aspects of metacognition were sufficiently present in
3
each subscale. Strategic planning in the preparatory phase was not present in the ex-
isting items and only four appraisal items were present. Therefore, an item measuring
strategic planning was added to the scale ‘metacognitive activity before learning’ (‘At
the start of a task I think about the study strategies I will use’), and two items measur-
ing reflection on learning progress and learning strategies were added to the scale
‘metacognitive activity after learning’ (‘After studying for this online course I reflect
on what I have learned’ and ‘After learning for this online course, I think about the
study strategies I used’). Specific attention was paid to words signaling timing when
formulating the new items.
Furthermore, three small adaptations were made to improve the validity and relia-
bility of the questionnaire. The first adaptation concerned the item ‘I know what the
instructor expects me to learn in this online course’, originating from the Metacogni-
tive Awareness Inventory scale for task definition [11]. Factor analyses during the
development of the SOL-Q placed the item in the scale ‘environmental structuring’.
As the item does not measure environmental structuring, and is therefore also not
conceptually similar to the other items in the scale, the item was removed from the
questionnaire. Second, there were three negatively phrased items in the original de-
sign of the SOL-Q. These items were removed after factor analyses, as they did not fit
the factor structure. Polar opposite items (i.e., ‘I often feel so lazy or bored when I
study for this online course, that I quit before I finish what I planned to do’) are how-
ever known to result in lower internal-consistency reliabilities [12]. These three items,
two in the persistence scale and one in the help-seeking scale, were rephrased to be
polar positive and added to the SOL-Q-R. Finally, the time management scale was
slightly adapted to improve its reliability as it was the scale with low reliability in the
SOL-Q, which was likely due to the small size of the scale (3 items). Therefore, two
items were added to the scale. The first item was already part of the originally devel-
oped questionnaire, but fell out during factor analyses. As the item conceptually fits in
the scale, it was re-added (‘I make good use of my study time for this online course’.).
The second item was formulated in line with the meaning of the scale (‘I make good
use of my study time for this online course.’).
The answering format was not changed for the SOL-Q-R. All questions had to be
answered on a 7-point Likert scale ranging from ‘not at all true for me’ (= 1) to ‘very
true for me’ (= 7). The full SOL-Q-R can be found at SOONER.NU/SOL-Q-R.
2.2 Participants and procedure
The SOL-Q-R was administered to two groups of MOOC participants and one group
of participants in a blended university course.
First, the questionnaire was implemented as a voluntary activity in a MOOC on
Clinical Epidemology offered by Utrecht University, The Netherlands, on Coursera.
This MOOC consisted of 7 modules: an introductory module, 4 content modules, a
module with a peer-graded assignment, and a module with a final exam. While stu-
dents were free to decide on their own pace of studying, one module per week was
recommended. The questionnaire was added as a voluntary activity at the end of
Module 2, to make sure students could reflect on their actual learning in the online
4
course, and would not answer based on what they planned or expected to do. Com-
plete data was gathered from 149 students. The responses of three students were con-
sidered outliers as they answered all questions identically (SD of their answers was
0). Responses of 146 students were used for analyses (Mage = 36.08, 48.6% male).
The questionnaire was also implemented as a voluntary activity in a MOOC on
Environmental Sustainability offered by Wageningen University, The Netherlands, on
edX. The MOOC consisted of seven modules: an introductory module and six content
modules. In this MOOC, students were also free to study at their own pace, while one
module per week was recommended. The questionnaire was added as a voluntary
activity at the end of Module 2. Complete data was gathered from 73 students. Three
students were considered outliers (SD = 0). Responses of 70 students were used for
analyses (Mage = 39.67 40.0% male).
The SOL-Q-R was also administered in a blended higher education course about
designing educational materials at Utrecht University, the Netherlands. The course
lasted 10 weeks, and followed a weekly structure of online preparation activities and
face to face teacher-guided sessions (i.e., a flipped classroom design). In week 10, the
students took an individual exam. The questionnaire was added as a voluntary online
activity in week 4 of the course. Complete data was gathered from 94 students. One
student was considered an outlier (SD = 0). Responses of 93 students were used for
analyses (Mage = 23.59, 10.8% male).
2.3 Analyses
The SOL-Q and SOL-Q-R were compared based on reliability analyses. Furthermore,
model fit was calculated using SPSS AMOS to test if the revised version had accepta-
ble model fit. In line with the analyses done for the development of the SOL-Q [5],
NC (normed Chi square) and RMSEA (root mean square error of approximation)
were used as absolute fit statistics [13, 14].
3 Results
Reliability analyses were conducted to compare the internal-consistency reliabilities
of the SOL-Q and the SOL-Q-R (Table 1). The results of the reliability analyses indi-
cate higher reliabilities for the scales time management, environmental structuring,
persistence, and help seeking in the SOL-Q-R. The reliability of the three metacogni-
tive subscales are slightly lower than the reliability of the metacognitive skills scale.
However, reliability is above .740 for all subscales, indicating good reliability.
5
Table 1. Internal-consistency reliabilities of the SOL-Q and SOL-Q-R scales.
1
2
3
1
2
3
Scale
Items
α
α
α
Items
α
α
α
Metacognitive skills
18
.93
.91
.88
Activities before
7
.87
.84
.77
Activities during
7
.82
.78
.75
Activities after
6
.86
.86
.81
Time management
3
.57
.72
.71
5
.68
.72
.80
Environmental
structuring
5
.78
.74
.66
4
.82
.77
.69
Persistence
5
.78
.70
.84
7
.82
.76
.88
Help seeking
5
.87
.91
.82
6
.88
.90
.84
Note. Dataset 1 = MOOC Clinical Epidemology, 2 = MOOC Environmental Sustain-
ability, and 3 = Flipped course educational materials.
An overview of the model fit statistics of the SOL-Q-R is presented in Table 2.
Normed Chi square (NC) is a measure of χ² corrected for sample size, as χ² is known
to be highly influenced by sample size [13]. Values of NC between 2.0 and 3.0 indi-
cate acceptable fit and smaller values are better [13]. All tested models score below
2.0thus indicating good fit of the SOL-Q-R in all three datasets. For RMSEA, smaller
values indicate better fit and values below .08 are reasonable [15]. Based on the
RMSEA statistic, the revised version shows adequate fit only in the first dataset,
which is also largest. RMSEA is known to indicate poor model fit for small samples
[16], which may explain the RMSEA values above .08 for dataset 2 and 3.
Table 2. Absolute model fit statistics of the SOL-Q-R.
MOOC 1
MOOC 2
Blended
NC
1.797
1.700
1.713
RMSEA
.074
.101
.088
4 Discussion
In this paper, a revised version of the SOL-Q was presented and tested: the SOL-Q-R.
The revised version has increased face validity, as the items within the scales were
conceptually more similar. The separation of the large scale metacognitive skills into
three smaller subscales (metacognitive activity before, during, and after learning)
increases the usability of the questionnaire, as specific aspects of metacognition can
be measured with the revised version. The theoretical and practical value of the ques-
tionnaire thus increases in the revised version. The results of the reliability analyses
showed that the adaptations furthermore led to reliable scales overall (all α above
.67), with increased reliability for most scales. Model fit statistics are somewhat am-
biguous, but provide no argument against acceptance of the SOL-Q-R. To conclude,
the revised version of the SOL-Q is an improved version of the SOL-Q in terms of
6
validity, reliability and usability and is therefore considered a valuable tool for re-
searchers to measure students’ SRL in online education. The full SOL-Q-R can be
found at SOONER.NU/SOL-Q-R.
5 Acknowledgement
This work is financed via a grant by the Dutch National Initiative for Education Re-
search (NRO)/The Netherlands Organisation for Scientific Research (NWO) and the
Dutch Ministry of Education, Culture and Science under the grant nr. 405-15-705
(SOONER/http://sooner.nu).
This manuscript has been published as part of the EC-TEL 2018 conference pro-
ceedings. The article can be accessed via
https://link.springer.com/chapter/10.1007%2F978-3-319-98572-5_9.
References
1. Garrison, D.R.: Self-directed learning and distance education. In: Moore, M.G. and An-
derson, W.G. (eds.) Handbook of distance education. pp. 161–168. Lawrence Erlbaum As-
sociates, Mahwah, NJ, US (2003).
2. Wang, C.-H., Shannon, D.M., Ross, M.E.: Students’ characteristics, self-regulated learn-
ing, technology self-efficacy, and course outcomes in online learning. Distance Educ. 34,
302–323 (2013).
3. Kizilcec, R.F., Pérez-Sanagustín, M., Maldonado, J.J.: Self-regulated learning strategies
predict learner behavior and goal attainment in Massive Open Online Courses. Comput.
Educ. 104, 18–33 (2017).
4. Beishuizen, J., Steffens, K.: A conceptual framework for research on self-regulated learn-
ing. In: Carneiro, R., Lefrere, P., Steffens, K., and Underwood, J. (eds.) Self-Regulated
Learning in Technology Enhanced Learning Environments. pp. 3–19. Sense Publishers,
Rotterdam, The Netherlands (2011).
5. Jansen, R.S., van Leeuwen, A., Janssen, J., Kester, L., Kalz, M.: Validation of the self-
regulated online learning questionnaire. J. Comput. High. Educ. 29, 6–27 (2017).
6. Zimmerman, B.J.: Becoming a self-regulated learner: An overview. Theory Pract. 41, 64–
70 (2002).
7. Puustinen, M., Pulkkinen, L.: Models of self-regulated learning: A review. Scand. J. Educ.
Res. 45, 269–286 (2001).
8. Pintrich, P.R., Smith, D.A.F., García, T., McKeachie, W.J.: A manual for the use of the
motivated strategies for learning questionnaire (MSLQ). University of Michigan, National
Center for Research to Improve Postsecondary Teaching and Learning, Ann Arbor, MI
(1991).
9. Zimmerman, B.J.: Investigating Self-Regulation and Motivation: Historical Background,
Methodological Developments, and Future Prospects. Am. Educ. Res. J. 45, 166–183
(2008).
10. Sitzmann, T., Ely, K.: A meta-analysis of self-regulated learning in work-related training
and educational attainment: What we know and where we need to go. Psychol. Bull. 137,
421–442 (2011).
7
11. Schraw, G., Dennison, R.S.: Assessing Metacognitive Awareness. Contemp. Educ. Psy-
chol. 19, 460–475 (1994).
12. Woods, C.M.: Careless Responding to Reverse-Worded Items: Implications for Confirma-
tory Factor Analysis. J. Psychopathol. Behav. Assess. 28, 186–191 (2006).
13. Kline, R.B.: Details of Path Analysis. In: Kenny, D.A. (ed.) Principles and Practice of
Structual Equation Modeling. The Guilford Press, New York, NY (2005).
14. Hooper, D., Coughlan, J., Mullen, M.: Structural Equation Modelling: Guidelines for De-
termining Model Fit. Electron. J. Bus. Res. Methods. 6, 53–60 (2008).
15. Gatignon, H.: Confirmatory Factor Analysis. In: Gatignon, H. (ed.) Statistical Analysis of
Management Data. pp. 59–122. Springer New York, New York, NY (2010).
16. Kenny, D.A., Kaniskan, B., McCoach, D.B.: The Performance of RMSEA in Models With
Small Degrees of Freedom. Sociol. Methods Res. 44, 486–507 (2015).
