Conference PaperPDF Available

Abstract and Figures

In the past decade, the popularity of systematic reviews in the field of educational technology has grown rapidly. They allow researchers to explicitly and critically appraise data from a range of studies, synthesizing findings to identify research gaps and to inform practice and policy. This paper focuses primarily on one systematic review currently being undertaken in Germany, which seeks to understand how student engagement is affected by educational technology in higher education. As student engagement is a complex construct, an overview of the theoretical framework will be provided, alongside how its reimagining effected the searching, screening and coding of articles. The potential and possible pitfalls of undertaking systematic reviews will also be reflected upon, as identified in the current review and within the broader educational technology literature.
No caption available
… 
No caption available
… 
Content may be subject to copyright.
Systematic Reviews in Educational Technology Research: Potential and pitfalls
Abstract
In the past decade, the popularity of systematic reviews in the field of educational
technology has grown rapidly. They allow researchers to explicitly and critically
appraise data from a range of studies, synthesizing findings to identify research gaps
and to inform practice and policy. This paper focuses primarily on one systematic
review currently being undertaken in Germany, which seeks to understand how student
engagement is affected by educational technology in higher education. As student
engagement is a complex construct, an overview of the theoretical framework will be
provided, alongside how its reimagining effected the searching, screening and coding
of articles. The potential and possible pitfalls of undertaking systematic reviews will also
be reflected upon, as identified in the current review and within the broader educational
technology literature.
Background and Motivation
Conducting systematic reviews has become more prominent in the field of educational
research (Gough & Thomas, 2016). Defined as “a review of a clearly formulated question that
uses systematic and explicit methods to identify, select, and critically appraise relevant
research, and to collect and analyse data from the studies that are included in the review”
(Moher et al., 2009, p. 1), they serve two purposes: establishing a base for evidence-based
decision making on the policy level and also thoroughly analyzing a specific field of research
to identify research gaps (Gough, Oliver & Thomas, 2012). This also applies to the field of
educational technology and its applications, which recent systematic reviews within the field
have shown, for example the application of augmented reality in education (Akcayir & Akcayir,
2016), Web 2.0 technologies for student learning (Hew & Cheung, 2012), and learning and
engagement within MOOCs (Joksimovic, et al., 2017).
In the broader context of teaching and learning research, the construct of student engagement
(e.g. Dunne & Owen, 2013; Kahu, 2013, Christensen et al., 2012) has received increased
attention in the past decade, as it is directly linked to students’ learning outcomes and cognitive
development (Ma, Han, Yang, & Cheng, 2015). Research has shown that using technology
can predict increased student engagement (Rashid & Asghar, 2016; Chen, Lambert, & Guidry,
2010), including through improved self-efficacy and self-regulation (Alioon & Delialioglu, 2017;
Bouta, Retalis, & Paraskeva, 2012), and increased participation and involvement (Salaber,
2014; Northey, Bucic, Chylinkski, & Govind, 2015; Alioon & Delialioglu, 2017). However,
without careful planning and sound pedagogy, technology can promote disengagement and
impede rather than help learning (Popenici, 2013; Howard, Ma, & Yang, 2016).
The overarching question of how educational technology can support student engagement in
higher education is the focus of the research project ActiveLeaRn, funded from 2016 to 2019
by the German Federal Ministry for Education and Research (BMBF). A systematic review is
being conducted as part of this project, whose results will be validated through discussion with
practitioners in the field hence closing the often times perceived research practitioner gap
(Belli, 2010).
Contribution
In this Brief Paper contribution, the authors place emphasis on three central aspects of the
systematic review and share their hands-on experience with this method. Topics covered
include student engagement, the process of searching and screening studies, and the potential
and pitfalls of the method, as identified within the current review and within the field of
educational technology.
Student engagement
The ‘meta-construct’ of student engagement is multifaceted and complex (Kahu, 2013;
Appleton, Christenson, & Furlong, 2008; Christenson, Reschly, & Wylie, 2012; Fredricks, et
al., 2004), with ongoing disagreement about its definition and form (e.g. Reeve & Tseng, 2011;
Zepke & Leach, 2010). Recent reviews (Joksimovic, et al., 2017; Henrie, Halverson, &
Graham, 2015) have attempted to synthesize educational technology literature, in order to
further develop the construct, and this systematic review further adds to this body of work. An
overview of the construct will be provided, alongside how its complexity influenced the
systematic review approach.
Searching and screening studies
In order to identify the studies relevant for inclusion in the review (Brunton et al., 2012), an
intentionally broad search string was developed, piloted and then applied to four major
databases in the field (Web of Science, PsychINFO, ERIC and Scopus). Pre-defined inclusion
criteria were applied; contributions are peer-reviewed articles in English language journals,
published in 2007 or after, discuss technology-enhanced learning and student engagement
and target students in higher education. By means of the PRISMA statement (Moher et al.,
2009), collected references were documented during the searching process and then screened
on title and abstract. Following initial screening, 4,152 potential includes remained for closer
analysis. Peculiarities of these two parts of the systematic reviews will be discussed in the
presentation.
Potential and pitfalls
In the course of the execution of the review, a number of issues have emerged that require
further attention and are helpful for other researchers to consider when conducting future
reviews in the field. Alongside an appraisal of the systematic review method, examples shared
in the presentation will include questions regarding how to develop complex search strings for
study identification, the management of large reference corpi, the importance of research
teams for conducting systematic reviews, and deciding for or against text mining for use within
reviews in the broad field of educational technology research.
The contribution closes with an outlook on the further steps within the review.
References
Alioon, Y., & Delialioglu, O. (2017). The effect of authentic m-learning activities on student
engagement and motivation. British Journal of Educational Technology, 1-14.
Appleton, J., Christenson, S., & Furlong, M. (2008). Student Engagement with School: Critical
conceptual and methodological issues of the construct. Psychology in the Schools,
45(5), 369-386.
Belli, G. (2010, July). Bridging the researcher-practitioner gap: Views from different fields.
In ICOTS8 Invited Paper. In C Reading (ed). Data and context in statistics education:
towards an evidence-based society. Proceedings of the 8th International Conference
on Teaching Statistics (ICOTS 8)(pp. 11-16).
Bouta, H., Retalis, S., & Paraskeva, F. (2012). Utilising a collaborative macro-script to
enhance student engagement: A mixed method study in a 3D virtual environment.
Computers & Education, 58, 501-517. Chen, P.-S., Lambert, A., & Guidry, K. (2010).
Engaging online learners: The impact of Web-based learning technology on college
student engagement. Computers & Education, 54(4), 1222-1232.
Christenson, S. L., Reschly, A. L., & Wylie, C. (Eds.). (2012). Handbook of Research on
Student Engagement. Boston, MA: Springer US. https://doi.org/10.1007/978-1-4614-
2018-7
Dunne, E., & Owen, D. (Eds.). (2013). The student engagement handbook: practice in higher
education (1. edition). United Kingdom North America Japan India Malaysia China:
Emerald.
Fredricks, J., Blumenfeld, P., & Paris, A. (2004). School Engagement: Potential of the
Concept, State of the Evidence. Review of Educational Research, 74(1), 59-109.
Gough, D., Oliver, S., & Thomas, J. (2012). Introducing systematic reviews. In D. Gough, S.
Oliver, & J. Thomas (Eds.), An introduction to systematic reviews. (pp. 116).
London: SAGE Publications.
Gough, D., & Thomas, J. (2016). Systematic reviews of research in education: aims, myths
and multiple methods. Review of Education, 4(1), 84102.
Henrie, C. R., Halverson, L. R., & Graham, C. R. (2015). Measuring student engagement in
technology-mediated learning: A review. Computers & Education, 90, 3653.
https://doi.org/10.1016/j.compedu.2015.09.005
Hew, K. F., & Cheung, W. S. (2013). Use of Web 2.0 technologies in K-12 and higher
education: The search for evidence-based practice. Educational Research Review, 9,
4764. https://doi.org/10.1016/j.edurev.2012.08.001
Howard, S., Ma, J., & Yang, J. (2016). Student rules: Exploring patterns of students’ computer
efficacy and engagement with digital technologies in learning. Computers & Education,
101, 29-42.
Joksimovic, S., Poquet, O., Kovanovic, V., Dowell, N., Mills, C., Gasevic, D., Dawson, S.,
Graesser, A., Brooks, C. (2017). How do we Model Learning at Scale? A Systematic
Review of Research on MOOCs. Review of Educational Research, 1-62.
Kahu, E. R. (2013). Framing student engagement in higher education. Studies in Higher
Education, 38(5), 758773. https://doi.org/10.1080/03075079.2011.598505
Ma, J., Han, X., Yang, J., & Cheng, J. (2015). Examining the necessary condition for
engagement in an online learning environment based on learning analytics approach:
The role of the instructor. Internet and Higher Education, 24, 26-34.
Northey, G., Bucic, T., Chylinkski, M., & Govind, R. (2015). Increasing Student Engagement
Using Asynchronous Learning. Journal of Marketing Education, 37(3), 171-180.
Popenici, S. (2013). Towards a New Vision for University Governance, Pedagogies and
Student Engagement. In E. Dunne, & D. Owen (Hrsg.), The Student Engagement
Handbook: Practice in Higher Education (S. 23-42). Bingley: Emerald Publishing Group
Ltd.
Rashid, T., & Asghar, H. (2016). Technology use, self-directed learning, student engagement
and academic performance: Examining the interrelations. Computers in Human
Behaviour, 63, 604-612.
Reeve, J., & Tseng, C.-M. (2011). Agency as a fourth aspect of students’ engagement during
learning activities. Contemporary Educational Psychology, 36(4), 257-267.
Salaber, J. (2014). Facilitating student engagement and collaboration in a large postgraduate
course using wiki-based activities. The International Journal of Management
Education, 12, 115-126.
Zepke, N., & Leach, L. (2010). Improving student engagement: Ten proposals for action.
Active Learning in Higher Education, 11(3), 167-177.
doi:10.1177/1469787410379680
Article
The main aim of this article is to explore how performance-based budgeting has an effect on primary health care service delivery in the Eastern Cape. This article assessed the performance-based budgeting system in the Eastern Cape Department of Health in an attempt to advocate for effective and efficient health service delivery. The Eastern Cape Province is succumbing to insufficient funding at the provincial level and this has a clear-cut reflection on the administration of services within respective departments. The article is premised on a constructivist paradigm that adopts a qualitative approach where document analysis was employed for collecting data. The Eastern Cape Province is succumbing to insufficient funding at the provincial level and this has a clear-cut reflection on the administration of services within respective departments. The article employed the Public Choice Theory and the Concept of New Public Management. Three main findings emerged from the article of the crippled funds in the province, the human resources crisis, poor public finance management, and shortage of health equipment. Data were analyzed using thematic analysis to examine the distinctive themes and sub-themes identified from within the reviewed literature. Recommendations were thus given based on the discussed and derived themes to ensure that there is an appropriate budgeting system to ensure efficient and effective provision of health care services in the Eastern Cape Department of Health.
Article
Full-text available
Despite a surge of empirical work on student participation in online learning environments, the causal links between the learning-related factors and processes with the desired learning outcomes remain unexplored. This study presents a systematic literature review of approaches to model learning in Massive Open Online Courses offering an analysis of learning related constructs used in the prediction and measurement of student engagement and learning outcome. Based on our literature review, we identify current gaps in the research, including a lack of solid frameworks to explain learning in open online setting. Finally, we put forward a novel framework suitable for open online contexts based on a well-established model of student engagement. Our model is intended to guide future work studying the association between contextual factors (i.e., demographic, classroom, and individual needs), student engagement (i.e., academic, behavioral, cognitive, and affective engagement metrics) and learning outcomes (i.e., academic, social, and affective). The proposed model affords further inter-study comparisons as well as comparative studies with more traditional education models.
Article
Full-text available
Student success and retention continue to be of concern for higher education institutions. Wider participation, combined with lower completion rates for non-traditional students, highlights the need for new ways of understanding the student experience to ground policy and practice. This article provides this insight by drawing together a number of key constructs to refine a recent framework of student engagement. We argue that the transition metaphor, focusing on the first year, is limited because it depicts differences between students and institutions as both transient and temporal. Instead, we use a cultural lens to introduce the educational interface as a metaphor for the individual psychosocial space within which institutional and student factors combine and student engagement in learning occurs. Incorporating the interface into the existing framework of student engagement makes three contributions to our understanding of the student experience. First, the educational interface is a tangible way of representing the complex interactions between students and institutions, and how those interactions influence engagement. Second, the refined framework highlights four specific psychosocial constructs: self-efficacy, emotions, belonging and well-being, which, we contend, are critical mechanisms for mediating the interactions between student and institutional characteristics and student engagement and success. Finally, the refined framework helps to explain why some students with demographic characteristics associated with lower completion rates are retained and do go on to successfully complete their studies, while similar others do not. These three contributions, the interface, the key constructs within it being mediating mechanisms and their explanatory utility, provide focus for the design and implementation of curricula and co-curricular initiatives aimed at enhancing student success and retention, and importantly to evaluate the impact of these interventions.
Book
Full-text available
This book is all about Student Engagement
Article
Authentic collaborative m-learning activities were designed, developed and implemented for a computer networking course. The effect of the activities on student engagement and motivation were analyzed using a mixed method research design. Moreover, the effect of the iterative design of the content and instructional process of authentic m-learning activities on students' engagement and motivation were analyzed in the study. The activities were implemented for two consecutive semesters and were modified based on the findings from the first semester. Student engagement survey and motivation questionnaire were used to collect quantitative data, student interview protocol was used to collect qualitative data for further investigation. The findings from the first semester indicated that the engagement categories “personal development” and “satisfaction from the course” had the highest mean scores. In the second semester, the highest mean score belonged to the “personal development” category, followed by “collaborative learning.” Comparison of the results from two semesters revealed that the improvements in the content of the activities and instructional process increased the “collaboration” among students as well as their “interaction with instructor.” Paired sample t-tests revealed a difference in indicators of student motivation within groups in both semesters. Analysis of the interview data showed that students perceived the authentic activities as an appropriate tool for enhancement in “communication” and “collaboration” opportunities.
Article
This systematic review examined multiple indicators of adolescent students' engagement in school, and the indicators' associations with teacher–student relationships (TSRs). Seven psychology, education, and social sciences databases were systematically searched. From this search, 46 published studies (13 longitudinal) were included for detailed analysis. Cross-sectional studies showed better quality TSRs were associated with enhanced engagement in school. These associations with TSRs were demonstrated among multiple indicators of student engagement (i.e., psychological engagement, academic grades, school attendance, disruptive behaviors, suspension, and dropout). Similar associations were found in longitudinal studies. Longitudinal and cross-sectional associations remained when covariates from the individual, family, school, and teacher contexts known to influence student engagement were controlled for. TSRs were shown to have an important but not exclusive role in their association with a comprehensive range of indicators of student engagement.
Article
Teachers' beliefs about students' engagement in and knowledge of digital technologies will affect technologically integrated learning designs. Over the past few decades, teachers have tended to feel that students were confident and engaged users of digital technologies, but there is a growing body of research challenging this assumption. Given this disparity, it is necessary to examine students' confidence and engagement using digital technologies to understand how differences may affect experiences in technologically integrated learning. However, the complexity of teaching and learning can make it difficult to isolate and study multiple factors and their effects. This paper proposes the use of data mining techniques to examine unique patterns among key factors of students' technology use and experiences related to learning, as a way to inform teachers' practice and learning design. To do this, association rules mining and fuzzy representations are used to analyze a large student questionnaire dataset (N = 8,817). Results reveal substantially different patterns among school engagement and computer-efficacy factors between students with positive and negative engagement with digital technologies. Findings suggest implications for learning design and how teachers may attend to different experiences in technologically integrated learning and future research in this area.
Article
Systematic reviews are still a controversial topic in some quarters, with the arguments for and against their use being well-rehearsed. In an attempt to advance a more nuanced approach to thinking about systematic reviewing, this paper illustrates the wide range of theoretical perspectives, methodologies and purposes that underpin the vast range of systematic review approaches now available; and in the light of this picture, re-examines some of the perennial arguments against reviews, arguing that they are often poorly targeted, based on a misreading of what systematic reviews aim to do, or simply incommensurable with the tenets that underpin academic enquiry.