ArticlePDF Available

Abstract and Figures

Video-based self-reflection and annotation is receiving increasing attention within the education literature. The importance of such technologies in education relate, in part, to the interactive nature and functionality these tools bring to aid learning engagement. In particular, these tools are well aligned with the need to promote and develop student meta-cognitive skills through the use of self-reflection activities. However, in the context of video-based learning environments, the nature of a students’ self-reflective process is not well understood. We attempt to address this gap in the literature in two main ways. First, we developed a coding instrument to assess the depth of a students’ self-reflection captured through the use of a video annotation tool. We then explored the linguistic and discourse properties of the student self-reflections using Coh-Metrix, a theoretically grounded computational linguistics facility. The adopted approach applies comprehensive analysis of language and discourse features associated with the specificity of students’ internal self-feedback, which is externalized as self-reflections in video annotations. The results suggest that levels of self-reflection have characteristically different linguistic properties, and these differences align with the underlying cognitive mechanisms associated with distinct reflective activities. The paper provides a detailed discussion of the findings in the context of the theoretical, methodological, and practical implications associated with video-based self-reflection and video annotation.
Content may be subject to copyright.
A preview of the PDF is not available
... Several models were developed to analyze the content of reflective writing, both in terms of the broad themes covered [6], [9] and the depth of reflection exhibited [5], [7], [10] by learners. While traditional approaches to assessing reflective writing have utilized manual content analysis, with the advances in natural language processing and computational content analytics, there have been few attempts to use automated approaches to assess student reflective writing [1], [3], [11]- [15]. These automated assessment studies have mainly evaluated the descriptive nature of the writing contents triggered by an experience, with a lesser focus on understanding the specificity of learners' reflections, primarily positioned in traditional classroom settings and professional degree programs within higher education. ...
... While the first two approaches are expert-driven and utilize patterns elicited by experts, the machine learning approaches are fully automated, detecting the patterns without any human supervision [12]. The studies adopting machine-learning approaches are based on natural language processing (NLP) methods, utilizing textual features derived from Coh-Metrix indices [1], [11], Linguistic Inquiry and Word Count variables [1], [35]- [38], or simple linguistic utterances such as N-grams [1], [12] to evaluate reflective writing into different categories. ...
... While there has been a considerable number of attempts within educational research to automate the process of evaluating self-reflection and reflective writing, most of these automated systems are based on iterative or hybrid iterativevertical models [1], [3], [11], [12] with a few exceptions [14], [15]. As a consequence, these models thereby cannot distinguish learners based on reflective depth. ...
... Nonetheless, the effectiveness of annotations in VBL depends on students' learning strategies and motivation (Pardo et al., 2015;Mirriahi, Liaqat, Dawson, & Gašević, 2016), which emphasises the need to design adaptive interventions on video annotation. In recent studies, text and learning analytics are leveraged to characterise the learning process in VBL (Joksimović et al., 2019;Dodson et al., 2018a;Seo et al., 2021). However, the insights derived from these analyses are not used to develop automatic personalised support for video-based annotations. ...
... Several assessment frameworks were proposed for different types of reflective writing in the literature (Ullmann, 2017(Ullmann, , 2019Joksimović et al., 2019;Dimitrova et al., 2013). However, we developed a new labelling scheme for AVW-Space due to its distinctive nature, based on the explorations of comments (Mohammadhassan, Mitrovic, Neshatian, & Dunn, 2020) collected in the 2018 and 2019 studies. ...
... The quality schemes include ordinal categories, meaning the first category has the lowest quality, and the last category shows the highest quality. The ordinal categories are also used in the ICAP framework, and in other research focusing on higher-order thinking (Wang, Wen, & Rosé, 2016;Joksimović et al., 2019). We evaluated the proposed quality schemes by selecting 167 comments from 2018 to 2019 studies (110 and 57 comments on tutorial and example videos, respectively) via stratified sampling. ...
Article
Full-text available
Developing and maintaining constructive engagement is a crucial challenge in learning by watching videos. AVW-Space is an online video-based learning platform which enhances student engagement via note-taking and personalised support. Previous studies with AVW-Space show that students who write comments, especially high-quality comments, learn more. The goal of the study reported in this paper is to encourage students to write better-quality comments. After automating the assessment of comment quality using machine learning approaches, we developed Quality nudges which encourage students to write better comments. We conducted a study in a first-year engineering course to analyse the learning effects of the Quality nudges. The results show that Quality nudges enhanced constructive engagement and learning. The contribution of this research is in proposing methodology for increasing the quality of student comments in video-based learning.
... The focus of linguistic analysis is on syntactic and semantic components of textual data. A study on video annotations in the CLAS note-taking environment (Risko et al., 2012) proposed a scheme to identify the level of reflections in the annotations (Joksimović et al., 2019). This study used Coh-Metrix, a computational linguistics facility (Graesser et al., 2014), to derive linguistic properties to determine the depth of students' self-reflection. ...
... A study on academic reflective essays identifies different types of reflection such as personal belief, lessons learned or future intentions (Ullmann, 2017;2019). Another framework for assessing the depth of students' self-reflections on their performance groups the contents into observation, effect, or motivation and goal (Joksimović et al., 2019). A simulation environment for cross-cultural communications classifies the user's textual interactions with the system into different groups, such as statements on the situation and real-world stories (Dimitrova et al., 2013). ...
Conference Paper
Full-text available
Active Video Watching (AVW-Space) is an online platform for video-based learning which supports engagement via note-taking and personalized nudges. In this paper, we focus on the quality of the comments students write. We propose two schemes for assessing the quality of comments. Then, we evaluate these schemes by computing the inter-coder agreement. We also evaluate various machine learning classifiers to automate the assessment of comments. The selected cost-sensitive classifier shows that the quality of comments can be assessed with high weighted-F1 scores. This study contributes to the automation of comments assessment and the development of personalized educational support for engagement in video-based learning through commenting.
... In addition, the system may help to identify the cognitive depth of accumulated timeline-anchored comments through content analysis and adapt the layout accordingly. Recent researchers (Joksimović et al., 2019) have proposed coding instruments to identify the depth of self-reflection that is revealed in annotations to video lecture timelines. Such text-analytic methods can be adopted to design platforms that can adapt the layout to content depth automatically. ...
Article
Full-text available
Many video sites or learning platforms allow real-time chatting or asynchronous commenting on specific time points during video lectures. Comments, as user-generated knowledge, facilitate social interaction but also affect cognitive learning. The visual layout of these comments can affect learners' attention and learning, but the effect has rarely been studied. This study compares two common layouts (embedded vs. separated) and considers the content depth of comments through a laboratory eye-tracking experiment involving 40 participants. The results suggest that, with both layouts, learners switched attention to the comments every 10 seconds and stayed focused for 1.3 seconds on average before returning attention to the video. With an embedded layout, learners switched attention more frequently to the comments and remembered more surface-level comments. With a separate layout presenting deep-level comments, learners searched for information faster and performed better on open-book quizzes. We outline the design implications of using timeline-anchored comments to promote online learning.
... The focus is on gaining insight into distinct learning skills and can take place before, during or after the learning activity. Joksimovic et al. (2019) show that metacognitive awareness can promote reflective states of consciousness. Their study builds on the assumption concerning how metacognitive knowledge shapes this awareness of own cognitive processes and how one understands, manages or regulates these processes in order to enhance learning. ...
... The focus is on gaining insight into distinct learning skills and can take place before, during or after the learning activity. Joksimovic et al. (2019) show that metacognitive awareness can promote reflective states of consciousness. Their study builds on the assumption concerning how metacognitive knowledge shapes this awareness of own cognitive processes and how one understands, manages or regulates these processes in order to enhance learning. ...
Book
Full-text available
This book aims to contribute to the discourse of learning through assessment within a self-directed learning environment. It adds to the scholarship of assessment and self-directed learning within a face-to-face and online learning environment. As part of the NWU Self-Directed Learning Book Series, this book is devoted to scholarship in the field of self-directed learning, focusing on ongoing and envisaged assessment practices for self-directed learning through which learning within the 21st century can take place. This book acknowledges and emphasises the role of assessment as a pedagogical tool to foster self-directed learning during face-to-face and online learning situations. The way in which higher education conceptualises teaching, learning and assessment has been inevitably changed due to the COVID- 19 pandemic, and now more than ever we need learners to be self-directed in their learning. Assessment plays a key role in learning and, therefore, we have to identify innovative ways in which learning can be assessed, and which are likely to become the new norm even after the pandemic has been brought under control. The goal of this book, consisting of original research, is to assist with the paradigm shift regarding the purpose of assessment, as well as providing new ideas on assessment strategies, methods and tools appropriate to foster self-directed learning in all modes of delivery.
... Chatti et al. (2016) embed text comparison techniques to define the similarity between annotations in order to facilitate annotation organisation and search. Joksimović et al. (2019) explore linguistic and discourse properties to classify student self-reflections in video-based annotations. Hoppe et al. (2016) apply network text analysis of video comments while students share videos to extract information on the learners' domain understanding and detect possible misconceptions. ...
Article
Full-text available
Video-based learning is widely used today in both formal education and informal learning in a variety of contexts. Videos are especially powerful for transferable skills learning (e.g. communicating, negotiating, collaborating), where contextualization in personal experience and ability to see different perspectives are crucial. With the ubiquity of widely available video content, video-based learning is seen as one of the main strategies to provide engaging learning environments. However, numerous studies show that to learn effectively while watching videos, students need to engage actively with video content. We have developed an active video watching platform (AVW-Space) to facilitate engagement with video content by providing means for constructive learning. The initial studies with AVW-Space on presentation skills show that only students who commented on videos and who rated comments written by their peers have improved their understanding of the target transferable skill. In order to foster deeper engagement, we designed a choice architecture and a set of nudges to encourage students to engage deeper. We conducted two studies investigating the effect of nudges. The results provide evidence that the initial set of implemented nudges is effective: the students who received nudges wrote more comments, used different aspects, and there were more students who wrote comments. The nudges were particularly helpful for undergraduate students who were less experienced in self-regulated learning. Future work includes designing additional nudges to enhance student engagement by improving the quality of comments and by encouraging participation in collaborative activities.
... A continuación, debe realizar una actividad de reflexión, de forma que seleccione qué aspectos de los tratados en el texto podrían enriquecerse o explicarse mejor. La siguiente actividad consiste en realizar una búsqueda en otras fuentes de información para obtener los contenidos con los que se va a crear la anotación, para lo cual deberá desarrollar varias competencias de aprendizaje (Joksimović, 2019) dado que tendrá que saber cómo buscar, dónde buscar y saber valorar la calidad de los contenidos. Una vez encontrados los contenidos, se debe elaborar la anotación. ...
Chapter
Full-text available
Este artículo muestra los resultados del estudio de algunos rasgos de personalidad de los estudiantes de Ingeniería Mecánica Agrícola (IMA) de la Universidad Autónoma Chapingo (UACh) con el objetivo de fortalecer su proceso formativo mediante estrategias que se ajusten a las necesidades individuales y grupales. La investigación es empírica, transversal y cuantitativa de carácter descriptivo. El instrumento utilizado fue el software Turborienta que evalúa 160 factores de manera integral y se aplicó a una muestra de 244 estudiantes de los grados 4°, 5° y 6°. Los resultados muestran los valores promedio grupales de los indicadores de la competencia afectiva siguientes: del autoconcepto de valía – autoestima = 6.2, seguridad en sí mismo = 6.0, disposición al perdón = 6.2, control de agresividad = 6.2, tolerancia = 5.6 y respeto que recibe de terceros = 6.9. Para la competencia ética son: disposición a la ética = 5.7, sentido de justicia = 6.3, respeto a los mayores = 6.4, respeto a sí mismo = 6.3, rechazo a diversiones desenfrenadas = 5.9, respeto a su íntima conciencia = 5.9 y generosidad = 5.7. En la competencia actitudinal los promedios son: libertad = 5.3, sexualidad = 5.4, logro de seguridad = 4.9, temor y agresividad = 6.2, gregario = 4.8, curiosidad = 3.9, lúdico = 5.7 y competencia = 5.2. Estos resultados son de gran ayuda para el desarrollo del programa de tutorías académicas.
... A continuación, debe realizar una actividad de reflexión, de forma que seleccione qué aspectos de los tratados en el texto podrían enriquecerse o explicarse mejor. La siguiente actividad consiste en realizar una búsqueda en otras fuentes de información para obtener los contenidos con los que se va a crear la anotación, para lo cual deberá desarrollar varias competencias de aprendizaje (Joksimović, 2019) dado que tendrá que saber cómo buscar, dónde buscar y saber valorar la calidad de los contenidos. Una vez encontrados los contenidos, se debe elaborar la anotación. ...
Chapter
Full-text available
Este artículo muestra los resultados del estudio de algunos rasgos de personalidad de los estudiantes de Ingeniería Mecánica Agrícola (IMA) de la Universidad Autónoma Chapingo (UACh) con el objetivo de fortalecer su proceso formativo mediante estrategias que se ajusten a las necesidades individuales y grupales. La investigación es empírica, transversal y cuantitativa de carácter descriptivo. El instrumento utilizado fue el software Turborienta que evalúa 160 factores de manera integral y se aplicó a una muestra de 244 estudiantes de los grados 4°, 5° y 6°. Los resultados muestran los promedios grupales de los indicadores siguientes del temperamento: activo = 5.9, impulsivo = 4.6, vigoroso = 6.1, dominante =4.6, estable = 5.7, sociable = 4.7, reflexivo = 5.5, personalidad = 5.14 (normal), grado de madurez = 6.1, calidad humana = 5.8, adaptabilidad = 5.3, ambición =5.9, respuesta a la autoridad = 6.3. Estos resultados son de gran ayuda para el desarrollo del programa de tutorías académicas.
... Previous studies on AVW-Space show that commenting causes deeper thinking and reflection . The analysis of learners' reflections has been an area of research which aims at gaining insights on learners' engagement and their educational characteristics (Hoppe et al., 2016;Joksimović et al., 2018;Taskin et al., 2019). The knowledge extracted from the analysis of comments can help in developing pedagogical support for fostering engagement in different types of learners. ...
Conference Paper
Full-text available
AVW-Space is an online video-based learning platform which aims to improve engagement by providing a note-taking environment and personalized support. This paper presents a PhD project focusing on the nudges about the quality of comments learners make on the videos in AVW-Space. We first automated the quality assessment of comments using machine learning approaches. Based on the predicted quality of the comments and the learners' profiles, we developed nudges which encourage students to write better comments. Finally, we conducted a study in a first-year engineering course to analyze the learning effects of these nudges. This PhD research contributes to the development of personalized educational support for engagement in video-based learning via fostering better comment writing behavior.
Conference Paper
Full-text available
Completion rates for massive open online classes (MOOCs) are notoriously low, but learner intent is an important factor. By studying students who drop out despite their intent to complete the MOOC, it may be possible to develop interventions to improve retention and learning outcomes. Previous research into predicting MOOC completion has focused on click-streams, demographics, and sentiment analysis. This study uses natural language processing (NLP) to examine if the language in the discussion forum of an educational data mining MOOC is predictive of successful class completion. The analysis is applied to a subsample of 320 students who completed at least one graded assignment and produced at least 50 words in discussion forums. The findings indicate that the language produced by students can predict with substantial accuracy (67.8 %) whether students complete the MOOC. This predictive power suggests that NLP can help us both to understand student retention in MOOCs and to develop automated signals of student success.
Article
Full-text available
This paper presents the results of a natural experiment investigating the effects of instructional conditions and experience on the adoption and sustained use of a learning tool. The experiment was conducted with undergraduate students, enrolled into four performing art courses (N=77) at a research intensive university in Canada. The students used the video annotation software CLAS for course‐based self‐assessment on their performances. Although existing research offers insights into the factors predicting students' intentions of accepting a learning tool, much less is known about factors that affect actual adoption and sustained tool use. The study explored the use of CLAS amongst undergraduate students in four courses across two consecutive semesters. Trace data of students' tool use, graph‐based measures of metacognitive monitoring, and text cohesion of video annotations were used to estimate the volume of tool use and the quality of the learning strategy and learning products created. The results confirmed that scaffolding (e.g., graded activity with instructional feedback) is required to guide students' initial tool use, although scaffolding did not have an independent significant effect on the quantity of tool use. The findings demonstrated that the use of the tool is strongly influenced by the experience an individual student gains from scaffolded conditions. That is, the students sustained their use of the learning tool in future courses even when the tool use was not graded nor was instructional feedback provided. An important implication is that students' tool use is not solely driven by motivation – rather, it is shaped by instructional conditions and experience with the tool use.
Article
Full-text available
The purpose of this article is to explore a model of learning that proposes that various learning strategies are powerful at certain stages in the learning cycle. The model describes three inputs and outcomes (skill, will and thrill), success criteria, three phases of learning (surface, deep and transfer) and an acquiring and consolidation phase within each of the surface and deep phases. A synthesis of 228 meta-analyses led to the identification of the most effective strategies. The results indicate that there is a subset of strategies that are effective, but this effectiveness depends on the phase of the model in which they are implemented. Further, it is best not to run separate sessions on learning strategies but to embed the various strategies within the content of the subject, to be clearer about developing both surface and deep learning, and promoting their associated optimal strategies and to teach the skills of transfer of learning. The article concludes with a discussion of questions raised by the model that need further research.
Conference Paper
Full-text available
The rapidly-developing 21st century world of work and knowledge calls for self-directed lifelong (SDL) learners. While higher education must embrace the types of pedagogies that foster SDL skills in graduates, the pace of change in education can be glacial. This paper describes a social annotation technology, the Collaborative Lecture Annotation System (CLAS), that can be used to leverage existing teaching and learning practices for acquisition of 21st Century SDL skills. CLAS was designed to build upon the artifacts of traditional didactic modes of teaching, create enriched opportunities for student engagement with peers and learning materials, and offer learners greater control and ownership of their individual learning strategies. Adoption of CLAS creates educational experiences that promote and foster SDL skills: motivation, self-management and self-monitoring. In addition, CLAS incorporates a suite of learning analytics for learners to evaluate their progress, and allow instructors to monitor the development of SDL skills and identify the need for learning support and guidance. CLAS stands as an example of a simple tool that can bridge the gap between traditional transmissive pedagogy and the creation of authentic and collaborative learning spaces. © 2012 Dawson, S., Macfadyen, L., Risko, E., Foulsham, T. & Kingstone, A.
Book
The 4th edition of the Handbook of Research on Educational Communications and Technology expands upon the previous 3 versions, providing a comprehensive update on research pertaining to new and emerging educational technologies. Chapters that are no longer pertinent have been eliminated in this edition, with most chapters being completely rewritten, expanded, and updated Additionally, new chapters pertaining to research methodologies in educational technology have been added due to expressed reader interest. Each chapter now contains an extensive literature review, documenting and explaining the most recent, outstanding research, including major findings and methodologies employed. The Handbook authors continue to be international leaders in their respective fields; the list is cross disciplinary by design and great effort was taken to invite authors outside of the traditional instructional design and technology community. © Springer Science+Business Media New York 2014. All rights reserved.
Chapter
The focus of this chapter is to examine the psychological principles of metacognition and identify how second language (L2) educators can increase learners’ awareness of their metacognition. The chapter will present research findings to support the integration of metacognitive strategy awareness training within an L2 curriculum. Training learners to be more cognizant and reflective of how they engage in language learning facilitates the development of learner autonomy.
Article
Learners self-regulate learning when they examine the learning environment and monitor learning processes to inform decisions about managing learning processes. Under the assumption that learners are agents with inherent capability to make and act on decisions, self-regulated learning (SRL) is ubiquitous. Decisions learners make about regulating learning, however, can promote or impede achievement and other valued outcomes. Research on SRL and its roles in education faces multiple challenges. Evidence suggests productive SRL can be fostered through evidenced-based designs for learning activities and environments, by promoting particular beliefs among teachers, and helping learners discover value in education.