Content uploaded by Srecko Joksimovic
Author content
All content in this area was uploaded by Srecko Joksimovic on Mar 03, 2018
Content may be subject to copyright.
A preview of the PDF is not available
Linguistic Characteristics of Reflective States in Video Annotations under Different Instructional Conditions
Abstract and Figures
Video-based self-reflection and annotation is receiving increasing attention within the education literature. The importance of such technologies in education relate, in part, to the interactive nature and functionality these tools bring to aid learning engagement. In particular, these tools are well aligned with the need to promote and develop student meta-cognitive skills through the use of self-reflection activities. However, in the context of video-based learning environments, the nature of a students’ self-reflective process is not well understood. We attempt to address this gap in the literature in two main ways. First, we developed a coding instrument to assess the depth of a students’ self-reflection captured through the use of a video annotation tool. We then explored the linguistic and discourse properties of the student self-reflections using Coh-Metrix, a theoretically grounded computational linguistics facility. The adopted approach applies comprehensive analysis of language and discourse features associated with the specificity of students’ internal self-feedback, which is externalized as self-reflections in video annotations. The results suggest that levels of self-reflection have characteristically different linguistic properties, and these differences align with the underlying cognitive mechanisms associated with distinct reflective activities. The paper provides a detailed discussion of the findings in the context of the theoretical, methodological, and practical implications associated with video-based self-reflection and video annotation.
Figures - uploaded by Srecko Joksimovic
Author content
All figure content in this area was uploaded by Srecko Joksimovic
Content may be subject to copyright.
... Several models were developed to analyze the content of reflective writing, both in terms of the broad themes covered [6], [9] and the depth of reflection exhibited [5], [7], [10] by learners. While traditional approaches to assessing reflective writing have utilized manual content analysis, with the advances in natural language processing and computational content analytics, there have been few attempts to use automated approaches to assess student reflective writing [1], [3], [11]- [15]. These automated assessment studies have mainly evaluated the descriptive nature of the writing contents triggered by an experience, with a lesser focus on understanding the specificity of learners' reflections, primarily positioned in traditional classroom settings and professional degree programs within higher education. ...
... While the first two approaches are expert-driven and utilize patterns elicited by experts, the machine learning approaches are fully automated, detecting the patterns without any human supervision [12]. The studies adopting machine-learning approaches are based on natural language processing (NLP) methods, utilizing textual features derived from Coh-Metrix indices [1], [11], Linguistic Inquiry and Word Count variables [1], [35]- [38], or simple linguistic utterances such as N-grams [1], [12] to evaluate reflective writing into different categories. ...
... While there has been a considerable number of attempts within educational research to automate the process of evaluating self-reflection and reflective writing, most of these automated systems are based on iterative or hybrid iterativevertical models [1], [3], [11], [12] with a few exceptions [14], [15]. As a consequence, these models thereby cannot distinguish learners based on reflective depth. ...
Self-reflection and reflective writing have been pivotal for developing a deep understanding of concepts and fostering professional competency in learners. The confluence of the importance of reflective practices within the educational curriculum and the increased proliferation of technology have resulted in numerous studies of how to use automated approaches to analyze broad themes of reflection exhibited by learners in higher educational settings. However, there is a dearth of research in the context of automated analysis of reflective writing demonstrated by professionals within workplace learning. Building on a four-level reflective content analysis model, this article evaluates the depth of reflection exhibited by learners within a professional development MOOC using an automated assessment classifier. Our results identify the varying association of linguistic features across these different levels of reflection. The proposed approach can effectively support the deployment at scale of the labor-intensive evaluation of reflective writing by highly trained professionals.
... Nonetheless, the effectiveness of annotations in VBL depends on students' learning strategies and motivation (Pardo et al., 2015;Mirriahi, Liaqat, Dawson, & Gašević, 2016), which emphasises the need to design adaptive interventions on video annotation. In recent studies, text and learning analytics are leveraged to characterise the learning process in VBL (Joksimović et al., 2019;Dodson et al., 2018a;Seo et al., 2021). However, the insights derived from these analyses are not used to develop automatic personalised support for video-based annotations. ...
... Several assessment frameworks were proposed for different types of reflective writing in the literature (Ullmann, 2017(Ullmann, , 2019Joksimović et al., 2019;Dimitrova et al., 2013). However, we developed a new labelling scheme for AVW-Space due to its distinctive nature, based on the explorations of comments (Mohammadhassan, Mitrovic, Neshatian, & Dunn, 2020) collected in the 2018 and 2019 studies. ...
... The quality schemes include ordinal categories, meaning the first category has the lowest quality, and the last category shows the highest quality. The ordinal categories are also used in the ICAP framework, and in other research focusing on higher-order thinking (Wang, Wen, & Rosé, 2016;Joksimović et al., 2019). We evaluated the proposed quality schemes by selecting 167 comments from 2018 to 2019 studies (110 and 57 comments on tutorial and example videos, respectively) via stratified sampling. ...
Developing and maintaining constructive engagement is a crucial challenge in learning by watching videos. AVW-Space is an online video-based learning platform which enhances student engagement via note-taking and personalised support. Previous studies with AVW-Space show that students who write comments, especially high-quality comments, learn more. The goal of the study reported in this paper is to encourage students to write better-quality comments. After automating the assessment of comment quality using machine learning approaches, we developed Quality nudges which encourage students to write better comments. We conducted a study in a first-year engineering course to analyse the learning effects of the Quality nudges. The results show that Quality nudges enhanced constructive engagement and learning. The contribution of this research is in proposing methodology for increasing the quality of student comments in video-based learning.
... Todos ellos son nuevas oportunidades, técnicas y modelos para la formación de docentes que han surgido al albor del desarrollo tecnológico en la digitalización y computación del vídeo, pero que de algún modo sirven de soporte y recurso, y en algunos casos algo más que esto, al definir el sentido de los modelos pedagógicos con los que se asocian, como de las concepciones acerca de lo que se entiende que debe ser la formación de docentes. Tal es el caso del Modelo microteaching (Sezen-Barrie et al., 2014), el Modelo TPACK (Cabero, 2014;Leiva-Núñez et al., 2018) y otros modelos que preconizan el análisis sobre la práctica y la interpretación de significados a través de esta reflexión -en la acción y sobre la acción- (Schön, 1992), reflexión crítica y la autorregulación del aprendizaje (Kim Chau & Mei Po, 2021), el análisis del discurso narrativo (Joksimović et al., 2018), la investigación acción (Ardley & Johnson, 2018), las redes profesionales, y comunidades de prácticas -CoP-(Brown & Poortman, 2018) y de indagación -CoI- (Blau & Shamir-Inbal, 2021), siendo estas dos últimas las más generalizadas actualmente y más propias del uso de las metodologías y plataformas tecnológicas de AV, al dotarlas de un aprendizaje más activo, formativo, reflexivo, y con más retroalimentación sobre el recurso vídeo, tanto entre los estudiantes del mismo grupo de clase como dotando de significados y valores en la comunicación evaluativa entre los docentes y los estudiantes. ...
... All of them are new opportunities, techniques and models for teacher training that have emerged at the dawn of technological development in video digitization and computing, but that somehow serve as a support and resource, and in some cases something more than this, by defining the meaning of the pedagogical models with which they are associated, as of the conceptions about what it is understood that teacher training should be. Such is the case of the Microteaching Model (Sezen- Barrie et al., 2014), the TPACK Model (Cabero, 2014;Leiva-Núñez et al., 2018) and other models that advocate the analysis on practice and the interpretation of meanings through this reflection -in action and on action- (Schön, 1992), critical reflection and self-regulation of learning (Kim Chau & Mei Po, 2021), narrative discourse analysis (Joksimović et al., 2018), action research (Ardley & Johnson 2018), professional networks, and communities of practice -CoP-(Brown & Poortman, 2018) and communities of inquiry -CoI- (Blau & Shamir-Inbal, 2021), the latter two being the most widespread nowadays and more typical of the use of VA methodologies and technological platforms, by providing them with a more active, formative, reflective learning, and with more feedback on the video resource, both among students in the same class group and by providing meanings and values in the evaluative communication between teachers and students. ...
xiste una tradición en el uso de videos digitales en la formación del profesorado donde las anotaciones de vídeo son una tecnología emergente que ofrece un uso más activo a los vídeos disponibles en internet y los creados en las prácticas de aula. Sus posibilidades técnicas consisten en crear anotaciones en los vídeos de forma individual para el portafolios o compartida en grupos de clases y redes profesionales. Como tecnología en evolución ofrece nuevas posibilidades para la formación docente mediante etiquetado social e hipervínculos que pueden ser analizados. El presente estudio es una revisión de la literatura reciente desde 2018 a 2022 sobre el uso de las anotaciones multimedia en la formación docente mediante el protocolo PRISMA, seleccionando desde bases de datos (WOS, Scopus y ERIC) 244 referencias que tras aplicar los criterios de exclusión e inclusión se obtuvieron 25 referencias que aplican sólo en el tópico de la formación de docentes. Se ofrece una lista de las plataformas más empleadas, los métodos de investigación utilizados, el perfil de los participantes y los tópicos de aplicación. Es compartida la utilidad de las anotaciones de video en la formación de docentes especialmente para el análisis y reflexión sobre las prácticas.
... In addition, the system may help to identify the cognitive depth of accumulated timeline-anchored comments through content analysis and adapt the layout accordingly. Recent researchers (Joksimović et al., 2019) have proposed coding instruments to identify the depth of self-reflection that is revealed in annotations to video lecture timelines. Such text-analytic methods can be adopted to design platforms that can adapt the layout to content depth automatically. ...
Many video sites or learning platforms allow real-time chatting or asynchronous commenting on specific time points during video lectures. Comments, as user-generated knowledge, facilitate social interaction but also affect cognitive learning. The visual layout of these comments can affect learners' attention and learning, but the effect has rarely been studied. This study compares two common layouts (embedded vs. separated) and considers the content depth of comments through a laboratory eye-tracking experiment involving 40 participants. The results suggest that, with both layouts, learners switched attention to the comments every 10 seconds and stayed focused for 1.3 seconds on average before returning attention to the video. With an embedded layout, learners switched attention more frequently to the comments and remembered more surface-level comments. With a separate layout presenting deep-level comments, learners searched for information faster and performed better on open-book quizzes. We outline the design implications of using timeline-anchored comments to promote online learning.
... The focus is on gaining insight into distinct learning skills and can take place before, during or after the learning activity. Joksimovic et al. (2019) show that metacognitive awareness can promote reflective states of consciousness. Their study builds on the assumption concerning how metacognitive knowledge shapes this awareness of own cognitive processes and how one understands, manages or regulates these processes in order to enhance learning. ...
... The focus is on gaining insight into distinct learning skills and can take place before, during or after the learning activity. Joksimovic et al. (2019) show that metacognitive awareness can promote reflective states of consciousness. Their study builds on the assumption concerning how metacognitive knowledge shapes this awareness of own cognitive processes and how one understands, manages or regulates these processes in order to enhance learning. ...
This book aims to contribute to the discourse of learning through assessment within a self-directed learning environment. It adds to the scholarship of assessment and self-directed learning within a face-to-face and online learning environment. As part of the NWU Self-Directed Learning Book Series, this book is devoted to scholarship in the field of self-directed learning, focusing on ongoing and envisaged assessment practices for self-directed learning through which learning within the 21st century can take place. This book acknowledges and emphasises the role of assessment as a pedagogical tool to foster self-directed learning during face-to-face and online learning situations. The way in which higher education conceptualises teaching, learning and assessment has been inevitably changed due to the COVID- 19 pandemic, and now more than ever we need learners to be self-directed in their learning. Assessment plays a key role in learning and, therefore, we have to identify innovative ways in which learning can be assessed, and which are likely to become the new norm even after the pandemic has been brought under control. The goal of this book, consisting of original research, is to assist with the paradigm shift regarding the purpose of assessment, as well as providing new ideas on assessment strategies, methods and tools appropriate to foster self-directed learning in all modes of delivery.
In der Lehrerinnen- und Lehrerbildung hat der Einsatz videobasierter Methoden Tradition und gewinnt weiter an Zuspruch. Den Vorteilen dieser Form stehen jedoch auch Nachteile gegenüber: Die Flüchtigkeit der Videoform erschwert eine gehaltvolle gemeinsame Auseinandersetzung mit den Inhalten, besonders in asynchronen Online-Settings. Tools zur kollaborativen Annotation von Videos können hier einen Ausweg bieten. Der Markt hierfür ist jedoch disparat und unübersichtlich, sodass es schwierig ist, die richtige Anwendung zu finden. Dabei will der vorliegende Beitrag helfen, indem die maßgeblichen Anwendungen identifiziert und mit ihren Vor- und Nachteilen aufgeführt werden.
This chapter reports on teaching an Auditing and Assurance course using team-based scenario learning (TBSL) for undergraduates. Introducing TBSL as a collaborative active learning tool encourages students to learn from one another and provides opportunities for communication, discussion, argument, and elaboration. The chapter utilises simulated scenarios in three different situations such as analysis of audit risk factors, material misstatements, and audit and non-audit services. It is followed by discussion forums, and videos are utilised to represent real-life audit situations. The combination of these various learning activities encourages diverse arguments being put forward by students in their discussions, and creates opportunities for critical thinking and to interact with the group members. In this regard, attention is given to student group formation and workshops on collaborative skills are conducted. The TBSL requires students to be confident to give peer feedback and engage in self-reflection on their learning journey. In line with the authentic learning interactions, the scenario-based assessment focuses on rewarding students on the development of essential skills and capabilities needed in auditing.
Video annotation (VA), a tool which allows commentaries to be synchronized with video content has recently received significant research attention in education. However, the application contexts of these studies are varied and fragmented. A review was therefore undertaken with the objectives to find out the extent to which the use of VA has been explored for different instructional purposes and summarize the potential affordances of VA in supporting student learning. Articles related to the use of VA in education context were searched from 2011 to 2020 (Nov). Of the final 32 eligible studies, it was found that VA tools were used predominantly to develop teaching practices, enhance learners’ conceptual understanding of video content and develop workplace skills as well as clinical practices. Five most dominant educational affordances of VA tools were summarized as follows: (1) facilitating learners’ reflection (2) facilitating feedback process (3) enhancing comprehension of video content (4) promoting students’ learning satisfaction and positive attitude and (5) convenience and ease. With the outstanding weight of research evidence gained on educational affordances offered by VA, it is convincing that advancing the use of VA in education can further expand the learning opportunities in 21st century classrooms. Keywords: Affordances, Education, Feedback, Learners’ reflection, Video annotation
Completion rates for massive open online classes (MOOCs) are notoriously low, but learner intent is an important factor. By studying students who drop out despite their intent to complete the MOOC, it may be possible to develop interventions to improve retention and learning outcomes. Previous research into predicting MOOC completion has focused on click-streams, demographics, and sentiment analysis. This study uses natural language processing (NLP) to examine if the language in the discussion forum of an educational data mining MOOC is predictive of successful class completion. The analysis is applied to a subsample of 320 students who completed at least one graded assignment and produced at least 50 words in discussion forums. The findings indicate that the language produced by students can predict with substantial accuracy (67.8 %) whether students complete the MOOC. This predictive power suggests that NLP can help us both to understand student retention in MOOCs and to develop automated signals of student success.
This paper presents the results of a natural experiment investigating the effects of instructional conditions and experience on the adoption and sustained use of a learning tool. The experiment was conducted with undergraduate students, enrolled into four performing art courses (N=77) at a research intensive university in Canada. The students used the video annotation software CLAS for course‐based self‐assessment on their performances. Although existing research offers insights into the factors predicting students' intentions of accepting a learning tool, much less is known about factors that affect actual adoption and sustained tool use. The study explored the use of CLAS amongst undergraduate students in four courses across two consecutive semesters. Trace data of students' tool use, graph‐based measures of metacognitive monitoring, and text cohesion of video annotations were used to estimate the volume of tool use and the quality of the learning strategy and learning products created. The results confirmed that scaffolding (e.g., graded activity with instructional feedback) is required to guide students' initial tool use, although scaffolding did not have an independent significant effect on the quantity of tool use. The findings demonstrated that the use of the tool is strongly influenced by the experience an individual student gains from scaffolded conditions. That is, the students sustained their use of the learning tool in future courses even when the tool use was not graded nor was instructional feedback provided. An important implication is that students' tool use is not solely driven by motivation – rather, it is shaped by instructional conditions and experience with the tool use.
The purpose of this article is to explore a model of learning that proposes that various learning strategies are powerful at certain stages in the learning cycle. The model describes three inputs and outcomes (skill, will and thrill), success criteria, three phases of learning (surface, deep and transfer) and an acquiring and consolidation phase within each of the surface and deep phases. A synthesis of 228 meta-analyses led to the identification of the most effective strategies. The results indicate that there is a subset of strategies that are effective, but this effectiveness depends on the phase of the model in which they are implemented. Further, it is best not to run separate sessions on learning strategies but to embed the various strategies within the content of the subject, to be clearer about developing both surface and deep learning, and promoting their associated optimal strategies and to teach the skills of transfer of learning. The article concludes with a discussion of questions raised by the model that need further research.
The rapidly-developing 21st century world of work and knowledge calls for self-directed lifelong (SDL) learners. While higher education must embrace the types of pedagogies that foster SDL skills in graduates, the pace of change in education can be glacial. This paper describes a social annotation technology, the Collaborative Lecture Annotation System (CLAS), that can be used to leverage existing teaching and learning practices for acquisition of 21st Century SDL skills. CLAS was designed to build upon the artifacts of traditional didactic modes of teaching, create enriched opportunities for student engagement with peers and learning materials, and offer learners greater control and ownership of their individual learning strategies. Adoption of CLAS creates educational experiences that promote and foster SDL skills: motivation, self-management and self-monitoring. In addition, CLAS incorporates a suite of learning analytics for learners to evaluate their progress, and allow instructors to monitor the development of SDL skills and identify the need for learning support and guidance. CLAS stands as an example of a simple tool that can bridge the gap between traditional transmissive pedagogy and the creation of authentic and collaborative learning spaces. © 2012 Dawson, S., Macfadyen, L., Risko, E., Foulsham, T. & Kingstone, A.
The 4th edition of the Handbook of Research on Educational Communications and Technology expands upon the previous 3 versions, providing a comprehensive update on research pertaining to new and emerging educational technologies. Chapters that are no longer pertinent have been eliminated in this edition, with most chapters being completely rewritten, expanded, and updated Additionally, new chapters pertaining to research methodologies in educational technology have been added due to expressed reader interest. Each chapter now contains an extensive literature review, documenting and explaining the most recent, outstanding research, including major findings and methodologies employed. The Handbook authors continue to be international leaders in their respective fields; the list is cross disciplinary by design and great effort was taken to invite authors outside of the traditional instructional design and technology community. © Springer Science+Business Media New York 2014. All rights reserved.
The focus of this chapter is to examine the psychological principles of metacognition and identify how second language (L2) educators can increase learners’ awareness of their metacognition. The chapter will present research findings to support the integration of metacognitive strategy awareness training within an L2 curriculum. Training learners to be more cognizant and reflective of how they engage in language learning facilitates the development of learner autonomy.
Learners self-regulate learning when they examine the learning environment and monitor learning processes to inform decisions about managing learning processes. Under the assumption that learners are agents with inherent capability to make and act on decisions, self-regulated learning (SRL) is ubiquitous. Decisions learners make about regulating learning, however, can promote or impede achievement and other valued outcomes. Research on SRL and its roles in education faces multiple challenges. Evidence suggests productive SRL can be fostered through evidenced-based designs for learning activities and environments, by promoting particular beliefs among teachers, and helping learners discover value in education.