Fig 24 - uploaded by Valerie J. Shute
Content may be subject to copyright.
2 Physics understanding competency model in PP
Source publication
Learning analytics (LA) dashboards refer to digital tools designed to help learners keep track of their progress and goals. There is growing interest and research around the topic of LA dashboards in online learning environments, with many lessons to be learned by educational game developers and researchers. However, we need more research in this a...
Contexts in source publication
Context 1
... physics parameters (i.e., gravity, air resistance, mass, and bounciness of the ball) and also manipulate external forces exerted from puffers or blowers to hit the balloon-no drawing is allowed in manipulation levels (Fig. 24.1b). PP's number of game levels is dynamic-we have created about 150 game levels covering nine physics competencies ( Fig. 24.2). We can add game levels to the online version of PP at any time using the game's level ...
Context 2
... designed a multipurpose dashboard in PP called My Backpack where students can see their progress-shown at the top part of Fig. 24.3 (i.e., the number of levels they solved, the number of gold or silver coins they collected, and the amount of money they earned). Each gold coin (given for an elegant solution for a game level) earns the student $20, and each silver coin (given for a solution that did not meet the criteria needed for a gold coin) earns $10. Students can ...
Context 3
... addition to showing game progress (e.g., 6 out of 22 sketching levels solved), students can monitor their level of physics understanding (Fig. 24.3) based on the current stealth assessment estimates. These estimates are for (a) each of the specific nine competencies (shown in Fig. 24.3 with the orange bar charts) and (b) their overall physics understanding (shown at the bottom of Fig. 24.3 in green). My Backpack also includes a store (see Fig. 24.4) where students can spend the ...
Context 4
... addition to showing game progress (e.g., 6 out of 22 sketching levels solved), students can monitor their level of physics understanding (Fig. 24.3) based on the current stealth assessment estimates. These estimates are for (a) each of the specific nine competencies (shown in Fig. 24.3 with the orange bar charts) and (b) their overall physics understanding (shown at the bottom of Fig. 24.3 in green). My Backpack also includes a store (see Fig. 24.4) where students can spend the game money they earned through gameplay to customize their game by "buying" new background music, background images, and different ball ...
Context 5
... (e.g., 6 out of 22 sketching levels solved), students can monitor their level of physics understanding (Fig. 24.3) based on the current stealth assessment estimates. These estimates are for (a) each of the specific nine competencies (shown in Fig. 24.3 with the orange bar charts) and (b) their overall physics understanding (shown at the bottom of Fig. 24.3 in green). My Backpack also includes a store (see Fig. 24.4) where students can spend the game money they earned through gameplay to customize their game by "buying" new background music, background images, and different ball types. We designed My Backpack through an iterative process considering various design decisions that we mentioned in the ...
Context 6
... their level of physics understanding (Fig. 24.3) based on the current stealth assessment estimates. These estimates are for (a) each of the specific nine competencies (shown in Fig. 24.3 with the orange bar charts) and (b) their overall physics understanding (shown at the bottom of Fig. 24.3 in green). My Backpack also includes a store (see Fig. 24.4) where students can spend the game money they earned through gameplay to customize their game by "buying" new background music, background images, and different ball types. We designed My Backpack through an iterative process considering various design decisions that we mentioned in the introduction. Game store in My Backpack which ...
Context 7
... proper actions; see recommendation 1.3.4). Consequently, we simplified the estimates. That is, instead of using three probabilities (associated with being high, medium, or low) per competency, we computed a single number (i.e., the expected a posteriori, or EAP value) ranging from −1 (low) to 1 (high) and presented that data in a bar chart (see Fig. 24.3). The EAP value for a competency is expressed as P (θ ij = High) -P (θ ij = Low), where θ ij is the value for student i on competency j, and [1 × P(High)] + [0 × P(Med)] + [−1 × P(Low)] = P(High) -P(Low). Finally, to make this value even more understandable, we normalized it on a scale ranging from 0 to 1 (using this formula: (EAP + 1) ...
Context 8
... full explanation of PP's architecture is outside of the scope of this chapter. Therefore, we only focus on the parts related to the stealth assessment processes and how My Backpack gets updated during gameplay. PP uses two separate servers: the PP Server (shown in Fig. 24.5 on the left) which hosts the game engine and the Assessment Server (shown in Fig. 24.5 on the right). The Assessment Server has two main components: (1) the Dongle component which is responsible for providing a student's prior data and their latest statistics per competency (i.e., EAPs) and (2) the assessment engine which includes ...
Context 9
... explanation of PP's architecture is outside of the scope of this chapter. Therefore, we only focus on the parts related to the stealth assessment processes and how My Backpack gets updated during gameplay. PP uses two separate servers: the PP Server (shown in Fig. 24.5 on the left) which hosts the game engine and the Assessment Server (shown in Fig. 24.5 on the right). The Assessment Server has two main components: (1) the Dongle component which is responsible for providing a student's prior data and their latest statistics per competency (i.e., EAPs) and (2) the assessment engine which includes two processes: evidence identification (EI) and evidence accumulation ...
Similar publications
Currently, the application of educational games is becoming a hot topic, but the application of educational games in college English learning is scarcely explored. To explore the integration of educational games into the college English teaching and learning process, this study adopted a rapid evidence evaluation review (Yu et al. 2020). Since the...
Citations
... By viewing the information displayed in LADs, learners can gain awareness of their learning progress, prompting them to evaluate their learning performance, including strategies and plans. In other words, the self-monitoring and evaluation of learning facilitated by using LADs help students become active learners who proactively adjust their learning strategies and behaviors (Rahimi & Shute, 2021;Susnjak et al., 2022;Verbert et al., 2013). ...
... This study proved that LADs grounded in SDT could be effective tools in supporting student engagement in AOL environments. Generally, LADs are expected to foster student engagement by helping learners become aware of and reflect on their learning process, thereby enabling them to adjust their strategies or plans to achieve their goals (Rahimi & Shute, 2021;Susnjak et al., 2022;Verbert et al., 2013). The interview results, which revealed learners' experiences with the fulfillment of their BPNs through the LAD and their autonomous data-driven decision-making processes fueled by this fulfillment, demonstrate that the LAD this study designed aligns with these expectations. ...
Although learning analytics dashboards (LADs) are being recognized as tools that can enhance engagement—a crucial factor for the success of asynchronous online higher education—their impact may be limited without a solid theoretical basis for motivation. Furthermore, the processes through which students make decisions using dashboards and engage are not well understood. This study aimed to design a LAD informed by self-determination theory and to investigate university students’ experiences with it. The findings, including those from stimulated recall interviews using eye-tracking data, shed light on how the LAD fosters student engagement. Interacting with the LAD fulfilled students’ basic psychological needs. Awareness and reflection on learning status facilitated by the LAD boosted enthusiasm for active learning participation. The LAD offered essential information to support autonomous, strategic decisions, empowering students to take proactive actions toward personal goals while reinforcing their belief in achieving them. Despite its potential benefits, various improvements have been identified to further enhance its effectiveness. Based on the findings, we discuss the implications of this study for future research in the field.
... PK: Online and digital learning environments, instructional technologies, and gamebased learning environments have seen a rise in recent years (e.g., Feng, Krumm, & Grover, 2018;Heffernan & Heffernan, 2014;Rahimi & Shute, 2021;Sinatra, et al., 2020). Students in several districts across the United States now have district-provided laptop and tablet devices which gives them access to various types of digital and online learning platforms. ...
The field of score reporting continues to evolve because of new challenges, opportunities, and needs of society (e.g., COVID, remote teaching and learning). In this paper, the new challenges and opportunities in score reporting are discussed from the personal perspective of four experts who have previously conducted research in designing score reports in education. Comments are organized around four key questions concerning challenges raised by the Covid pandemic, how research will change, what current research is being conducted, and new directions in the field of score reporting.
... Online and digital learning environments, adaptive instructional technologies, and game-based learning and assessment environments have seen a rise in recent years (Heffernan and Heffernan, 2014;Feng et al., 2018;Sinatra et al., 2020;Rahimi and Shute, 2021). Large and varied types of data about students' overall learning experiences (including process and log data) are now available within these digital environments, and it would be most helpful if these data are used to provide interpretable, useful, and actionable feedback to teachers in the classroom context. ...
... Learning analytic dashboards (LADs) have become increasingly popular for providing feedback in these digital contexts (Papamitsiou and Economides, 2014;Sahin and Ifenthaler, 2021). The data visualizations and reports used within LADs are intended to help students understand their progress toward goals and help teachers make data-informed decisions in formative learning contexts (Bayrak et al., 2021;Dickler, 2021;Keskin and Yurdugül, 2021;Rahimi and Shute, 2021). However, research has found that the data presented within LADs are often not based on an evaluation of stakeholder needs (Sahin and Ifenthaler, 2021), and are often not clearly interpretable and actionable (Molenaar and Knoop-van Campen, 2018;Sahin and Ifenthaler, 2021;Valle et al., 2021) in a way that lends effective pedagogical support to teachers; therefore, more research on LADs is warranted (Rahimi and Shute, 2021). ...
... The data visualizations and reports used within LADs are intended to help students understand their progress toward goals and help teachers make data-informed decisions in formative learning contexts (Bayrak et al., 2021;Dickler, 2021;Keskin and Yurdugül, 2021;Rahimi and Shute, 2021). However, research has found that the data presented within LADs are often not based on an evaluation of stakeholder needs (Sahin and Ifenthaler, 2021), and are often not clearly interpretable and actionable (Molenaar and Knoop-van Campen, 2018;Sahin and Ifenthaler, 2021;Valle et al., 2021) in a way that lends effective pedagogical support to teachers; therefore, more research on LADs is warranted (Rahimi and Shute, 2021). ...
Learning analytic dashboards (LADs) are data visualization systems that use dynamic data in digital learning environments to provide students, teachers, and administrators with a wealth of information about student’s engagement, experiences, and performance on tasks. LADs have become increasingly popular, particularly in formative learning contexts, and help teachers make data-informed decisions about a student’s developing skills on a topic. LADs afford the possibility for teachers to obtain real-time data on student performance, response processes, and progress on academic learning tasks. However, data presented on LADs are often not based on an evaluation of stakeholder needs, and have been found to not be clearly interpretable and actionable for teachers to readily adapt their pedagogical actions based on these insights. We elaborate on how insights from research focused on interpretation and use of Score Reporting systems and research on open learner models (OLMs) can be used to inform a research agenda aimed at exploring the design and evaluation of LADs.
Research fields related to learning (e.g., educational technology and learning sciences) have historically focused on what questions using traditional methods (e.g., comparing different learning tools and methods). New methodologies that are grounded in learning, engagement, and motivational theories are needed to additionally address the how questions. Methodologies that use learners’ process data shed more light on how learners learn and if the learning tools are effective, compared to methodologies that use just outcome data. In this paper, we discuss stealth assessment—an evidence-based methodology that can be used in technology-rich environments (e.g., games) to assess and support hard-to-measure constructs (e.g., creativity) as well as knowledge acquisition (e.g., physics). We also discuss evidence-centered design (ECD), and present specific steps to design, embed in a digital learning environment, and evaluate a stealth assessment. Additionally, we provide two examples of stealth assessment studies in the context of an educational game called Physics Playground: Study 1 illustrates a stealth assessment of creativity and Study 2 describes a stealth assessment of physics understanding and how we used it to make the game adaptive. The purpose of this paper is to provide sufficient detail about stealth assessment to help researchers in the field of educational technology and related fields to adopt this method to assess, foster, and investigate learning processes in various technology-rich environments.
