Valerie J. Shute

Florida State University | FSU · Department of Educational Psychology and Learning Systems

Stealth assessment, like all assessments, must have three essential psychometric properties: validity, reliability, and fairness. Evidence-centered assessment design (ECD) provides a psychometrically sound framework for designing assessments based on a validity argument. This chapter describes how using ECD in the design of a stealth assessment hel...

With advances in technology and the learning and assessment sciences, educators can develop learning environments that can accurately and engagingly assess and improve learners' knowledge, skills, and other attributes via stealth assessment. Such learning environments use real-time estimates of learners' competency levels to adapt activities to a l...

Interactions with digital environments can yield huge amounts of electronic data—in terms of what was done, when, where, and how. What's known today as stealth assessment arose from a single question four decades ago: How can we mine digital data for meaningful information about a person's cognitive and noncognitive states, then use it to enhance l...

Stealth assessment is an innovative way to measure a set of student competencies through gameplay. The process starts with a competency model that is comprised of everything you want to measure during the assessment, the theoretical concepts being assessed. The competency model is the glue of the stealth assessment, as alignment of the task model a...

We present the design of CPSCoach, a fully-automated system that assesses and provides feedback on collaborative problem solving (CPS) competencies during remote collaborations. We leveraged existing data to develop deep NLP models that automatically assess the CPS competencies from speech, achieving moderate to high accuracies (average area under...

Research fields related to learning (e.g., educational technology and learning sciences) have historically focused on what questions using traditional methods (e.g., comparing different learning tools and methods). New methodologies that are grounded in learning, engagement, and motivational theories are needed to additionally address the how quest...

Over the past several decades, machine learning has been used in many areas—including education. In this article, we generally describe artificial intelligence (AI) and machine learning (ML) and discuss how they relate to one another using everyday examples. Next, we zoom in on ML in terms of its definition, some well-known approaches, its historic...

Learning does not automatically occur by playing educational games; instead, learning opportunities should be carefully designed in such games. For instance, research has indicated the importance of embedding learning supports within educational games to promote learning and other outcomes (e.g., enjoyment). However, more research is needed to dete...

Engagement is a strong predictor of learning in educational contexts, but the definition of engagement can vary from study to study, with small differences in definition leading to substantial differences in findings. In addition, students frequently employ strategies in online learning systems that the system designers may not have expected, which...

In this study, we use feature engineering and machine learning (ML) to develop a regression model to predict students' learning gain while playing an educational game. Specifically, this study used the log data from 199 students' gameplay to build a logistic regression model. The model we created includes 14 features with an accuracy of .61.

The current study investigated students' gameplay behavioral patterns as a function of in-game learning supports delivery timing when played a computer-based physics game. Our sample included 134 secondary students (M = 14.40, SD = .90) from all over the United States, who were randomly assigned into three conditions: receiving instructional videos...

Five types of affective supports were designed to induce an appropriate emotional regulation strategy in players of an educational video game. These supports were based on the emotional regulation strategies of situation selection, situation modification, attentional deployment, cognitive change, and response modulation. A series of qualitative stu...

The present study explores how gender, ethnicity, and performance-based perceived competence impact students’ learning, performance, and enjoyment from playing a digital STEM learning game. We had 199 9th-11th grade students play a 2D digital STEM learning game across six science classes. Based on the results of demographic surveys, matched pretest...

Learning analytics (LA) dashboards refer to digital tools designed to help learners keep track of their progress and goals. There is growing interest and research around the topic of LA dashboards in online learning environments, with many lessons to be learned by educational game developers and researchers. However, we need more research in this a...

Background and Context: The Inclusive Assessment of Computational Thinking (CT) designed for accessibility and learner variability was studied in over 50 classes in US schools (grades 3-8). Objective: The validation studies of IACT sampled thousands of students to establish IACT’s construct and concurrent validity as well as test-retest reliability...

Creativity is one of the most essential skills for success in life in our dynamic, complex world. For instance, we are currently facing major problems with the COVID-19 pandemic, which requires creative thinking for solutions. To increase the pool of creative thinkers, we need tools that can assess and support people's creativity. With advances in...

Collaborative problem solving (CPS) is an essential skill for the 21st century workforce, but remains difficult to assess. Understanding how CPS skills affect CPS performance outcomes can inform CPS training, task design, feedback design, and automated assessment. We investigated CPS behaviors (individually and in co-occurring patterns) in 101 (N =...

Educational video games are hypothesized to be good environments for promoting learning; however, research on conceptual learning from games is mixed. We tested whether embedding a learning support in the form of short animations illustrating physics concepts that can be used to aid gameplay improved learning. Ninety-six 7th to 11th grade students...

Over three years, our research team has designed various learning supports for promoting content knowledge and solving game levels. In this case study, we examined the optimal design and the evaluation of learning support videos for a physics educational game. Often studies focus on investigating the effects of research-based principles without a s...

In-game learning supports aim to help students solve game levels (i.e., game-related supports), and connect to underlying content (i.e., content-related and hybrid supports). Students with different levels of prior knowledge may have different needs for in-game supports. In this study, we designed a 2D physics game with game-related, content-relate...

Persistence is an important part of student success—both in and out of school. To enhance persistence, we first need to assess it accurately. Digital games can be used as vehicles for measuring and enhancing persistence. The purpose of this study is to test the effects of (a) game-level characteristics (i.e., game mechanics and conceptual difficult...

We examined the use and effectiveness of an incentive system—one of the five elements of a theory-based motivational architecture in educational games that we proposed—in a computer-based physics game on students’ learning and performance. The incentive system’s purpose was to motivate students to access learning supports designed to facilitate con...

To achieve inclusion of diverse professionals in STEM fields, educators must try to get more children, particularly underrepresented minorities, excited about STEM areas such as physics (Shute et al., 2020a). As researchers, we understand our educational responsibility in supporting active learning for a wide range of learners. In this study, we ex...

Creativity has been of research interest to psychologists dating back many decades and is currently recognized as one of the essential skills needed to succeed in our complex, interconnected world. One medium that has affordances to assess and support creativity in young people is video games. In this paper, we briefly discuss the literature on vid...

In this study, we investigated the validity of a stealth assessment of physics understanding in an educational game, as well as the effectiveness of different game‐level delivery methods and various in‐game supports on learning. Using a game called Physics Playground, we randomly assigned 263 ninth‐ to eleventh‐grade students into four groups: adap...

In honor of Jim Greer, we share our recent work—a design and development study of various learning supports embedded within the game Physics Playground. This 2-dimensional computer game is designed to help students learn Newtonian physics and uses stealth assessment to measure, in real-time, their physics understanding. The game operates according...

Collaborative problem solving (CPS) is a crucial 21st century skill; however, current technologies fall short of effectively supporting CPS processes, especially for remote, computer-enabled interactions. In order to develop next-generation computer-supported collaborative systems that enhance CPS processes and outcomes by monitoring and responding...

Well-designed digital games hold promise as effective learning environments. However, designing games that support both learning and engagement without disrupting flow [1] is quite tricky. In addition to including various game design features (e.g., interactive problem solving, adaptive challenges, and player control of gameplay) to engage players,...

We adopt a multimodal approach to investigating team interactions in the context of remote collaborative problem solving (CPS). Our goal is to understand multimodal patterns that emerge and their relation with collaborative outcomes. We measured speech rate, body movement, and galvanic skin response from 101 triads (303 participants) who used video...

Understanding why students quit a level in a learning game could inform the design of appropriate and timely interventions to keep students motivated to persevere. In this paper, we study student quitting behavior in Physics Playground (PP)-a Physics game for secondary school students. We focus on student cognition that can be inferred from their i...

Collaborative problem solving (CPS) is an essential skill in the 21st century. There is a need for an appropriate framework and operationalization of CPS to guide its assessment and support and across multiple domains. Accordingly, we synthesized prior research on CPS to construct a generalized CPS competency model (i.e., skills and abilities) cons...

Digital, educational games have many promises (e.g., increasing students’ content knowledge as well as competencies like problem solving, spatial skills, and persistence). However, there are challenges to overcome before using these games more broadly in educational settings. One challenge involves identifying effective, theoretically based learnin...

The following research focuses on Game-Based Learning (GBL) for assessment through the lens of designing and validating a stealth assessment for the calculus game Variant: LimitsTM. The process of using Evidence-Centered Design (ECD) to create a valid assessment is highlighted through the development of a competency model and scoring rubrics. A sam...

A common skepticism about educational games is that learning and play are frequently not well integrated—the skill or content to be used and learned lacks a semantic or meaningful relation with the fantasy and challenge elements and can be easily swapped without influencing gameplay. In this chapter, we describe and analyze design challenges associ...

There are two important design issues related to game-based learning (GBL) in school settings: (a) the intrinsic integration of content-related tasks in gameplay and (b) the real-time capture and analysis of in-game performance data. In this chapter, we describe an integrative design approach that is aimed to interweave game-based task design with...

Games are not just a vehicle to enhance learning but a new way of understanding and organizing learning. The starting point of an optimal integration of learning and play in the game setting is to identify the salient elements of learning itself and the inherent learning processes in gameplay. In this introductory chapter, we discuss relevant theor...

In this concluding chapter, we discuss an evolving, experiential design framework for game-based learning platforms, by synthesizing the salient design problem-solving events, experiences, and solution-exploration findings reported in the previous chapters. It is not aimed to be prescriptive or exhaustive, but acts as a starting point for specifyin...

Our phenomenological examination of learning game design is situated in a four-year, longitudinal design-based research project that encompasses iterative design processes to develop, refine, and study a game-based learning platform called E-Rebuild. This chapter presents an introductory overview of the four facets of the interdisciplinary educatio...

In this chapter, we provide a reflective and analytical description of the interdisciplinary design activities of E-Rebuild, identify driving design questions and salient design patterns that capture and frame the essence of E-Rebuild development, and discuss distilled meta-generalizations that help to decompose the interdisciplinary learning game...

The role of support for game-based learning cannot be overemphasized. It remains inconclusive as to what, when, and how support for learning should be designed and implemented to foster learners’ extended engagement, in-game performance, and game-based disciplinary knowledge learning and transfer. In this chapter, we review prevalent support featur...

According to Pearson’s Law: “That which is measured improves.” But some important constructs, like problem-solving skill, are generally viewed as very difficult to measure for a variety of reasons (e.g., lack of a clear and agreed-upon definition, psychological and/or statistical multidimensionality of the construct, subjectivity of scoring, and so...

Identifying struggling students in real-time provides a virtual learning environment with an opportunity to intervene meaningfully with supports aimed at improving student learning and engagement. In this paper, we present a detailed analysis of quit prediction modeling in students playing a learning game called Physics Playground. From the interac...

Digital games have evolved as an engaging medium for learning. This paper studies the interaction between game design and student affective experience. Data from 137 students playing a learning game was analyzed to identify the factors correlating with student frustration. Results suggest that in this well-designed game, difficulty associated with...

Immersive Environments (IEs) hold many promises for learning. They represent an active approach to learning and are intended to facilitate better, deeper learning of competencies relevant for success in today’s complex, interconnected world. To harness the power of these environments for educational purposes (i.e., to support learning), we need val...

Digital games are very popular in modern culture. We have been examining ways to leverage these engaging environments to assess and support important student competencies, especially those that are not optimally measured by traditional assessment formats. In this chapter, we describe a particular approach for assessing and supporting student learni...

In this chapter, we describe the importance of assessing and developing conscientiousness in students and how we are approaching this challenge. After discussing the benefits conscientiousness has for learning, we describe the process we are using to create a valid stealth assessment of conscientiousness. We then discuss the current state of this w...

This paper examines the growing field of computational thinking (CT) in education. A review of the relevant literature shows a diversity in definitions, interventions, assessments, and models. After synthesizing various approaches used to develop the construct in K-16 settings, we have created the following working definition of CT: The conceptual...

Lacking a digital crystal ball, we cannot predict the future of education or the precise instructional role games will have going forward. Yet we can safely say that games will play some role in the future of K-12 and higher education, and members of the games community will have to choose between being passive observers or active, progressive cont...

Educators are increasingly using games as a method for enabling engagement and learning in students, but research has suggested potentially inconsistent outcomes for the use of these digital tools. One explanation for these mixed findings may be different preferred playstyles of game players, such as Bartle's (1996) player taxonomies. This research...

In this chapter we discuss challenges with traditional assessment when researchers deal with hard-to-measure constructs such as systems-thinking, creativity, or persistence. We then describe how the frameworks of evidence-centered design and stealth assessment may be used to develop assessments that can be embedded in video games to make valid infe...

The educational system in the U.S. needs a "paradigm change" (Reigeluth & Karnopp, 2013) or an "educational sea change" (Shute, 2007). Schools should prepare students for the information age society in which we are living (Reigeluth & Karnopp, 2013; Shute, 2007). There are many lessons we, as educators and instructional designers, can learn from ga...

We used stealth assessment, embedded in a game called Use Your Brainz (a slightly modified version of Plants vs. Zombies 2), to measure middle-school students’ problem solving skills. We began by developing a problem solving competency model based on a review of the relevant literature. We then identified in-game indicators that would provide evide...

Affect detection is a key component in intelligent educational interfaces that respond to students' affective states. We use computer vision and machine-learning techniques to detect students' affect from facial expressions (primary channel) and gross body movements (secondary channel) during interactions with an educational physics game. We collec...

In this paper, we review computer‐based assessment for learning (CBAfL), in elementary and secondary education, as a viable way to merge instruction and assessment of students' developing proficiencies. We begin by contextualizing our topic relative to summative and formative assessment before presenting the current literature, which we categorized...

Designing, developing, and administering assessments has remained fairly unchanged across the past century. However, recent developments in instructional technology, learning science theory, and advances in the design of assessments necessitate a newfound perspective on assessment. The objective of the present paper is to review the topic of assess...

A number of assessment experts have advanced a vision of assessment in schools that relies on ongoing, performance-based, formative assessment. While there are many potential benefits of such a system, it also raises concerns about assessment quality. This chapter is a review of the current state of the evidence for psychometric properties undergir...

Many games offer players opportunities to express their creativity, from posting their clever solutions in online forums, to beating particular boss levels, to creating their own levels using built-in level editors. Yet there is scant evidence supporting the link between video games and creativity. This is partially due to the difficulty of measuri...

Game-based assessment (GBA) is a specific use of educational games that employs game activities to elicit evidence for educationally valuable skills and knowledge. While this approach can provide individualized and diagnostic information about students, the design and development of assessment mechanics for a GBA is a nontrivial task. In this artic...

This paper discusses multimodal affect detection from a fusion of facial expressions and interaction features derived from students' interactions with an educational game in the noisy real-world context of a computer-enabled classroom. Log data of students' interactions with the game and face videos from 133 students were recorded in a computer-ena...

Stealth assessment provides an innovative way to assess and ultimately support knowledge, skills, and other personal attributes within learning and gaming environments without disrupting students' flow. In this paper, the authors briefly discuss two challenges they encountered during the development of stealth assessments in two past projects (i.e....

This study investigated the relationships among incoming knowledge, persistence, affective states, in-game progress, and consequently learning outcomes for students using the game Physics Playground. We used structural equation modeling to examine these relations. We tested three models, obtaining a model with good fit to the data. We found evidenc...

Abstract Educators today are increasingly interested in using game-based assessment to assess and support students' learning. In the present study, we investigated how changing a game design element, linearity in gameplay sequences, influenced the effectiveness of game-based assessment in terms of validity, reliability, fairness, learning, and enjo...

Increased attention to the relationships between affect and learning has led to the development of machine-learned models that are able to identify students' affective states in computerized learning environments. Data for these affect detectors have been collected from multiple modalities including physical sensors, dialogue logs, and logs of stud...

The goal of this paper was to explore the possibility of generalizing face-based affect detectors across multiple days, a problem which plagues physiological-based affect detection. Videos of students playing an educational physics game were collected in a noisy computer-enabled classroom environment where students conversed with each other, moved...

Affect detection is a key component in developing intelligent educational interfaces that are capable of responding to the affective needs of students. In this paper, computer vision and machine learning techniques were used to detect students' affect as they used an educational game designed to teach fundamental principles of Newtonian physics. Da...

Physics Playground is an educational game that supports physics learning. It accepts multiple solutions to most problems and does not impose a stepwise progression through the content. Assessing student performance in an open-ended environment such as this is therefore challenging. This study investigates the relationships between student action se...

In this chapter, we describe the processes of designing and validating game-based learning assessment and/or support in two different games—Portal 2 (by Valve Corporation) and Earthquake Rebuild. The games represent cases of possible game-based learning (i.e., domain-generic and domain-specific) and provide good vehicles for testing the design deci...

The study explored instructional benefits of a storyline gaming characteristic (GC) on learning effectiveness, efficiency, and engagement with the use of an online instructional simulation for graduate students in an introductory statistics course. A storyline is a game-design element that connects scenes with the educational content. In order to e...