Content uploaded by Eric Klopfer
Author content
All content in this area was uploaded by Eric Klopfer on Nov 24, 2015
Content may be subject to copyright.
A preview of the PDF is not available
A Framework for Structuring Learning Assessment in a Massively Multiplayer Online Educational Game: Experiment Centered Design
International Journal of Game-Based Learning
Abstract and Figures
Educational games offer an opportunity to engage and inspire students to take interest in science, technology, engineering, and mathematical STEM subjects. Unobtrusive learning assessment techniques coupled with machine learning algorithms can be utilized to record students' in-game actions and formulate a model of the students' knowledge without interrupting the students' play. This paper introduces "Experiment Centered Assessment Design" XCD, a framework for structuring a learning assessment feedback loop. XCD builds on the "Evidence Centered Assessment Design" ECD approach, which uses tasks to elicit evidence about students and their learning. XCD defines every task as an experiment in the scientific method, where an experiment maps a test of factors to observable outcomes. This XCD framework was applied to prototype quests in a massively multiplayer online MMO educational game. Future work would build upon the XCD framework and use machine learning techniques to provide feedback to students, teachers, and researchers.
Figures - uploaded by Eric Klopfer
Author content
All figure content in this area was uploaded by Eric Klopfer
Content may be subject to copyright.
... • [Kam et al. 2007], [Kam et al. 2009]; • MMORPGs [Conrad et al. 2014]. ...
... • Crianças até 6 anos [Baron et al. 2014 • Crianças de 6 até 15 anos [Rowe et al. 2017], [Del Blanco et al. 2010], [Asbell-Clarke et al. 2013], [Harpstead et al. 2013], [Conrad et al. 2014], [Mota and Pimentel 2017], [Falakmasir et al. 2016], [Min et al. 2019], [Kam et al. 2007], [Kam et al. 2009 [Harpstead et al. 2013], [Mota and Pimentel 2017], [Falakmasir et al. 2016], [Kam et al. 2007], [Kam et al. 2009], [Conrad et al. 2014 [Georgiadis et al. 2018], [Georgiadis et al. 2019b], [Georgiadis et al. 2019a], [Georgiadis et al. 2019c], [Min et al. 2019]; ...
... • Crianças até 6 anos [Baron et al. 2014 • Crianças de 6 até 15 anos [Rowe et al. 2017], [Del Blanco et al. 2010], [Asbell-Clarke et al. 2013], [Harpstead et al. 2013], [Conrad et al. 2014], [Mota and Pimentel 2017], [Falakmasir et al. 2016], [Min et al. 2019], [Kam et al. 2007], [Kam et al. 2009 [Harpstead et al. 2013], [Mota and Pimentel 2017], [Falakmasir et al. 2016], [Kam et al. 2007], [Kam et al. 2009], [Conrad et al. 2014 [Georgiadis et al. 2018], [Georgiadis et al. 2019b], [Georgiadis et al. 2019a], [Georgiadis et al. 2019c], [Min et al. 2019]; ...
Avaliar o estudante é importante para entender como o processo de ensino e aprendizagem ocorre. Em jogos sérios, alguns métodos são utilizados para ajudar nessa avaliação, no entanto, poucos consideram abordagens pedagógicas humanistas. Na abordagem educacional sociointeracionista, são valorizados a criatividade, a interação e o contexto sócio-histórico-cultural do estudante, sendo assim, a avaliação de jogos sociointeracionistas distancia-se de práticas behavioristas. Essa revisão sistemática foi realizada no intuito de entender quais são os métodos, técnicas, critérios, frameworks e/ou algoritmos que podem ser utilizados para a avaliação de jogos sérios sociointeracionistas.
... This would encourage collaborative learning alongside the ordinary constructivistic learning which takes place during the single-player experience [11]. There has been a lot of literature analyzing the effect of having collaboration in games, as well as the resultant learning outcome [11], [12], [20], [21], [13]- [17], [17]- [19]. It must be mentioned that certain researchers have explored serious games in the context of 'learning labs'. ...
... This way, instructors have the role of guiding the learner rather than directly giving them information. While the topic has been one of tremendous interest in fields such as healthcare and medical [30], [49], [50], various pedagogical studies have examined learning pedagogy in engineering games in particular [5], [19], [54]- [58], [21], [41], [41], [43], [48], [51]- [53]. ...
A scalable and immersive game was developed to serve as a monthly concept review game for theory-heavy engineering courses (such as fluid dynamics or heat transfer). It was designed such that in-game items and content may be dynamically replaced easily with an Excel data table, without the need for further programming. It is expected that course instructors may use such a tool by simply updating an Excel data table to rapidly tailor the game for any course. Even room/zone designs may be parametrized using a data table. Given the automation level of such a tool, its scalability and adaptability, the results of this work demonstrate the an innovative tool for educators to use for rapid construction of fun learning environments for engineering students in fun, game worlds. Future multiplayer game development can aid with collaborative learning to an even higher degree.
... Game-based assessments are also frequently used in educational settings, and their design is typically well-informed by evidence-based assessment frameworks. This type of assessment can support learning objectives and outcomes (Conrad et al., 2014;Hamari et al., 2014;Kapp et al., 2019). For instance, Shute et al. (2013Shute et al. ( , 2015 have demonstrated how games, coupled with evidence-based embedded assessment, can validly assess hard-to-measure constructs in educational contexts such as persistence (Ventura and Shute, 2013;DiCerbo, 2014), problem-solving, and creativity (Kim and Shute, 2015). ...
Modern technologies have enabled the development of dynamic game- and simulation-based assessments to measure psychological constructs. This has highlighted their potential for supplementing other assessment modalities, such as self-report. This study describes the development, design, and preliminary validation of a simulation-based assessment methodology to measure psychological resilience—an important construct for multiple life domains. The design was guided by theories of resilience, and principles of evidence-centered design and stealth assessment. The system analyzed log files from a simulated task to derive individual trajectories in response to stressors. Using slope analyses, these trajectories were indicative of four types of responses to stressors: thriving, recovery, surviving, and succumbing. Using Machine Learning, the trajectories were predictive of self-reported resilience (Connor-Davidson Resilience Scale) with high accuracy, supporting construct validity of the simulation-based assessment. These findings add to the growing evidence supporting the utility of gamified assessment of psychological constructs. Importantly, these findings address theoretical debates about the construct of resilience, adding to its theory, supporting the combination of the “trait” and “process” approaches to its operationalization.
... They may include readings, pre-recorded videos, discussion areas, tasks and assessments. A new feature of these systems is the possibility of learning analytics to track learner engagement, including, not only traditional assessments and teacher gradebooks, but analyses based on incidental "data exhaust" including keystroke patterns, edit histories, clickstream and navigation paths, social interaction patterns (Cope and Kalantzis 2016) (Conrad, Clarke-Midura, and Klopfer 2014). Intelligent tutors work best in problem domains where highly structured progressions are possible, such as algebra or chemistry (Koedinger, Brunskill, Baker, and McLaughlin 2013). ...
This book explores a phenomenon we call "e-learning ecologies". We use this metaphor because a learning environment is in some senses like an ecosystem, consisting of the complex interaction of human, textual, discursive and spatial dynamics. These take a coherent, systemic form. Traditional classrooms, with their linear arrangement of seating and desks, their lecturing teachers, their textbooks, their student workbooks, their classroom discussions are also learning systems. Moving from one of these classrooms to another, the modes of interaction are familiar and predictable because they are so systematically patterned. After a while, they seem "normal". However, these are strange human artifacts that were not put together into this configuration until the nineteenth century. They quickly became universal and compulsory sites of socialization of mass-institutionalized education. In terms of the long arc of human history, it was not until our the time of about our great, great grandparents that we first encountered these modern educational systems. But will these institutional forms survive long into the twenty-first century? Is it time for them to be reformed? And if change is to come, what will be the role of new technologies of knowledge representation and communication in bringing about change? This book explores transformations in the patterns of pedagogy that accompany e-learning, or the use of computing devices to mediate or supplement the relationships between learners and teachers, to present and assess learnable content, to provide spaces where students do their work, and to mediate peer-to-peer interactions. Our thesis is this: e-learning ecologies may play a key part in the largest shift in the systems of modern education since their rise to dominance in the nineteenth century. Everything may change-configurations of space, learner-to-teacher and learner-to-learner relationships, the textual forms of knowledge to which learners are exposed, the kinds of knowledge artifacts that students create, and they way their outcomes of their learning are measured. Or, we may introduce a whole lot of technology into schools, and nothing changes in institutional or epistemic senses. Technology is pedagogically neutral. So our questions of e-learning ecologies becomes these: how can they be different?
... The Balanced Design framework provides a more concise framework with the instructional perspective of game design. A few serious games such as The Radix Endeavor, a multiplayer online game to improve STEM learning in higher schools, adopted this design framework in the design process [20]. ...
Article is devoted to the problem of usage of game technologies in educational practice. Main objective is to define opportunities and restrictions of usage of game technologies as means of improvement of active learning in higher education. Authors consider main barriers to further implementation of gamification in educational practice (barriers to adoption, barriers to design and development, barriers to sustainability; barriers to innovations). Authors analyzed the potential of game approaches and estimated possible risks of their usage in educational practice. They admit that soft-gamification, context-gamification and sandbox-gamification approaches are the most suitable for usage in active learning. Hard-gamification approach is the most risky and organizationally difficult because it demands additional technical providing. Authors encouraged to use integrated solutions of Evidence-Centered Design (ECD) approach for overcoming barriers by using gamification in education. ECD concept includes three main elements (Content Model, Evidence Model, Task Model). Case study from educational practice is presented. According to their own experience with this approach, authors have come to conclusion, that the combination of ECD elements really provide a framework for: 1) specifying the knowledge and skills to be explored; 2) the tasks that can engage students in regard to that knowledge and skills; 3) useful information (the data and evidence) and ways of its interpretation to make inferences about the students' aptitudes. Authors admit that ECD offer a powerful tool for improving the design and opportunities in learning games.
Applying new innovative learning methods in schools can strongly influence and reform them. Schools are vital organizations adequate under constant changes, reformations and developments. Teachers, students and parents, all contribute into these changes materializing educational envisions. We strongly questioned ourselves how these innovative educational approaches can influence and reform an urban and a rural school respectively. We selected an urban, Model Experimental School and a typical rural High School in Greek countryside. We applied almost the same innovative approaches to both schools inquiring gradually the out coming changes. In the case of the urban school new education methods were necessary to be applied in order to raise students’ interest, entangle them into educational practices that are not so font of and reduce school abandoning. On the other hand applying almost the same practices to an Experimental School has to do with its fundamentals and basic principles. Students attending a Model Experimental School were chosen after exams and are in general willing to take part into educational programs, excellence groups, experimental workshops, exchange mobilities and other innovative projects. Although, we founded that the rural school reformed rapidly into a highly developing school, achieving gradually some of its pronounced educational goals. Teachers’ and administration’s vision began by encouraging teachers to participate into educational meetings, conferences and developing courses. Additionally, groups of students were organized to prepare several projects according to their interests. We can mention the Astronomy group, the Environmental group and the Drama group. Accordingly, we invested on extroversion and presentation of projects. Astronomical and Environmental events and Drama performances were some activities that joined the rural school with the local society, parents and communities. Furthermore, students revealed a remarkable interest about sciences and culture. On the other hand at the Model Experimental School, all these activities regarded as standard activities, sometimes as obligations. Of course also this school had some remarkable educational programs, interesting experimental workshops and extroversion to the local community. What differentiates the Experimental School is its excellence groups, were teachers and students are approaching non typical subjects of Science, Art and Culture. These groups seem to be the comparative advantage of the urban school leading to extra-curriculum knowledge. In any case the urban school seems to develop and reform itself, but not as fast and crucial as the rural school.
How to personalize learners' learning with digital technology so that learners derive optimal experiences in learning is a key question facing learning scientists, cognitive psychologists, teachers, and professional instructional designers. One of the challenges surrounding personalization and digital technology is how to promote learners' cognitive processes at a deeper level so that they become optimally engaged in critical and creative thinking, making inferences in learning, transferring knowledge to new learning situations, and constructing new knowledge during innovative learning process. This chapter examines the literature relating to deep cognitive processes and the idiosyncratic features of digital technology that support learners' deep cognitive processes in learning. Guidelines pertaining to personalization with digital technology in regard to deep cognitive processing are proposed, followed by the discussions on future research with a focus on verifying the theoretical constructs proposed in the guidelines.
The purpose of this review is to investigate the trends in the body of research on machine learning in educational technologies, published between 2007 and 2017. The criteria for article selection were as follows: (1) study on machine learning in educational/learning technologies, (2) published between 2007–2017, (3) published in a peer-reviewed outlet, and (4) an empirical study, literature review, or meta-analysis. Eighty-nine articles were chosen, after the first round of the article selection process. Through a second look at the articles, fifteen articles that did not match the criteria were eliminated. After the close examination of the seventy-four articles, certain demographical and thematic trends emerged. The top contributors to the body of research were Taiwan and the United States while the most productive year was 2017. The most utilized machine learning methods were vectors and decision trees. Commonly researched areas, on the other hand, were automation, cognitive process assessment, prediction, intelligent tutoring systems, and opportunities and challenges in the use of big data & learning analytics. Recommendations for future research focus on expanding geographical diversity, incorporating Bayesian and fuzzy logic methods more in educational machine learning work.
Educational assessment has evolved over the past several years from traditional pen and paper-based tests to the use of technology (such as games) and continues to evolve. The assessments must provide feedback to learners and diagnostic information to teachers. Game-based learning offers an interactive environment for the students to learn in a fun and challenging way while keeping them engaged in the learning process. Game-based assessment (GBA) offers a way to assess them in this setting while they are interacting with a game. GBA may be composed of built-in quizzes and surveys to assess the student learning while they are playing. However, such methods tend to distract their attention from learning to complete the assessment. Stealth assessment is a way to assess the learners while they are playing an educational video game without breaking their flow. The future of GBA will be made up of a content-agnostic stealth assessment with a model of student’s knowledge built into it. The student model will help to adapt the game-play and accommodate the game to an individual learner. Content-agnostic game engineering (CAGE) is a framework that helps provide multiple learning contents within a single game to achieve content-agnostic assessment. Finally, adding a student model which makes the game and learning adapt to an individual student driven by their pace and performance while learning in the game is the need of the hour.
In the past we have referred to games as good " learning engines. " Here we argue that games are good learning engines because they are first good assessment engines, and that games require the kind of thinking that we need in the 21 st Century. They use actual learning as the basis for assessment. They test not only current knowledge and skills, but also preparation for future learning. They measure 21st Century skills like collaboration, innovation, production, and design by tracking many different kinds of information about a student, over time. Thus we suggest that the road to better schools starts by making the tests in school more like the games that students are already playing out of school.
An approach to performance-based assessments that embeds assessments in digital games in order to measure how students are progressing toward targeted goals.
To succeed in today's interconnected and complex world, workers need to be able to think systemically, creatively, and critically. Equipping K-16 students with these twenty-first-century competencies requires new thinking not only about what should be taught in school but also about how to develop valid assessments to measure and support these competencies. In Stealth Assessment, Valerie Shute and Matthew Ventura investigate an approach that embeds performance-based assessments in digital games. They argue that using well-designed games as vehicles to assess and support learning will help combat students' growing disengagement from school, provide dynamic and ongoing measures of learning processes and outcomes, and offer students opportunities to apply such complex competencies as creativity, problem solving, persistence, and collaboration. Embedding assessments within games provides a way to monitor players' progress toward targeted competencies and to use that information to support learning.
Shute and Ventura discuss problems with such traditional assessment methods as multiple-choice questions, review evidence relating to digital games and learning, and illustrate the stealth-assessment approach with a set of assessments they are developing and embedding in the digital game Newton's Playground. These stealth assessments are intended to measure levels of creativity, persistence, and conceptual understanding of Newtonian physics during game play. Finally, they consider future research directions related to stealth assessment in education.
The first challenge of accomplishing the goals of any successful instruc-tional system depends on accurately assessing learners and leveraging the information to improve learning (e.g., Conati, 2002; Park & Lee, 2003; Shute, Lajoie, & Gluck, 2000; Snow, 1994). This paper describes an app-roach for modeling key competencies and developing valid assessments embedded within an immersive game. Specifically, we describe theoretical-ly-based research relating to stealth assessment, diagnosis, and instructional decisions, operational within an immersive game environment. Stealth assessment and diagnosis occur during the learning (playing) process, and instructional decisions are based on inferences of learners' current and pro-jected competency states. Can games be used to support meaningful learning? Most likely the answer is yes, conditional on more research being conducted in this area. In general, we believe that (a) learning is at its best when it is active, goal-oriented, contextual-ized, and interesting (e.g., Bransford, Brown, & Cocking, 2000; Bruner, 1961; Quinn, 2005; Vygotsky, 1978); and (b) learning environments should thus be interactive, provide ongoing feedback, grab and sustain attention, and have appro-priate and adaptive levels of challenge—i.e., the features of good games (e.g., Prensky, 2001; Salen & Zimmerman, 2004).
We present work toward automatically assessing and estimating science inquiry skills as middle school students engage in inquiry
within a physical science microworld. Towards accomplishing this goal, we generated machine-learned models that can detect
when students test their articulated hypotheses, design controlled experiments, and engage in planning behaviors using two
inquiry support tools. Models were trained using labels generated through a new method of manually hand-coding log files,
“text replay tagging”. This approach led to detectors that can automatically and accurately identify these inquiry skills
under student-level cross-validation. The resulting detectors can be applied at run-time to drive scaffolding intervention.
They can also be leveraged to automatically score all practice attempts, rather than hand-classifying them, and build models
of latent skill proficiency. As part of this work, we also compared two approaches for doing so, Bayesian Knowledge-Tracing
and an averaging approach that assumes static inquiry skill level. These approaches were compared on their efficacy at predicting
skill before a student engages in an inquiry activity, predicting performance on a paper-style multiple choice test of inquiry,
and predicting performance on a transfer task requiring data collection skills. Overall, we found that both approaches were
effective at estimating student skills within the environment. Additionally, the models’ skill estimates were significant
predictors of the two types of inquiry transfer tests.
KeywordsScientific inquiry–Exploratory learning environment assessment–Skill prediction–Machine-learned models–Microworlds–Behavior detection–Designing and conducting experiments–Bayesian Knowledge-Tracing
In today’s increasingly “flat” world of globalization (Friedman 2005), the need for a scientifically literate citizenry has grown more urgent. Yet, by some measures, we have done a poor job
at fostering scientific habits of mind in schools. Recent research on informal games-based learning indicates that such technologies and the communities they evoke
may be one viable alternative—not as a substitute for teachers and classrooms, but as an alternative to textbooks and science
labs. This paper presents empirical evidence about the potential of games for fostering scientific habits of mind. In particular,
we examine the scientific habits of mind and dispositions that characterize online discussion forums of the massively multiplayer
online game World of Warcraft. Eighty-six percent of the forum discussions were posts engaged in “social knowledge construction” rather than social banter.
Over half of the posts evidenced systems based reasoning, one in ten evidenced model-based reasoning, and 65% displayed an
evaluative epistemology in which knowledge is treated as an open-ended process of evaluation and argument.
In recent years, models of student inquiry skill have been developed for relatively tightly-scaffolded science simulations. However, there is an increased interest in researching how video games and virtual environments can be used for both learning and assessment of science inquiry skills and practices. Such environments allow students to explore scientific content in a more open-ended context that is designed around actions and choices. In such an environment, students move an avatar around a world, speak to in-game characters, obtain objects, and take those objects to laboratories to run specific tests. While these environments allow for more autonomy and choice, assessing skills in these environments is a more difficult challenge than in closed environments or simulations. In this paper, we present models that can infer two aspects of middle-school students’ inquiry skill, from their interactive behaviors within an assessment in a virtual environment called a “virtual performance assessment” or VPA: 1) whether the student successfully demonstrates the skill of designing controlled experiments within the VPA, and 2) whether a middle-school student can successfully use their inquiry skill to determine the answer to a scientific question with a non-intuitive in-game answer.
Causal reasoning is difficult for middle school students to grasp. In this research, we wanted to test the possibility of using machine learning for modeling students’ causal reasoning in a virtual environment designed to assess this skill. Our findings suggest it is possible to use machine learning to emulate student pathways that are able to predict their causal understanding.
Evidence-centered assessment design (ECD) is an approach to constructing educational assessments in terms of evidentiary arguments. This paper provides an introduction to the basic ideas of ECD, including some of the terminology and models that have been developed to implement the approach. In particular, it presents the high-level models of the Conceptual Assessment Framework and the Four-Process Architecture for assessment delivery systems. Special attention is given to the roles of probability-based reasoning in accumulating evidence across task performances, in terms of belief about unobservable variables that characterize the knowledge, skills, and/or abilities of students. This is the role traditionally associated with psychometric models, such as those of item response theory and latent class models. To unify the ideas and to provide a foundation for extending probability-based reasoning in assessment applications more broadly, however, a more general expression in terms of graphical models is indicated. This brief overview of ECD provides the reader with a feel for where and how graphical models fit into the larger enterprise of educational and psychological assessment. A simple example based on familiar large-scale standardized tests such as the GRE is used to fix ideas. The document contains two appendices: (1) further reading about the ECD Project; and (2) a glossary of evidence-centered design terms.