Diego Zapata-Rivera

• Ph. D.
• Distinguished Presidential Appointee at ETS Research Institute

Learner models maintain representations of students’ cognitive, metacognitive, affective, personality, social and perceptual skills. This information can be used to adapt the adaptive instructional system’s interactions with the student. Our work on caring assessments has provided us with an opportunity to explore learner modelling issues applied t...

Research in the area of Open Student Models (OSMs) has shown that external representations of the student model can be used to facilitate educational processes such as student reflection, knowledge awareness, learning, collaboration, negotiation, and student model diagnosis. OSMs can be integrated into existing learning systems or become a framewor...

Learner models are representations of the learner’s knowledge, skills and other attributes used by Adaptive Instructional Systems (AISs) to personalize their interactions with the learners (e.g., by implementing adaptive feedback, and recommending tasks/activities). Top-down and bottom-up approaches to learner modeling provide various affordances a...

Conversation-based assessments (CBAs) build on advances in areas such as dialog systems, performance-based assessments, and adaptive, technology rich learning and assessment environments. CBAs have the potential to engage students in meaningful exchanges of information that can result in additional insights on the cognitive processes and strategies...

Conversation-based assessments (CBAs) make use of dialogue systems to create situations that can be used to gather evidence of the decisions learners make on performance-based assessments such as computer‐based scenarios, simulations, or other forms of technology rich tasks. CBAs are adaptive in nature providing subsequent questions, feedback and e...

Large Language Model (LLM) agents are increasingly used in AI-supported education. Effective inter-agent communication boosts collaborative problem-solving efficiency and lowers the cost of deploying LLM-driven educational systems. However, few studies have systematically examined how different communication strategies affect on agents' problem-sol...

Learner performance data collected by Intelligent Tutoring Systems (ITSs), such as responses to questions, is essential for modeling and predicting learners' knowledge states. However, missing responses due to skips or incomplete attempts create data sparsity, challenging accurate assessment and personalized instruction. To address this, we propose...

The advancement of multi-agent workflows leveraging Large Language Models (LLMs) for tackling complex tasks, such as mathematical problem-solving, has garnered significant interest among researchers. However, the communication strategies within these workflows remain underexplored, hindering comprehensive performance comparisons. This gap is partic...

n the field of education, there is a growing interest in the use of Generative Artificial Intelligence (Generative AI) to reshape the educational landscape. This Research Topic investigates the transformative potential of Generative AI in various aspects of education. The papers in this edited volume shed light on the latest discoveries, new insigh...

In recent years, large language models (LLMs) have seen rapid advancement and adoption, and are increasingly being used in educational contexts. In this perspective article, we explore the open challenge of leveraging LLMs to create personalized learning environments that support the “whole learner” by modeling and adapting to both cognitive and no...

In line with the positive effects of personalized learning, personalized assessments are expected to maximize learner motivation and engagement, allowing learners to show what they truly know and can do. Considering the advances in Generative Artificial Intelligence (GenAI), in this perspective article, we elaborate on the opportunities of integrat...

Learner models are used to support the implementation of personalization features in Adaptive Instructional Systems (AISs; e.g., adaptive sequencing of activities, adaptive feedback), which are important aspects of Intelligent Adaptive Systems. With the increased computational power, more advanced methodologies, and more available data, learner mod...

Caring assessments is an assessment design framework that considers the learner as a whole and can be used to design assessment opportunities that learners find engaging and appropriate for demonstrating what they know and can do. This framework considers learners’ cognitive, meta-cognitive, intra-and inter-personal skills, aspects of the learning...

The field of score reporting continues to evolve because of new challenges, opportunities, and needs of society (e.g., COVID, remote teaching and learning). In this paper, the new challenges and opportunities in score reporting are discussed from the personal perspective of four experts who have previously conducted research in designing score repo...

Learning analytic dashboards (LADs) are data visualization systems that use dynamic data in digital learning environments to provide students, teachers, and administrators with a wealth of information about student’s engagement, experiences, and performance on tasks. LADs have become increasingly popular, particularly in formative learning contexts...

Substantial progress has been made toward applying technology enhanced conversation‐based assessments (CBAs) to measure the English‐language proficiency of English learners (ELs). CBAs are conversation‐based systems that use conversations among computer‐animated agents and a test taker. We expanded the design and capability of prior conversation‐ba...

Jim Greer and his colleagues argued that student modelling is essential to provide adaptive instruction in tutoring systems and showed that effective modelling is possible, despite being enormously challenging. Student modelling plays a prominent role in many intelligent tutoring systems (ITSs) that address problem-solving domains. However, conside...

Many adult workers need to keep up with advances in technology to remain relevant in the job market. Adults now need 21 st century skills including Computational Thinking. It is challenging for adults to find training opportunities that take into account their limited time, educational, and resource constraints. Our approach provides support to adu...

The notion of intelligent systems that “care” about students is at the center of ITS research. Caring assessment systems are defined as systems that provide students with a positive assessment experience while improving the quality of evidence collected about the student’s knowledge, skills and abilities (KSAs). This workshop provides a great oppor...

Assessments can be a challenging experience for students. Students often have to consider more than just the knowledge being assessed, such as how to manage emotions that can impede performance (e.g., anxiety). But what if assessments cared about students and allowed them to just focus on the content of the assessment? In the present paper, we prop...

Providing relevant information about students to various educational stakeholders is one of the main functions of an Adaptive Instructional System (AIS). External representations of the student model are used to communicate information to students, teachers, and other educational stakeholders. AISs can provide a variety of information including cog...

This study investigated a new conversation-based assessment for English language learners. In this assessment, students converse with animated agents in natural language conversations to assess various aspects of English language learning. In a between subjects design, 31 students (N = 31) were asked questions by either the newly created system or...

An intelligent agent can play a significant role in interactive learning, assessment, and teamwork (Bay-lor, 2011; Johnson, Phillips, & Chase, 2009; Chou, Chan, & Lin, 2003; Johnson & Lester, 2016; Kumar, Ai, Beuth, & Rosé, 2010; Moreno, Mayer, Spires & Lester., 2001; Schroeder, Adesope, & Gilbert, 2013). Like Barrón-Estrade, Zatarain-Cabada, Orama...

Description. Score reporting research is no longer limited to the psychometric properties of scores and subscores. Today, it encompasses design and evaluation for particular audiences, appropriate use of assessment outcomes, the utility and cognitive affordances of graphical representations, interactive report systems, and more. By studying how a...

Pedagogical agents are widely employed in intelligent tutoring systems and game-based learning environments, but research suggests that learning benefits from virtual agents vary as a function of agent features (e.g., the form or register of agent dialogue) and student characteristics (e.g., prior knowledge). Students’ responses to agent-based conv...

The effective communication of assessment results to the intended audience is an important issue that has implications for accomplishing the goals of an assessment. New assessments can provide score report users with a variety of additional evidence about the test-taker’s knowledge, skills, and abilities, than has been possible with traditional ass...

The degree to which diverse score users can understand and consume information presented in score reports dictates whether they are able to draw reasonable conclusions. The goal of this study was to understand how parents (a particularly understudied and heterogeneous stakeholder population) make interpretations from a hypothetical interactive scor...

Students can experience a variety of emotions while completing assessments. Some emotions can get in the way of students performing their best (e.g., anxiety, frustration), whereas other emotions can facilitate student performance (e.g., engagement). Many new, non-traditional assessments, such as automated conversation-based assessments (CBA), are...

Most existing software tools for online collaboration are designed to support the collaboration itself instead of the study of collaboration with a systematic team and task management system. In this report, we identify six important features for a platform to facilitate the study of online collaboration. We then introduce the Educational Testing S...

Introduction Assessment design methodologies such as evidence-centered design (ECD) provide an approach for representing the " argument " underlying an assessment. Argument-based structures articulate the chain of reasoning connecting task-level data to the evidence required to support assessment claims about the student. Intelligent tutoring syste...

The notion of intelligent systems that "care" is at the center of research in areas such as Intelligent Tutoring Systems and Adaptive Systems. This paper elaborates on the notion of caring assessment systems, and presents work towards achieving this vision that has potential for improving students' assessment experiences.

Teachers often have difficulties understanding many aspects of score reports for assessments, thus hindering their ability to help students. Computerized environments with natural language conversations may help teachers better understand these reports. Thus, we created a tutor on score reports for teachers based on the AutoTutor conversational fra...

In this article, we discuss the use of prototype formative conversation-based assessments designed to measure English learners' language skills. Conversation-based assessments are technology enhanced assessment systems that simulate interactions between a test taker and one or more virtual agents. We discuss preliminary findings from two studies ex...

Innovative, interactive tasks that include conversations among humans and virtual (pedagogical) agents can be used to assess relevant cognitive skills (e.g., scientific inquiry skills). These new assessment systems aid the collection of additional information (e.g., timing data, information about conversation path sequences, and amount of help used...

Introduction Advancements in technology have led to a revolution in assessments. No longer limited to the bubble-and-booklet approach of the 20 th century, today's assessments may involve rich, interactive exercises, assess new and complex constructs, and automatically record and score evidence of skills. Still, the inferences drawn from technology...

Two trending areas in learning technology are intelligent tutoring systems (ITSs) that can create truly adaptive instruction for individual learners and teams and increased data interoperability through use of the experience application programming interface (xAPI). xAPI, a set of application programming interfaces (APIs), allows a common data stru...

Conversation-based assessments (CBAs) represents a new type of performance-based assessment where students engage in an adaptive dialogue exchange with a computer with one or more virtual agents. This assessment approach leverages the open response format of human-to-human conversations – similar to constructed response items – in order to allow st...

The goal of this study was to explore the effectiveness of a short web-based tutorial in helping teachers to better understand the portrayal of measurement error in test score reports. As described below, the short video tutorial included both verbal and graphical representations of measurement error. Results showed a significant difference in comp...

Adaptive learning environments and technology-rich assessments capture evidence of students' skills, knowledge, and other attributes and use it to adapt their interaction or support assessment claims. Data captured to support assessment claims or implement adaptive behavior can include responses to predefined questions and process data. However, st...

The purpose of our project is to explore the measurement of cognitive skills in the domain of science through collaborative problem solving tasks, measure the collaborative skills, and gauge the potential feasibility of using game-like environments with avatar representation for the purposes of assessing the relevant skills. We are comparing studen...

We explored the use of technology-assisted, trialogue-based tasks to measure the English language proficiency of students learning English as a second or foreign language. A presumed benefit of the system for language assessment is its suitability for use in scenario-based tasks that integrate multiple language skills. This integration allows test...

Simulation or game-based assessments produce outcome data and process data. In this article, some statistical models that can potentially be used to analyze data from simulation or game-based assessments are introduced. Specifically, cognitive diagnostic models that can be used to estimate latent skills from outcome data so as to scale these assess...

Score reports have one or more intended audiences: the people who use the reports to make decisions about test takers, including teachers, administrators, parents and test takers. Attention to audience when designing a score report supports assessment validity by increasing the likelihood that score users will interpret and use assessment results a...

Despite the emphasis on argumentation skills in the Common Core State State standards (CCSSO & NGA, 2010) and the importance of reasoning skills for many academic and professional domains (Graff, 2003), many students currently lack argumentation skills, and such skills have received little attention in educational game design. In response, we are d...

Trialogue-based tasks can be used to gather evidence that may be difficult to obtain using traditional assessment approaches, such as embedded questions. However, more research needs to be done in order to create valid, fair, and reliable conversation tasks that can be used for assessment purposes. This paper describes ongoing efforts at developing...

Research has shown that many educators do not understand the terminology or displays used in test score reports and that measurement error is a particularly challenging concept.We investigated graphical and verbal methods of representing measurement error associated with individual student scores. We created four alternative score reports, each con...

Every year in the United States, millions of score reports are produced and delivered to teachers, students, parents, school administrators, and policymakers. Research shows that many educators have trouble understanding and making appropriate use of score reports. This paper evaluates three graphical representations of test score distribution. Res...

Traditional science assessments, such as multiple-choice tests, have been criticized as too limited to appropriately evaluate students' knowledge and skill in complex scientific reasoning. This paper explores the potential of a game-like assessment for improving measurement of middle-school students' science inquiry skills. The assessment, in which...

Traditional science assessments, such as multiple-choice tests, have been criticized as too limited to appropriately evaluate students' knowledge and skill in complex scientific reasoning. This paper explores the potential of a game-like assessment for improving measurement of middle-school students' science inquiry skills. The assessment, in which...

This paper introduces the idea of adaptive score reports that can be used to provide educational stakeholders with a personalized experience aimed at facilitating student understanding and use of assessment information. These reports can also provide additional learning opportunities for users based on assessment results. An interactive score repor...

Adaptive educational systems monitor important learner characteristics and make appropriate adjustments to the instructional milieu to support and enhance learning. The goal of adaptive educational systems, in the context of this chapter, is to create an instructionally sound and flexible environment that supports learning for students with a range...

In an effort to make students active participants in their learning and to consider students’ score reporting needs, we have designed an interactive student score report for middle school students that implements a guided-instructional activity aimed at facilitating student understanding of score report information and improving student engagement....

Every year in the United States, millions of score reports are produced and delivered to teachers, students, parents, school administrators, and policymakers. However, existing research shows that many educators have trouble understanding and making appropriate use of score reports. We have developed and evaluated several teacher score reports (i.e...

This volume includes 3 papers based on presentations at a workshop on communicating assessment information to particular audiences, held at Educational Testing Service (ETS) on November 4th, 2010, to explore some issues that influence score reports and new advances that contribute to the effectiveness of these reports. Jessica Hullman, Rebecca Rhod...

This paper describes the characteristics of games and how they can be applied to the design of innovative assessment tasks for formative and summative purposes. Examples of current educational games and game-like assessment tasks in mathematics, science, and English language learning are used to illustrate some of these concepts. We argue that...

This volume includes 3 papers based on presentations at a workshop on communicating assessment information to particular audiences, held at Educational Testing Service (ETS) on November 4th, 2010, to explore some issues that influence score reports and new advances that contribute to the effectiveness of these reports. Jessica Hullman, Rebecca Rhod...

Score reports for administrators are often presented in ways that are not easily interpretable by these stakeholders. In order to design score reports for this audience, it is important to understand how policy is implemented at the local level and how administrators make use of achievement data to inform their decisions. We have reviewed existing...

Integrating sound assessment and instruction has been a long-sought goal in education. Recent advances in educational measurement, cognitive science, and artificial intelligence have placed this goal within our reach. Intelligent systems may be improved by leveraging assessment information that is gathered from various sources (e.g., summative and...

This article describes the development and evaluation of a computer-assisted language learning approach which integrates a finite state dialogue engine with an animated pedagogical agent. The design of the request game is theoretically motivated by interlanguage pragmatics and Long's Interaction Hypothesis. The tutoring system creates a venue in wh...

Assessment-based educational games can produce useful information to guide student instruction. This paper describes an approach for integrating components of video games with those of adaptive technologies and assessment into the design of educational games. Three examples in the areas of English language learning and mathematics are also presente...

District-level policymakers are challenged to use evidence of student achievement to make policy decisions, such as professional development and other school improvement plans. They currently receive reports of student achievement data that are complex, difficult to read, and even harder to interpret. Using the research literature on policymakers’...

This paper describes and demonstrates a process by which research-based score reports can be designed and evaluated. This work has been carried out as part of the Cognitively Based Assessments of, for, and as Learning (CBAL) project. Individual and classroom level CBAL reading score reports for teachers were designed and evaluated using this approa...

Peer review activities are almost ideal for supporting writing instruction. However, they also have a glaring weakness: quality of peer feedback. An obvious way to increase the quality of peer feedback is to improve students' reviewing skill. For peer review activities, students must obviously receive instruction in how to review classmates' essays...

Bayesian Belief Networks (BBNs) are increasingly used for understanding and simulating computational models in many domains. Though BBN techniques are elegant ways of capturing uncertainties, knowledge engineering effort required to create and initialize the network has prevented many researchers from using them. Even though the structure of the ne...

Purpose The purpose of this paper is to propose and demonstrate an evidence‐based scenario design framework for assessment‐based computer games. Design/methodology/approach The evidence‐based scenario design framework is presented and demonstrated by using BELLA, a new assessment‐based gaming environment aimed at supporting student learning of voc...

Students explore contextually and socially appropriate request strategies while the system scores each attempt, assigns points, and provides formative and summative feedback. This paper presents (a) related research on instruments that have been used in the area of interlanguage pragmatics (i.e., by request), (b) our assessment-based gaming environ...

Teachers viewing Bayesian network-based proficiency estimates from a classroom full of students face a different problem from a tutor looking at one student at a time. Fortunately, individual proficiency estimates can be aggregated into classroom and other group estimates through sums and averages. This paper explores a few graphical representation...

Assessment-based learning environments (ABLEs) make use of assessment information coming from a variety of sources (e.g., formative and summative assessments) to guide instruction. We have developed English ABLE, an assessment-based learning environment designed to help English language learners (ELLs) learn about English grammar. Key features of E...

Immersive games tend to induce "flow," a state in which a game player loses track of time and is absorbed in the experience of game play. Flow is conducive to engagement, and engagement is conducive to learning, yet immersive games lack the assessment infrastructure to maximize learning potential. Typical assessments are likely to disrupt flow in i...

This paper illustrates how Evidence Centered Design (ECD) can be used to design and develop high-quality learning-centered assessments that have qualities such as: (1) learning-effectiveness and efficiency, (2) validity of assessment results, (3) accessibility for students with special needs, and (4) student engagement. This paper draws on Cognitiv...

This paper illustrates how Evidence Centered Design (ECD) can be used to address the accessibility of learning-centered assessments, and how such efforts lay a critical foundation for improvements in other aspects of quality, particularly learning efficiency. Specifically, it illustrates how the same basic strategies used to ensure accessibility ca...

www.ets.org/legal/index.html.

Abstract This paper describes an approach to analyzing the educational potential of existing games and adapting existing games for educational purposes that makes use of Evidence Centered Design (ECD) principles and Cognitive Task Analysis (CTA) techniques. The main outcome,of this approach is the development of arguments for the quality of educati...

New demands associated with living in a highly-technological and globally-competitive world require today’s students to develop a very different set of competencies than previous generations of students needed. The general goal of education is to prepare young people to live independent and productive lives. Unfortunately, our current educational s...

Educational systems (in the U.S. and around the world) face huge challenges that require bold and creative solutions to prepare students for success in the 21st century. Part of the solution will require a new focus on students developing the ability to solve complex problems in innovative ways, as well as the ability to think clearly about systems...

To reveal what is being learned during the gaming experience, this report proposes an approach for embedding assessments in immersive games, drawing on recent advances in assessment design. Key to this approach are formative assessment to guide instructional experiences and evidence-centered design to systematically analyze the assessment argument...

Abstract Recent advances in educational assessment, cognitive science, and artificial intelligence have made it possible to integrate valid assessment and instruction in the formof modern,computer- based intelligent systems. These intelligent systems leverage assessment information that is gathered from various sources (e.g., summative and formativ...

This chapter describes a new idea for the design and development of assessments for mental models using “flexible belief networks” (FBNs). The idea involves joining and extending two assessment approaches—evidence-centered design (ECD) and concept mapping (CM). ECD will be extended beyond single, static proficiency models to dynamic models of learn...

One of the main problems facing U.S. education is the growing educational achievement gap (Kirsch, Braun, Yamamoto, & Sum, 2007). Snow and Biancarosa (2003) have argued that this gap is largely due to differential language proficiencies. Although there is a strong interest in helping English language learners achieve their language-learning goals,...

This paper defines Bayesian network models and examines their applications to IRT-based cognitive diagnostic modeling. These models are especially suited to building inference engines designed to be synchronous with the finer grained student models that arise in skills diagnostic assessment. Aspects of the theory and use of Bayesian network models...

Integrating sound assessment and instruction has been a long-sought goal in education. Recent advances in educational measurement, cognitive science, and artificial intelligence have placed this goal within our reach. Intelligent systems may be improved by leveraging assessment information that is gathered from various sources (e.g., summative and...

This paper describes research and development efforts related to adaptive technologies, which can be combined with other technologies and processes to form an adaptive system. The goal of an adaptive system, in the context of this paper, is to create an instructionally sound and flexible environment that supports learning for students with a range...

Research efforts focused on developing "active reports" are currently underway. Active reports are designed to foster communication among teachers, students, and parents by listening to all stakeholders, using assessment information to guide teaching and learning, and to reconcile potential conflicts. Open student models can handle different views...

Assessment-Based Learning Environments (ABLE) make use of assessment information coming from a variety of sources (e.g., formative and summative) to guide instruction. We have developed English ABLE, an assessment based learning environment designed to help English language learners (ELLs) learn about English grammar. Main features of English ABLE...

Inspectable Bayesian student models have been used to support student re∞ection, knowledge awareness and communication among teacher, students and parents. This paper presents a new approach to interact- ing with inspectable Bayesian student mod- els called indirectly visible Bayesian student models. In this approach, the student model is seen thro...

Bayesian belief networks (BBNs) are increasingly used for understanding and simulating computational models in many domains. Though BBN techniques are elegant ways of capturing uncertainties, knowledge engineering effort required to create and initialize the network has prevented many researchers from using them. Even though the structure of the ne...

Bayesian belief networks (BBNs) are increasingly used for understanding and simulating computational models in many domains. Though BBN techniques are elegant ways of capturing uncertainties, knowledge engineering effort required to create and initialize the network has prevented many researchers from using them. Even though the structure of the ne...

We present the results of research about teachers' use and understanding of assessment data, focusing on (a) questions they want answered, (b) their use of an instructional data management system, and (c) their reactions to a more teacher-friendly interface to the system that is designed to facilitate instruction.

The 12th International Conference on Artificial Intelligence in Education (AIED-2005) is being held July 18--22, 2005, in Amsterdam, the beautiful Dutch city near the sea. AIED-2005 is the latest in an on-going series of biennial conferences in AIED ...

User modelling shells and learner modelling servers have been proposed in order to provide reusable user/student model information over different domains, common inference mechanisms, and mechanisms to handle consistency of beliefs from different sources. Open and inspectable student models have been investigated by several authors as a means to pr...

Inspectable student models focus on the idea of letting students and teachers interact with the representation of the student that the system maintains. Both humans and the system can benefit from this interaction. By externalizing the student model and making it an object for inspection, several representational and interaction issues arise. This...