Rod Roscoe

• PhD, Cognitive Psychology
• Professor (Associate) at Arizona State University

Demographic data pertain to people’s identities and behaviors. Analyses of demographic data are used to describe patterns and predict behaviors, to inform interface design, and even institutional decision-making processes. Demographic data thus need to be complete and correct to ensure they can be analyzed in ways that reflect reality. This study c...

The artificial intelligence in education (AIED) community has produced technologies that are widely used to support learning, teaching, assessment, and administration. This work has successfully enhanced test scores, course grades, skill acquisition, comprehension, engagement, and related outcomes. However, the prevailing approach to adaptive and p...

This panel will focus on the emerging area of Learning Engineering. Learning Engineering is a transdisciplinary area focusing on the systematic application of evidence-based principles from science of learning disciplines to create effective learning experiences, addressing the challenges of learners. During the panel, examples of Learning Engineer...

Human factors focus on taking the users’ capabilities, limitations, and environment into consideration when developing products. Thus, it is essential to have diverse perspectives and voices when designing products to be used by a variety of users. However, this is not always done and can be a missed opportunity in developing inclusive products. In...

This paper seeks to bring further attention to transgender experiences and issues in the context of software applications. Research on transgender experiences has generally been quite limited, and the majority of that work has focused on non-digital environments. However, we argue that transgender individuals encounter several unique human-computer...

Researchers and educators have explored a variety of approaches for addressing diversity, equity, inclusion, and belonging (DEIB) challenges in engineering and design. This research builds on recommendations to teach future engineers and designers about DEIB principles and applications, and to challenge the dissociation of engineering and societal...

Writing is essential for success in academics and everyday tasks, but the development of writing skills depends on consistent access to high-quality instruction, extended practice, and personalized feedback. To address these demands and meet students’ needs, educators and researchers have turned to technology-based writing tools. Ideally, these too...

Literacy assessment is essential for effective literacy instruction and training. However, traditional paper-based literacy assessments are typically decontextualized and may cause stress and anxiety for test takers. In contrast, serious games and game environments allow for the assessment of literacy in more authentic and engaging ways, which has...

Literacy skills are critical to academic success and in life, and literacy assessments have been used extensively to help improve students’ skills. In this chapter, we propose an innovative implementation of natural language processing (NLP) to conduct stealth assessment of literacy skills. We describe literacy assessments, including applications o...

Human needs and goals are integral to engineering, but human-centered engineering requires understanding principles and methods from the social sciences. Instruction is needed to introduce engineering students to these concepts. This study explored (a) students’ initial attitudes toward human systems engineering (HSE) and the bridging of psychology...

Artificial Intelligence- (AI-) based assessments are commonly used in a variety of settings including business, healthcare, policing, manufacturing, and education. In education, AI-based assessments undergird intelligent tutoring systems as well as many tools used to evaluate students and, in turn, guide learning and instruction. This chapter provi...

Four "who-ristic" questions to ask yourself when designing artificially intelligent educational technologies that will actually benefit people.

Modern communication between health care professionals and patients increasingly relies upon secure messages (SMs) exchanged through an electronic patient portal. Despite the convenience of secure messaging, challenges include gaps between physician and patient expertise along with the asynchronous nature of such communication. Importantly, less re...

The current study an empirical evaluation of the PERvasive Learning System (PERLS). PERLS is a mobile microlearning platform designed for learning anytime and anywhere, taking advantage of planned and unplanned time during a learner’s daily schedule to enhance and reinforce learning. Soldiers taking classes from the Sabalauski Air Assault School at...

This panel session is for anyone in human factors and ergonomics (HFE) or related disciplines interested in recruiting, hiring, admitting, retaining, and promoting people within organizations from the perspective of pursuing authentic diversity. Applicants and hiring managers may also gain insight on what makes a meaningful diversity statement, a t...

The Interactive Strategy Training for Active Reading and Thinking (iSTART) game-based intelligent tutoring system (ITS) was developed with a foundation of comprehension theory and principles of learning science to improve students’ comprehension of complex scientific texts. iSTART has been shown to improve reading comprehension for learners from mi...

Researchers and educators have developed a variety of computer-based technologies intended to facilitate self-regulated learning (SRL), which refers to iterative learning processes wherein individuals set plans and goals, complete tasks, monitor their progress and outcomes, and adapt future efforts. This paper draws upon the SRL literature and rela...

The present study examined the extent to which adaptive feedback and just-in-time writing strategy instruction improved the quality of high school students’ persuasive essays in the context of the Writing Pal (W-Pal). W-Pal is a technology-based writing tool that integrates automated writing evaluation into an intelligent tutoring system. Students...

The benefits of writing strategy feedback are well established. This study examined the extent to which adding spelling and grammar checkers support writing and revision in comparison to providing writing strategy feedback alone. High school students (n = 119) wrote and revised six persuasive essays in Writing Pal, an automated writing evaluation a...

The increased reliance on online education and educational technologies more generally has laid bare the need to more deeply consider how researchers, designers, and educators can improve the quality of technology-mediated learning. To address this need, more than two dozen experts from a variety of fields came together to discuss the challenges th...

Despite decades of effort to broaden participation in science, technology, engineering, and mathematics (STEM), many fields remain demographically skewed. Marginalized and minoritized people are still underrepresented in and underserved by the sciences. In this paper, the author considers the question, “How do we improve representation in STEM?” by...

Virtual reality (VR) has a high potential to facilitate education. However, the design of many VR learning applications was criticized for lacking the guidance of explicit and appropriate learning theories. To advance the use of VR in effective instruction, this study proposed a model that extended the cognitive-affective theory of learning with me...

As the Human Factors and Ergonomics Society Diversity, Equity, and Inclusion Committee enters its fifth year, it continues to encourage the society, annual meeting attendees, and human factors professionals to improve diversity and equity within the field. At the center of this field are humans and their widely varying needs and abilities. While HF...

Experts in human factors and ergonomics (HF/E) and related fields have the ability and responsibility to broadly serve the needs and goals of diverse people, which encompasses issues of inclusion, equity, and justice. Importantly, HF/E designers, researchers, and practitioners can address these aims both as the intended outcomes of their work and h...

iSTART is a game-based intelligent tutoring system (ITS) designed to improve students’ reading skills by providing training on reading comprehension strategies. Game-based practice in iSTART follows two main approaches: generative practice and identification practice. Generative practice games ask students to author self-explanations using one or m...

This article is Part 1 of a two-part series reflecting on diversity within the Human Factors and Ergonomics Society (HFES) and how the pursuit of “authentic” diversity is essential to HFES’s overarching goals for inclusion and equity. In Part 1, authentic diversity is discussed – what it means and what it might look like. Through this lens of authe...

This article is Part 2 of a two-part series reflecting on diversity within the Human Factors and Ergonomics Society (HFES). Part 1 discussed what it means to pursue authentic diversity and reported recent demographic characteristics of HFES. Part 2 discusses a brief history of relevant efforts in HFES and recent scholarship that suggests sustained...

Engineering education refers to developing an understanding of the principles, methods, and ways of thinking that underlie engineering, and preparing students and engineers for productive engineering careers. The purpose of this review is to explore how head‐mounted display‐based virtual reality (HMD VR) can contribute to these goals. Historically,...

This study investigated changes in learners' motivation, engagement, performance, and spatial reasoning over time and across different levels of virtual reality (VR) immersion. Undergraduate participants explored a virtual solar system via a moderately immersive or highly immersive VR platform over three sessions. In a third condition, participants...

Global investments in artificial intelligence (AI) and robotics are on the rise, with the results to impact global economies, security, safety, and human well-being. The most heralded advances in this space are more often about the technologies that are capable of disrupting business-as-usual than they are about innovation that advances or supports...

The Diversity Committee of HFES has led sessions at the Annual Meeting for the past three years focused on improving diversity, equity and inclusion in the society as well as providing support to human factors and ergonomics (HF/E) researchers and practitioners who aim to apply HF/E knowledge and principles to improve diversity, equity and inclusio...

Researchers and educators have explored a variety of technologies to facilitate self-regulated learning (SRL). Drawing from contemporary perspectives on SRL, this paper articulates two fundamental design principles for SRL-promoting technologies: the platform principle and the support principle. This paper then discusses how usability inspection me...

Instruction and coursework that link engineering and psychology may enable future engineers to better understand the people they are engineering for (e.g., users and clients) and themselves as engineers (e.g., teammates). In addition, human-centered engineering education may empower engineering students to better solve problems at the intersection...

Societal impact refers to the influence of research on economic, environmental, cultural, and social outcomes that extend beyond the scientific sphere. In 2018, the Human Factors and Ergonomic Society launched the Societal Impact Committee to mindfully identify societal issues in which HF/E could contribute to beneficial impact while also advancing...

The expansion of online education into massive open online courses (MOOCs) and equipment have created a unique opportunity for delivering immersive learning experiences at scale. However, although the inclusivity of the MOOC ecosystem can be commended, many online courses lack key benefits associated with traditional classroom environments: immersi...

The success of engineering and design is facilitated by a working understanding of human thoughts, feelings, and behaviors. In this study, we explored how undergraduate engineering students included such human-centered and psychological concepts in their project documentation. Although, we observed a range of concepts related to design processes, t...

Computer security and user privacy are critical issues and concerns in the digital era due to both increasing users and threats to their data. Separate issues arise between generic cybersecurity guidance (i.e., protect all user data from malicious threats) and the individualistic approach of privacy (i.e., specific to users and dependent on user ne...

Writers are often judged by their audience, and these evaluations can encompass both the text and the authors. This study built upon prior research on writing evaluation and error perceptions to examine how interconnected or separable are these judgments. Using a within subjects design, college students evaluated four essays demonstrating no errors...

SoLaR-SoAR consists of a review of the current state of the art for distributed learning environments. The report is structured for use by multiple types of end users. The main report provides a 22-page, high-level overview of findings of the current state of the art. This section can serve as a quick reference. The report’s Appendixes provide a de...

This study examined differences in consumer preferences and willingness to pay for sustainable bottled water based on pro-environmental guidance, Internet information search, and research setting (i.e., laboratory or online). Specifically, we investigated willingness to pay for bottled water produced with plant-based plastics and post-consumer wast...

Literacy is a critically important and contemporary issue for educators, scientists, and politicians. Efforts to overcome the challenges associated with illiteracy, and the subsequent development of literate societies, are closely related to those of poverty reduction and sustainable human development. In this paper, the authors examine literacy fr...

This study investigated the effect of incorporating spelling and grammar checking tools within an automated writing tutoring system, Writing Pal. High school students (n = 119) wrote and revised six persuasive essays. After initial drafts, all students received formative feedback about writing strategies. Half of the participants were also given ac...

Engineering grand challenges and big ideas not only demand innovative engineering solutions, but also typically involve and affect human thought, behavior, and quality of life. To solve these types of complex problems, multidisciplinary teams must bring together experts in engineering and psychological science, yet fusing these distinct areas can b...

The present study extended research on the effectiveness of automated writing evaluation (AWE) systems. Sixth-graders were randomly assigned by classroom to an AWE condition that used PEG Writing (n = 56) or a word-processing condition that used GoogleDocs (n = 58). Effectiveness was evaluated using multiple metrics: writing self-efficacy, holistic...

Traditional user experience assessments rely on self-report, human-system performance, and observational data that incompletely capture users' psychological demands, processing, or affect. Specifically, self-report measures require users to identify and articulate subjective responses to product features, yet users may not possess accurate awarenes...

The Human Factors and Ergonomics Society Diversity Committee is entering its second year and continuing to explore ways to increase the diversity of the society. Following last year’s panel on “Challenges and Opportunities for Involvement,” we, and others, recognized that human factors and ergonomics (HFE) professionals are equipped and able to adv...

Human systems engineering education seeks to infuse principles of applied psychology, cognitive science, human factors, and user-centered design into the engineering curriculum to help students understand the people they are engineering for (e.g., clients) and their own roles as engineers. This paper outlines a conceptual qualitative framework for...

Educational technologies with multimedia content can support effective learning, but these outcomes are moderated by learners’ level of cognitive engagement or self-regulation. As a way to encourage deeper cognitive engagement without redesigning or redeveloping software (e.g., building more prompts, scaffolds, or automated support), this study inv...

This study evaluates high school students’ perceptions of automated writing feedback, and the influence of these perceptions on revising, as a function of varying modes of computer-based writing instruction. Findings indicate that students’ perceptions of automated feedback accuracy, ease of use, relevance, and understandability were favorable. Imm...

Research has observed substantial value in learning interventions where peers tutor and assess each other, and has outlined activities (e.g., explaining, questioning, and feedback) that enable interactive and reflective knowledge-building. This chapter considers how the underlying learning processes of peer tutoring and peer assessment can be enact...

A critical challenge for computer-based writing instruction is providing appropriate and adaptive practice. The current study examined three modes of computer-based writing practice with the goal of identifying those with the greatest learning and motivational value. High school students learned about writing strategies by studying lessons within t...

The design and development of educational technologies is a complex, interdisciplinary endeavor. Learning science research reveals principles of learning and instruction, and advances in computer science implement these principles in innovative technologies. This chapter promotes a complementary discipline-human systems engineering or "user science...

Both conventional wisdom and empirical research suggest that errors in writing impact perceptions of both writing quality and characteristics of the author. Texts that exhibit poor spelling and grammar, or lack compelling arguments and clear structure, are perceived as lower quality. Moreover, the authors themselves may be perceived as less intelli...

This study investigates a novel approach to conducting formative writing assessment that involves evaluating students' writing skills across three levels of language (word, sentence, and discourse) using automated measures of word choice, syntax, and cohesion. Writing from students in Grades 6 and 8 (n= 240 each) was analyzed using Coh-Metrix. Mult...

The Human Factors and Ergonomics Society Diversity Committee met initially in January 2017, and on a regular basis thereafter to assess and improve diversity and inclusion in the society, profession, and discipline. Charged by president Bill Marras in 2016, the Committee replaced the Diversity Task Force established in 1994, and formally became a p...

Emerging technologies have enhanced the learning capabilities and opportunities in modern school systems. To continue the effective development of such innovations, the intended users must be taken into account. End-User Considerations in Educational Technology Design is a pivotal reference source for the latest scholarly material on usability tes...

Traditional user experience assessments rely on self-report, human-system performance, and observational data that incompletely capture users' psychological demands, processing, or affect. Specifically, self-report measures require users to identify and articulate subjective responses to product features, yet users may not possess accurate awarenes...

This chapter explores three broad principles of user-centered design methodologies, including participatory design, iteration, and usability considerations. We discuss characteristics of teachers as an important type of ITS end user, including barriers teachers face as users and their role in educational technology design. To exemplify key points,...

The design and development of educational technologies is a complex, interdisciplinary endeavor. Learning science research reveals principles of learning and instruction, and advances in computer science implement these principles in innovative technologies. This chapter promotes a complementary discipline-human systems engineering or "user science...

This study investigated how students’ implicit theory that public speaking abilities can be developed and improved (i.e., growth mindset) was related to their apprehension, perceived competence, and beliefs regarding public speaking. Growth mindset was associated with lower apprehension and higher self-perceived competence. Growth mindset was also...

Automated writing evaluation (AWE) is a popular form of educational technology designed to supplement writing instruction and feedback, yet research on the effectiveness of AWE has observed mixed findings. The current study considered how students' perceptions of automated essay scoring and feedback influenced their writing performance, revising be...

Writing is a complex and subjective endeavor. There are numerous ways to write well (Crossley, Roscoe, & McNamara, 2014) and writing involves a nonlinear, iterative process of generating, organizing, translating, and refining ideas (Bereiter & Scardamalia, 1987; Hayes, 2012: Kellogg, 2008). Importantly, assessment is integral to each stage. Student...

Multimedia instructional materials require learners to select, organize, and integrate information across multiple modalities. To facilitate these comprehension processes, a variety of multimedia design principles have been proposed. This study further explores the redundancy principle by manipulating the degree of partial redundancy between writte...

The Writing Pal is an intelligent tutoring system designed to support writing proficiency and strategy acquisition for adolescent writers. A fundamental aspect of the instructional model is automated formative feedback that provides concrete information and strategies oriented toward student improvement. In this paper, the authors explore computati...

Achieving sustained student engagement with practice in computer-based writing strategy training can be a challenge. One potential solution is to foster engagement by embedding practice in educational games; yet there is currently little research comparing the effectiveness of game-based practice versus more traditional forms of practice. In this s...

Prior research has established that learning by teaching depends upon peer tutors’ engagement in knowledge-building, in which tutors integrate their knowledge and generate new knowledge through reasoning. However, many tutors adopt a knowledge-telling bias defined by shallow summarizing of source materials and didactic lectures. Knowledge-telling c...

This study identifies multiple profiles of successful essays via a cluster analysis approach using linguistic features reported by a variety of natural language processing tools. The findings from the study indicate that there are four profiles of successful writers for the samples analyzed. These four profiles are linguistically distinct from one...

Automated essay scoring tools are often criticized on the basis of construct validity. Specifically, it has been argued that computational scoring algorithms may be unaligned to higher-level indicators of quality writing, such as writers' demonstrated knowledge and understanding of the essay topics. In this paper, we consider how and whether the sc...

The ICAP framework provides a theory of cognitive engagement based on overt learning activities that may inform instructional design. In ongoing work to investigate ICAP as a theoretically-grounded instructional design system, classroom teachers participated in a workshop to learn about the framework and design lessons at varying levels. In this pa...

Across a variety of domains, formative feedback is often regarded as beneficial, if not crucial to learning. Yet studies show that this assumption does not always hold true: some types of feedback do not benefit learners. This symposium brings together researchers investigating how feedback can be optimized to maximize potential benefits. The four...

The Writing-Pal (W-Pal) is an intelligent tutoring system (ITS) that provides writing strategy instruction to high school students and entering college students. One unique quality of W-Pal is that it provides feedback to students’ natural language input. Thus, much of our focus during the W-Pal project has been on Applied Natural Language Processi...

The Writing Pal (W-Pal) is a novel intelligent tutoring system (ITS) that offers writing strategy instruction, game-based practice, essay writing practice, and formative feedback to developing writers. Compared to more tractable and constrained learning domains for ITS, writing is an ill-defined domain because the features of effective writing are...

This study investigates students’ essay revising in the context of an intelligent tutoring system called Writing Pal (W-Pal), which combines strategy instruction, game-based practice, essay writing practice, and automated formative feedback. We examine how high school students use W-Pal feedback to revise essays in two different contexts: a typical...

We present an evaluation of the Writing Pal (W-Pal) intelligent tutoring system (ITS) and the W-Pal automated writing evaluation (AWE) system through the use ofcomputational indices related to text cohesion. Sixty-four students participated in this study. Each student was assigned to either the W-Pal ITS condition or the W-Pal AWE condition. The W-...

Writing is a necessary skill for success in the classroom and the workplace; yet, many students are failing to develop sufficient skills in this area. One potential problem may stem from a misalignment between students' and teachers' criteria for quality writing. According to the evaluative misalignment hypothesis, students assess their own writing...

To support self-regulated learning (SRL), computer-based learning environments (CBLEs) are often designed to be open-ended and multidimensional. These systems incorporate diverse features that allow students to enact and reveal their SRL strategies via the choices they make. However, research shows that students’ use of such features is limited; st...

Various computer tools have been developed to support educators' assessment of student writing, including automated essay scoring and automated writing evaluation systems. Research demonstrates that these systems exhibit relatively high scoring accuracy but uncertain instructional efficacy. Students' writing proficiency does not necessarily improve...

We present an evaluation of the Writing Pal (W-Pal) intelligent tutoring system (ITS) and the W-Pal automated writing evaluation (AWE) system through the use of computational indices related to text cohesion. Sixty-four students participated in this study. Each student was assigned to either the W-Pal ITS condition or the W-Pal AWE condition. The W...

In this chapter, the authors consider the value of educational games to support students' writing strategy acquisition and practice. Sixty-five high school students participated in a summer program using the Writing Pal, an intelligent tutoring system designed to support adolescents' persuasive writing across multiple phases of the writing process....

The ability of educational games to promote students' engagement in learning and practice depends on perceived enjoyment of those games. This study investigated high school students' perceptions and enjoyment of games within the Writing Pal intelligent tutoring system. In accord with research on motivation, results showed that perceived helpfulness...

This study compares automated scoring increases and linguistic changes for student writers in two groups: a group that used an intelligent tutoring system embedded with an automated writing evaluation component (Writing Pal) and a group that used only the automated writing evaluation component. The primary goal is to examine automated scoring diffe...

The Writing Pal is an intelligent tutoring system that provides writing strategy training. A large part of its artificial intelligence resides in the natural language processing algorithms to assess essay quality and guide feedback to students. Because writing is often highly nuanced and subjective, the development of these algorithms must consider...

Learning new concepts and ideas typically requires that the learners activate and bring to bear some prior knowledge for interpreting and assimilating the new ideas. This prior knowledge can be conceived of as some coherent body of relevant knowledge that can be referred to as a schema. Once the new ideas and concepts are assimilated within an exis...

Current computer based tools for writing instruction show high scoring accuracy but uncertain instructional efficacy. One explanation is that these systems may not effectively communicate valid and appropriate formative feedback. In this paper, we describe an exploratory method for developing feedback algorithms that are grounded in writing pedagog...

Studies exploring how students learn and understand science processes such as diffusion and natural selection typically find that students provide misconceived explanations of how the patterns of such processes arise (such as why giraffes' necks get longer over generations, or how ink dropped into water appears to "flow"). Instead of explaining the...

This study assesses the potential for computational indices to predict human ratings of essay quality. The results demonstrate that linguistic indices related to type counts, given/new information, personal pronouns, word frequency, conclusion n-grams, and verb forms predict 43% of the variance in human scores of essay quality.

Research on automated essay scoring (AES) indicates that computer-generated essay ratings are comparable to human ratings. However, despite investigations into the accuracy and reliability of AES scores, less attention has been paid to the feedback delivered to the students. This paper presents a method developers can use to quickly evaluate the us...

Natural language processing and statistical methods were used to identify linguistic features associated with the quality of student-generated paragraphs. Linguistic features were assessed using Coh-Metrix. The resulting computational models demonstrated small to medium effect sizes for predicting paragraph quality: introduction quality r2 = .25, b...