James W. Pellegrino

James W. Pellegrino
University of Illinois at Chicago | UIC · Department of Psychology

About

242
Publications
122,145
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
12,395
Citations
Additional affiliations
August 2001 - present
University of Illinois at Chicago
Position
  • Co-Director
August 1989 - July 2001
Vanderbilt University
Position
  • Mayborn Professor & Dean
April 1973 - July 1979
University of Pittsburgh
Position
  • Professor (Associate)

Publications

Publications (242)
Preprint
Full-text available
Handbook of Research on Science Education
Article
Full-text available
This study provides a solid validity inferential network to guide the development, interpretation, and use of machine learning-based next-generation science assessments (NGSAs). Given that machine learning (ML) has been broadly implemented in the automatic scoring of constructed responses, essays, simulations, educational games, and interdisciplina...
Article
Background & Context: We describe the rationale, design, and initial validation of computational thinking (CT) assessments to pair with curricular lessons that integrate fractions and CT. Objective: We used cognitive models of CT (learning trajectories; LTs) to design assessments and obtained evidence to support a validity argument Method: We use...
Preprint
Full-text available
This study provides a solid validity inferential network to guide the development, interpretation, and use of machine learning-based Next Generation Science Assessments (NGSAs). Given that machine learning (ML) has been broadly implemented in the automatic scoring of constructed responses, essays, simulations, educational games, and interdisciplina...
Article
Full-text available
To extend the existing research examining multiple text comprehension and its assessment, we developed a verification task approach to assessing of information that was explicitly and implicitly presented within and across nine texts. A nonparametric form of signal detection theory was used to analyse the performance of 47 middle school students us...
Article
Professor Gordon argues for a significant reorientation in the focus and impact of assessment in education. For the types of assessment activities that he advocates to prosper and positively impact education, serious attention must be paid to two important topics: (1) the conceptual underpinnings of the assessment practices we develop and use to su...
Presentation
Computational thinking (CT) assessment tasks were developed to assess students' CT knowledge, skills, and abilities. These tasks were developed from hypothesized learning trajectories for several CT topics (e.g., sequence, repetition). The tasks were administered to elementary students who were taught using an integrated math + CT curriculum. To ev...
Article
Full-text available
Machine learning (ML) is an emergent computerized technology that relies on algorithms built by 'learning' from training data rather than 'instruction', and which holds great potential to revolutionize science assessment. This study systematically reviewed the efforts in literature aiming to apply ML in science assessment through a triangle framewo...
Article
We describe an approach to characterizing and diagnosing complex professional competencies (CPCs) for the field of Intrapreneurship, i.e. activities of an entrepreneurial nature engaged by employees within their existing organizations. Our approach draws upon prior conceptual, empirical, and analytical efforts by researchers in Germany. Results are...
Article
Contemporary views on learning highlight that deep learning occurs not simply by accumulating knowledge, but by using and applying knowledge as one engages in disciplinary activity. Increasingly, those concerned with education policy and practice are shifting priorities toward supporting deeper learning by emphasizing the importance of students’ ab...
Chapter
This study developed an assessment that was used to evaluate the mathematics knowledge and skills that college students have mastered relative to what is supposedly needed for success in higher education as suggested by prior studies (Conley et al., Reaching the goal: The applicability and importance of the common core state standards to college an...
Article
In this article we describe the challenges associated with assessing complex competencies envisioned as the targets of learning in the 21st century. Such competencies typically reflect the integration of multiple dimensions of knowledge and skill. Technology plays a crucial role in their assessment, from conceptualisation through design, data gathe...
Article
This commentary extends and amplifies aspects of the Cantor, Osher, Berg, Steyer and Rose (2018) discussion about what we know about the nature of learning and its implications for how we educate students across the span from early childhood through advanced levels of formal education. The paper’s overall goal is to offer some guidance and clarific...
Technical Report
Full-text available
The purpose of this study was to examine the nature of text-task-reader interactions in relation to student performance on the National Assessment of Educational Progress (NAEP) reading assessment and, in doing so, suggest strategies that NAEP might use to select passages, develop items, and interpret data to more closely approximate the complex na...
Chapter
Higher education domains such as Business and Economics (B&E) generally lack evidence that standardized tests of competency actually assess those aspects of knowledge and reasoning that constitute important targets for learning and instruction. The construction of such a validity argument benefits from a framework to guide the collection of relevan...
Chapter
Increasing attention is being paid to the need for systems of assessments, with an emphasis on the design and use of assessments intended to function much closer to the processes of teaching and learning. This chapter reviews key aspects of the validity framework for assessments intended to directly benefit instructors and students. It provides an...
Chapter
This chapter draws upon the report Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century (Pellegrino and Hilton, 2012) to address questions about why the terms labelled “deeper learning” and “21st century skills” have achieved prominence in the thinking and actions of multiple stakeholder groups and what we k...
Article
The Standards for Educational and Psychological Testing indicate that validation of assessments should include analyses of participants’ response processes. However, such analyses typically are conducted only to supplement quantitative field studies with qualitative data, and seldom are such data connected to quantitative data on student or item pe...
Article
Assessments that function close to classroom teaching and learning can play a powerful role in fostering academic achievement. Unfortunately, however, relatively little attention has been given to discussion of the design and validation of such assessments. The present article presents a framework for conceptualizing and organizing the multiple com...
Article
Full-text available
There is considerable rhetoric about the need for our educational system to promote deeper learning and the development of 21st-century skills. Missing from the discourse is recognition that much of what we know from research on learning and instruction has yet to affect the design and enactment of everyday schooling in the form of curriculum, inst...
Article
Background Concept inventories (CIs) are commonly used in engineering disciplines to assess students' conceptual understanding and to evaluate instruction, but educators often use CIs without sufficient evidence that a structured approach has been applied to validate inferences about student thinking. PurposeWe propose an analytic framework for eva...
Conference Paper
The panelist will present a vision for assessing science education in the United States and recommend some steps for implementing that vision. He will also discuss examples of appropriate science assessments.
Article
Full-text available
Concept inventories (CIs) are increasingly being developed and used in engineering courses to assess student learning and understanding and to evaluate instructional practices. CIs differ from typical STEM assessments in that they tend to focus on a small set of key constructs and conceptual understandings within the domain-such as "the concept of...
Chapter
Chapter Overview and Goals. In 2001, a report was issued by the National Research Council (NRC) entitled “Knowing What Students Know: The Science and Design of Educational Assessment” (Pellegrino, Chudowsky, & Glaser, 2001). The goal was to evaluate the state of research and theory on educational assessment and establish the scientific foundations...
Article
Full-text available
This paper argues that assessment can serve as a positive influence on attaining 21st century learning goals. Section I focuses on 21st century education challenges and the types of assessments needed to support attainment of learning objectives relevant to a global society. Sections II and III discuss the purposes and contexts of educational asses...
Article
Full-text available
Purpose – The purpose of this article is to present alternative views on the theory and practice of formative assessment (FA), or assessment to support teaching and learning in classrooms, with the purpose of highlighting its value in education and informing discussions on educational assessment policy. Methodology/approach – The method used is a...
Conference Paper
Full-text available
This paper reports statistical and psychometric analyses that were part of an overall principled approach to designing concept inventories (CIs) using evidentiary arguments about claims that can be made about what students know and can do. An evidence-centered design (ECD) framework (Mislevy, Almond, & Lukas, 2004) was applied to a pre-existing CI,...
Conference Paper
Full-text available
In STEM undergraduate courses, instructors have been increasingly expanding classroom learning objectives from focusing mainly on procedural knowledge to also including conceptual understanding. One effort in this direction is the development of Concept Inventories (CI) to assess student conceptual understanding. The first CI, the Force Concept Inv...
Chapter
From teachers’ classroom quizzes and mid-term or final exams, to nationally and internationally administered standardized tests, assessments of students’ knowledge and skills have become a ubiquitous part of the educational landscape. Assessments of school learning provide information to help educators, administrators, policy makers, students, pare...
Article
Full-text available
An incidental learning procedure was used with 8-, 11-, and 19-year-olds to identify processing changes that might underlie a developmental shift in feature encoding. Presentation of each of 32 words was accompanied by a question designed to induce either acoustic or semantic encoding. Tests of recall and recognition followed an intervening task. A...
Article
Full-text available
Free recall learning of a mixed list of 20 related and 20 unrelated words was enhanced more if the related words belonged to common taxonomie categories than if they were hierarchically ordered as a single branch of a “semantic tree” (e.g., animal, mammal, dog, terrier). The same difference obtained in 1-week delayed recall. No significant taxonomi...
Article
The goal of these experiments was to introduce and test the constructive retrieval hypothesis, according to which retrieval practice will be most effective when it encourages constructive elaborations of text content. Experiment 1 provided baseline performance data for the materials included in Experiments 2 and 3. In Experiment 2, instilling infer...
Conference Paper
Full-text available
Traditional studies of intelligent tutoring systems have focused on their use in the classroom. Few have explored the advantage of using ITS as a web-based homework (WBH) system, providing correctness-only feedback to students. A second underappreciated aspect of WBH is that teachers can use the data to more efficiently review homework. Universitie...
Article
Full-text available
The encoding specificity effect was examined in the Tulving and Thomson (1973) generation-recognition-recall paradigm. The critical comparison involved the relative effectiveness of category cues which were either the same as or different from the input encoding of an item. Additionally, an attempt was made to gain control over any bias in respondi...
Article
Full-text available
Individual differences on spatial tasks were examined relative to differences in free-androgen levels. A spatial test battery was administered to 91 males and females who differed in free-androgen levels as determined by a radioimmunoassay. Polynomial regression analyses yielded significant curvilinear functions relating spatial scores and androgen...
Article
Full-text available
Free recall performance with both constant and varied input orders was compared to serial recall performance across three grade levels. Categorized lists composed of frequent instances from categories familiar to young children were presented under both blocked and nonblocked conditions. Results revealed high levels of recall performance and semant...
Article
Proficiency in science is being defined through performance expectations that intertwine science practices, cross-cutting concepts, and core content knowledge. These descriptions of what it means to know and do science pose challenges for assessment design and use, whether at the classroom instructional level or the system level for monitoring the...
Conference Paper
In redesigning the AP science courses and exams, it was critical to focus each disciplinary group on both the core ideas within their respective disciplines as well as the forms of scientific reasoning and the practices of science that students would be expected to employ to engage those ideas and demonstrate understanding. This led to a much riche...
Book
Americans have long recognized that investments in public education contribute to the common good, enhancing national prosperity and supporting stable families, neighborhoods, and communities. Education is even more critical today, in the face of economic, environmental, and social challenges. Today's children can meet future challenges if their sc...
Chapter
This chapter provides a discussion of issues associated with research and development to effect a more productive connection between technology and the design and deployment of assessments that can measure what matters and support learning in a digital world. Assessment is first discussed as a process of reasoning from evidence, emphasizing its nec...
Technical Report
Full-text available
Based on the changing demands of today’s workforce, advances in other nations, and original analysis, this report provides a set of criteria for high-quality student assessments. These criteria can be used by assessment developers, policymakers, and educators as they work to create and adopt assessments that promote deeper learning of 21stcentury s...
Article
Beginning with a reference to living in a time of both uncertainty and opportunity, this article presents a discussion of key areas where shared understanding is needed if we are to successfully realize the design and use of high quality, valid assessments of science. The key areas discussed are: (1) assessment purpose and use, (2) the nature of as...
Conference Paper
Full-text available
Educational designers and researchers are faced with new challenges when it comes to the design of open games for learning that allow players to choose their own solution path through the game. The goal of this empirical study with 70 university students was to be able to tell at any point in the game what the player knows and whether a specific le...
Conference Paper
Full-text available
The utility of concept inventories (CIs) as assessment tools in engineering education settings is considered in the context of contemporary theories of assessment design, measurement, and validity. A comprehensive approach to validity analysis is described and applied to available evidence for three STEM CIs – CATS, TTCI, & SCI. The results reveal...
Chapter
This article provides a discussion of how technology can be used to support implementing quality assessment, especially formative assessment. It first considers the meaning of formative assessment, why it is such an important assessment activity, and how technology might support implementation of a range of assessment functions including formative...
Article
This paper considers uses of technology in educational assessment from the perspective of innovation and support for teaching and learning. It examines assessment cases drawn from contexts that include large-scale testing programs as well as classroom-based programs, and attempts that have been made to harness the power of technology to provide ric...
Conference Paper
This poster will present research about student understandings of solutions while observing dissolution and reactions first qualitatively and then quantitatively using ion-selective probeware. Preliminary data show that students have difficulty describing dissolutions symbolically and have difficulty choosing products in a double displacement react...
Article
The involvement of working memory capacity (WMC) in ruled-based cognitive skill acquisition is well-established, but the duration of its involvement and its role in learning strategy selection are less certain. Participants (N = 610) learned four logic rules, their corresponding symbols, or logic gates, and the appropriate input–output combinations...
Chapter
Participation in Experiments and Reflection on the Meaning of the DataAnalysis of Empirical Articles and Connection to Class ContentCognition in Daily Life ExerciseGeneral DiscussionReferences
Article
Full-text available
Large-scale testing of educational outcomes benefits already from technological applications that address logistics such as development, administration, and scoring of tests, as well as reporting of results. Innovative applications of technology also provide rich, authentic tasks that challenge the sorts of integrated knowledge, critical thinking,...
Article
The July 2008 issue of the Journal of Engineering Education is based on an engineering design approach to identifying ways and means for creating more vital engineering programs. It focuses on the primary customer of engineering education and students. A team of authors were challenged to look at educational issues that are not typically explored....
Conference Paper
Full-text available
This study investigated upper-level college students' understanding of evidence use in quality scientific arguments. Responses to a required online 'contrasting-essays' assignment provided instructors with quick access to formative information about students' capacity to evaluate the quality of argument and the use of evidence therein. Students inc...
Chapter
This chapter discusses evolution of the thinking of the Cognition and Technology Group at Vanderbilt about teaching, learning, assessment, technology, and instructional design under the broad heading of their extended work on the principles and practices of Anchored Instruction. It begins by stating a general set of ideas about the characteristics...
Article
Full-text available
The literature base on technology professional development for teachers reveals that there is a long way to go in understanding methods of effective practice with respect to the various impacts of these activities on teaching and learning. In the No Child Left Behind era, with programs like Preparing Tomorrow’s Teachers to Use Technology, the Fund...
Article
This article presents findings from two projects designed to improve evaluations of technical quality of alternate assessments for students with the most significant cognitive disabilities. We argue that assessment technical documents should allow for the evaluation of the construct validity of the alternate assessments following the traditions ofC...
Conference Paper
Full-text available
Introducing an emerging science discipline such as nanoscience into the grade 7-12 classroom creates many challenges. As emerging sciences tend to be interdisciplinary in nature, changes in instruction, assessment and curriculum development are required. In particular, this interdisciplinary nature requires that a greater priority must be placed on...
Article
Full-text available
Students' responses to scientific reasoning questions posted on an online discussion board were coded for completeness, coherence, accuracy, use of evidence, reasoning, and comprehensibility. Early performance was low for every dimension. By the end of the term, ratings of coherence and completeness increased but use of evidence decreased, and othe...