Richard J. Shavelson

Stanford University | SU · Graduate School of Education

Free access to a wealth of unreliable, inaccurate and contradictory information re-quires students to take a critical approach to online media and its content. The skillset to critically evaluate and use sources in argumentative conclusions (Critical Thinking; CT) is becoming increasingly important for academic and professional success. For effecti...

Studierende nutzen im Lernprozess zunehmend digitale Ressourcen und insb. Studierende der Wirtschaftspädagogik (WiPäd), die sich den transformierenden Anforderungen der Arbeitswelt begegnen, greifen auf aktuell freiverfügbare Quellen zurück. Die Kompetenz zuverlässige Quellen zu finden, deren Inhalt kritisch zu bewerten und zu einem argumentativen...

A key skill for successful learning in higher education is critical evaluation of an overabundance of freely accessible, often unreliable, inaccurate, contradictory or even incorrect information in ever-changing information environments in a digital age. Critically evaluating multiple sources and drawing a well-founded conclusion (Critical Thinking...

Um das Critical Thinking von Studierenden valide zu messen, wurde in einem internationalen Konsortium ein komplexes Szenario-basiertes Perfomance Assessment (PA) entwickelt. Ein PA beinhaltet eine realitätsnahe Simulation einer Entscheidungssituation. Es enthält vorselektierte (un-)relevante, (un-)zuverlässige und teilweise widersprüchliche Quellen...

ABSTRACT Primary objective: The significance of critical thinking (CT) has grown worldwide in recent years. In teacher education, for example, educators are expected to impart this skill to students but we lack comprehensive international comparative research on CT in pre-service teachers. To begin to develop such assessments, we comparatively anal...

In recent decades, higher education has undergone far-reaching structural and programmatic changes. The quality of academic education has become a significant concern, resulting in challenges for objective, reliable and valid assessments. In response, the Germany-wide initiative, “Modeling and Measuring Competencies in Higher Education,” (KoKoHs),...

Quantitative reasoning is considered a crucial prerequisite for acquiring domain-specific expertise in higher education. To ascertain whether students are developing quantitative reasoning, validly assessing its development over the course of their studies is required. However, when measuring quantitative reasoning in an academic study program, it...

Universities increasingly use performance assessments (PAs) to measure students’ competencies and learning outcomes. Following a holistic criterion-sampling approach, the PAs represent real-life situations in which multifaceted competencies (e.g., critical thinking) required for meeting specific academic/professional demands are measured. Digital P...

Higher education institutions worldwide claim they impact students’ learning outcomes within and across academic domains. Critical thinking (CT) is prominent among the intended outcomes (Braun et al., 2020). In this context, there is increasing interest in ecologically valid performance assessments (PAs) of CT that can be used internationally (Zlat...

Critical evaluation skills when using online information are considered important in many research and education frameworks; critical thinking and information literacy are cited as key twenty-first century skills for students. Higher education may play a special role in promoting students' skills in critically evaluating (online) sources. Today, hi...

Critical evaluation skills when using online information are considered important in many research and education frameworks; critical thinking and information literacy are cited as key twenty-first century skills for students. Higher education may play a special role in promoting students' skills in critically evaluating (online) sources. Today, hi...

Critical reasoning (CR) when confronted with contradictory information from multiple sources is a crucial ability in a knowledge-based society and digital world. Using information without critically reflecting on the content and its quality may lead to the acceptance of information based on unwarranted claims. Previous personal beliefs are assumed...

Enhancing students’ critical thinking (CT) skills is an essential goal of higher education. This article presents a systematic approach to conceptualizing and measuring CT. CT generally comprises the following mental processes: identifying, evaluating, and analyzing a problem; interpreting information; synthesizing evidence; and reporting a conclus...

A few thousand years ago, Aristotle pointed out a gap between academic (research) knowledge and practical knowledge. The former is abstract, a generalization based on careful reasoning from evidence. The latter is specific, context-specific. Consequently, factors other than research knowledge come into play when teaching and when preparing teachers...

A few thousand years ago, Aristotle pointed out a gap between academic (research) knowledge and practical knowledge. The former is abstract, a generalization based on careful reasoning from evidence. The latter is specific, context-specific. Consequently, factors other than research knowledge come into play when teaching and when preparing teachers...

Contemporary higher education is committed to enhancing students’ scientific thinking in part by improving their capacity to think critically, a competence that forms a foundation for scientific thinking. We introduce and evaluate the characteristic elements of critical thinking (i.e. cognitive skills, affective dispositions, knowledge), problemati...

The Swiss teacher education and training system offers a practically and academically oriented path for aspiring commercial vocational education and training (VET) teachers. Although teachers’ content knowledge (CK) and pedagogical content knowledge (PCK) are considered crucial for teaching quality and students’ achievement, little is known about S...

Background A holistic approach to performance assessment recognizes the theoretical complexity of multifaceted critical thinking (CT), a key objective of higher education. However, issues related to reliability, interpretation, and use arise with this approach. Aims and Method Therefore, we take an analytic approach to scoring students’ written re...

Following employers’ criticisms and recent societal developments, policymakers and educators have called for students to develop a range of generic skills such as critical thinking (“twenty-first century skills”). So far, such skills have typically been assessed by student self-reports or with multiple-choice tests. An alternative approach is crite...

PLATO is conceived as a research initiative among multiple German universities and research institutes, and cooperating universities and researchers globally. PLATO investigates the representation of information on the Internet and social media and its effects on students’ positive learning as a vital, scientifically, morally, and ethically oriente...

The present study investigates hope in the context of a 4 year vocational training. The theoretical framework combines the theory of hope with core ideas of self-determination theory. The main hypothesis predict a positive relationship between general hope and the satisfaction of basic psychological needs in the learning environment at the workplac...

A critical issue in educational evaluation is whether evaluations should focus on standardized (summative, often quantitative) or contextualized (formative or often qualitative) evidence. The author of this article advises readers to beware of false dichotomies. The big issue is not whether evaluations should be “standardized” or “contextualized” b...

We present a short overview of the idea, genesis and developing process of PLATO (Positive Learning in the Age of InformaTiOn) as a new, complex, international, interdisciplinary program aimed at investigating the phenomena of positive and negative learning in the digital age. While the basic idea of PLATO originated in empirical educational resear...

Access to information in mass and social media was once thought of as a universal “leveler” for all. Unfortunately learning opportunities have been corrupted by biased, false, deliberately inaccurate, and unverified information. Negative learning (NL) occurs when Internet users are unable to recognize this. Positive learning (PL), the acquisition o...

About 20 years ago, Jan-Eric Gustafsson (e.g., Gustafsson and Undheim 1996) used confirmatory factor analysis to examine the hierarchical structure of intelligence, paralleling an earlier extraordinarily thorough exploratory factor analysis reported by Carroll (1993). Gustafsson’s model fitting and testing were elegant, he confirmed Carroll’s findi...

Technical schools are an integral part of the education system, and yet, little is known about student learning at such institutions. We consider whether assessments of student learning can be jointly administered to both university and technical school students. We examine whether differential test functioning may bias inferences regarding the rel...

In an increasingly globalized world, the call for internationally comparable competence measurements has emerged. After several international studies on pre-college education, the focus has shifted to international assessments of vocational education and training (VET). VET researchers in Germany developed a computer-based test (ALUSIM) that measur...

Changes in the nature of higher education are leading towards increased interest in the assessment of student learning. This study considers an attempt to apply value-added models for the purposes of comparing student learning across institutions, taking care to discuss special considerations inherent to the application of these models to higher ed...

The authors examine methodological characteristics of summative evaluations in informal science education (ISE), asking: What are the major types of designs used in summative evaluations, and what kinds of questions can they answer? What are the types of data collection methods and measures used, and how many are self-reports or direct measures? Th...

With the Program for International Student Assessment and Trends in International Mathematics and Science Study surveys, competency assessment became an important policy instrument in the school sector; only recently has international competency measurement gained attention in higher education with the Assessment of Higher Education Learning Outcom...

Lee J. Cronbach (1916–2001) had a long and distinguished career as an educational psychologist. He made major contributions to psychological testing, including coefficient alpha, generalizability theory, and construct validity theory; to evaluation, including a focus on program improvement, an emphasis on the role played by context and politics, an...

Generalizability (G) theory is a psychometric theory based on a statistical sampling approach that partitions scores into their underlying multiple sources of variation. It offers a framework that can be used to pinpoint and estimate the source of measurement error from which decisions can be made to optimize the measurement procedures. As an overv...

Summative evaluation plays a critical role in documenting the impacts of informal science education (ISE), potentially contributing to the ISE knowledge base and informing ongoing improvements in practice and decision-making. In response to the growing demand for capacity-building in ISE evaluation, this article presents a framework for summative e...

In a globalized knowledge-based society with increased demands for high levels of workforce competence, informa-tion about higher education's capacity to develop competen-cies is highly necessary. However, before obtaining such information, competencies need to be assessed. Research of this type is only just beginning. The objective of this top-ica...

In this paper, the state of research on the assessment of competencies in higher education is reviewed. Fundamental conceptual and methodological issues are clarified by showing that current controversies are built on misleading dichotomies. By systematically sketching conceptual controversies, competing competence definitions are unpacked (analyti...

The name “SAT” has become synonymous with college admissions testing; it has been dubbed “the gold standard.” Numerous studies on its reliability and predictive validity show that the SAT predicts college performance beyond high school grade point average. Surprisingly, studies of the factorial structure of the current version of today’s SAT, revis...

Chemistry courses remain a challenge for many undergraduate students. In particular, first-semester organic chemistry has been labeled as a gatekeeper with high attrition rates, especially among students of color. Our study examines a key factor related to conceptual understanding in science and predictive of course outcomes—knowledge structures. P...

Problem solving is a highly valued skill in chemistry. Courses within this discipline place a substantial emphasis on problem-solving performance and tend to weigh such performance heavily in assessments of learning. Researchers have dedicated considerable effort investigating individual factors that influence problem-solving performance. The purpo...

This study examined the effect of learning progression-aligned formal embedded formative assessment on conceptual change and achievement in middle-school science. Fifty-two sixth graders were randomly assigned to either an experimental group or a control group. Both groups were taught about sinking and floating by the same teacher with identical cu...

The study utilized a multi-method approach to explore the connection between critical thinking and epistemological beliefs in a specific problem-solving situation. Data drawn from a sample of ten third-year bioscience students were collected using a combination of a cognitive lab and a performance task from the Collegiate Learning Assessment (CLA)....

The shift from cookbook to authentic research-based lab courses in undergraduate biology necessitates the need for evaluation and assessment of these novel courses. Although the biology education community has made progress in this area, it is important that we interpret the effectiveness of these courses with caution and remain mindful of inherent...

This study sought to identify ethnically diverse students' study strategies in organic chemistry and their relationships to course outcomes. Study diaries, concept maps, and problem sets were used to assess study outcomes. Findings show that students engage in four commonly used reviewing-type strategies, regardless of ethnic group affiliation. How...

E. L. Thorndike contributed significantly to the field of educational and psychological testing as well as more broadly to psychological studies in education. This article follows in his testing legacy. I address the escalating demand, across societal sectors, to measure individual and group competencies. In formulating an approach to measuring com...

Based on the changing demands of today’s workforce, advances in other nations, and original analysis, this report provides a set of criteria for high-quality student assessments. These criteria can be used by assessment developers, policymakers, and educators as they work to create and adopt assessments that promote deeper learning of 21stcentury s...

The assessment of competence has emerged as a critical issue as nations seek to develop their education systems, work forces, and their citizens' capacity for life-long learning. While the assessment of competence may be critical to national well-being in the 21st century, as Hartig, Klieme and Leutner (2008, p. v) pointed out, «the theoretical mod...

Learning progressions have captured the imagination and the rhetoric of school reformers and education researchers as one possible elixir for getting K-12 education “on track” (Corcoran et al.’s metaphor, 2009, p. 8). Indeed, the train has left the station and is rapidly gathering speed in the education reform and research communities. As we are co...

This paper reviews the developments in generalizability theory from 1973 to 1980. The first section presents a sketch of generalizability theory. The second section reviews theoretical contributions about (1) problems associated with estimating variance components, including sampling variability and negative estimates, (2) fixed facets, (3) criteri...

Successful completion of organic chemistry is a prerequisite for many graduate and professional programs in science, technology, engineering, and mathematics, yet the failure rate for this sequence of courses is notoriously high. To date, few studies have examined why some students succeed while others have difficulty in organic chemistry. This stu...

Science, Technology, Engineering, and Mathematics (STEM) education has become a key focus in the U.S. government's public education agenda. Many STEM degrees require the successful completion of undergraduate introductory organic chemistry (O-Chem), which is notorious for its difficulty and high attrition rate. Concept Maps (CM) have been used as a...

Science assessment in secondary schools requires instruments and techniques that reveal the depth and complexity of students' science understandings and capabilities. Because students' knowledge, reasoning, and skills vary across science domains and contexts, assessments must encompass a range of tasks, formats, and techniques. This article focuses...

This article adapts the Evidence-Based Reasoning (EBR) Framework (Brown, Furtak, Timms, Nagashima, & Wilson, this issue) to create a coding system for assessing argumentation in science classroom discourse. The instrument, Evidence-Based Reasoning in Science Classroom Discourse, is intended to provide a means for measuring the quality of EBR in who...

Across multiple societal sectors, demand is growing to measure individual and group competencies. This paper unpacks Hartig et al.'s (2008) competency definition as a complex ability construct closely related to real-life-situation performance to make it amenable to measurement. Unpacked following the assessment triangle (construct, observation, in...

We constructed a shell (blueprint) for generating science performance assessments, and evaluated the characteristics of the assessments produced with it. The shell addressed four tasks: Planning, HandsOn, Analysis, and Application. Two parallel assessments were developed, Inclines (IN) and Friction (FR). Two groups of fifth graders who differed in...

The construct, self-concept, has been evoked to explain overt behavior across a wide spectrum of situations, and the attainment of a positive self-concept has been posited as a desirable goal in personality and child development, in clinical treatments, and in education. Its importance notwithstanding, research and evaluations using self-concept ha...

Because test scores are known to contain errors, test developers and administrators must concern themselves with test score consistency (or reliability). Such concerns are commonly investigated using coefficient alpha, an internal consistency reliability coefficient that is computed as follows: urn:x-wiley:9780470479216:media:corpsy0190:xm1 where k...

Accrediting boards, the federal government and state legislatures are now requiring a greater level of accountability from higher education. However, current accountability practices, including accreditation, No Child Left Behind, and performance reporting are inadequate to the task. If wielded indiscriminately, accountability can actually do more...

The 2009 National Assessment of Educational Progress uses an expanded variety of tasks to probe student science achievement.

The objective of many environmental education programs is to promote pro-environmental attitudes and behaviors in students. However, evaluation of these programs has focused on asking participants what they think (attitudes) and what they do (behaviors) regarding the environment problems through self-report questionnaires and interviews. These meth...

If we seek to enhance the quantitative reasoning of the American public, not only do we need to be able to say what quantitative reasoning is, we also need to know how to teach for it and how to measure progress toward the QR goal. My focus is on assessment, but inevitably I also need to address definitional questions. Moreover, teaching and assess...

Assessments associated with learning progressions are designed to provide diagnostic information about the level and nature of student understanding. Valid interpretations of such diagnoses are only possible when students consistently express the ideas associated with a single learning progression level. Latent class analysis was employed to evalua...

The Collegiate Learning Assessment (CLA) program measures value added in colleges and universities, by testing the ability of freshmen and seniors to think logically and write clearly. The program is popular enough that it has attracted critics. In this paper, we outline the methods used by the CLA to determine value added. We summarize the critici...

To test the reliability of concept map assessment, which can be used to assess an individual's "knowledge structure," in a medical education setting. In 2004, 52 senior residents (pediatrics and internal medicine) and fourth-year medical students at the University of California-Davis School of Medicine created separate concept maps about two differ...