Jonathan H Steinberg

• MA, Statistics
• Manager at Educational Testing Service

In a study of 370 postsecondary students in electronics, engineering, and other science classes, we investigated collaborative problem‐solving (CPS) skills that best predict performance at individual levels in an online electronics environment. The results showed that while monitoring was a consistent predictor across levels, other skills such as e...

Collaborative problem solving (CPS) is a critical competency for the modern workforce, as many of todays’ problems require groups to come together to find innovative solutions to complex problems. This has motivated increased interest in work dedicated to assessing and developing CPS skills. However, there has been limited attention in prior CPS as...

Standardizing accessible test design and development to meet students’ individual access needs is a complex task. The following study provides one approach to accessible test design and development using participatory design methods with school community members. Participatory research provides opportunities to empower collaborators by co-creating...

Competency in skills associated with collaborative problem solving (CPS) is critical for many contexts, including school, the workplace, and the military. Innovative approaches for assessing individuals’ CPS competency are necessary, as traditional assessment types such as multiple -choice items are not well suited for such a process-oriented compe...

New challenges in today's world have contributed to increased attention toward evaluating individuals' collaborative problem solving (CPS) skills. One difficulty with this work is identifying evidence of individuals' CPS capabilities, particularly when interacting in digital spaces. Often human-driven approaches are used but are limited in scale. M...

Bottom-up, data-driven response filtering methods that exclude unrealistically fast responses from calculating test scores have been successfully applied to improve test validity. We introduce a top-down, theory-driven method to detect cognitively disengaged behavior, compare it with a data-driven method using data from a nationally representative...

This study discusses the development of a basic electronics knowledge (BEK) assessment as a pretest activity for undergraduate students in engineering and related fields. The 28 BEK items represent 12 key concepts, including properties of serial circuits, knowledge of electrical laws (e.g., Kirchhoff's and Ohm's laws), and properties of digital mul...

In this report we provide preliminary evidence on the measurement characteristics for a new type of teaching performance assessment designed to be combined with complementary assessments of teacher content knowledge. The resulting test, which we refer to as the Foundational Assessment of Competencies for Teaching (FACT), is designed for use as part...

Collaborative problem solving (CPS) is considered a necessary skill for students and workers in the 21st century as the advent of technology requires more and more people to frequently work in teams. In the current study, we employed theoretically-grounded data mining techniques to identify four profiles of collaborative problem solvers interacting...

The primary purpose of this report is to provide preliminary evidence on the measurement properties for newly designed assessments of content knowledge for teaching (CKT) in elementary reading language arts (RLA) and mathematics. The goal is to offer the CKT tests through the PRAXIS® assessment. Additional analyses were conducted to provide initial...

In this research report, we describe the conceptual foundation and measurement properties of the Reading Inventory and Scholastic Evaluation (RISE). The RISE is a 6‐subtest, Web‐administered reading skills components battery. We review the theoretical and empirical foundations of each subtest in the battery, as well as item designs. The results inc...

This study relates collaborative problem solving (CPS) behavior and background characteristics of three-person student teams completing tasks in an online electronics environment to task performance. Task performance was primarily predicted by classroom membership and minimally impacted by CPS communication types and group dynamics. The online envi...

In this paper, we describe a theoretically-grounded data mining approach to identify types of collaborative problem solvers based on students' interactions with an online simulation-based task about electronics concepts. In our approach, we developed an ontology to identify the theoretically-grounded features of collaborative problem solving (CPS)....

Pedagogical agents are widely employed in intelligent tutoring systems and game-based learning environments, but research suggests that learning benefits from virtual agents vary as a function of agent features (e.g., the form or register of agent dialogue) and student characteristics (e.g., prior knowledge). Students’ responses to agent-based conv...

The purpose of this report is to examine the performance of assessment items designed to measure elementary teachers' content knowledge for teaching (CKT) science as part of the ETS® Educator Series. The Elementary Education: CKT Science assessment is 1 component of licensure examination through the PRAXIS® assessments. The Elementary Education: CK...

Background knowledge is a strong predictor of reading comprehension, yet little is known about how different types of background knowledge affect comprehension. The study investigated the impacts of both domain and topic-specific background knowledge on students’ ability to comprehend and learn from science texts. High school students (n = 3,650) c...

This study relates collaborative problem solving (CPS) behavior and background characteristics of three-person student teams completing tasks in an online electronics environment to task performance. Task performance was primarily predicted by classroom membership and minimally impacted by CPS communication types and group dynamics. The online envi...

This study explored the scalability of an innovative assessment approach, conversation-based assessment, to measure constructs related to scientific reasoning, where virtual students interact with students to demonstrate their abilities to use evidence to support a prediction. We designed two parallel tasks using the same design pattern and compare...

In this report, we describe the development of an extension of the SuccessNavigator® assessment for late high school settings. We discuss the assessment's conceptualization and support its application with psychometric studies detailing scale development in terms of structural analyses, reliability, and several other aspects of validity. High Schoo...

Results from a survey of 227 adult English learner (EL) faculty in community and technical colleges in the United States reveal a clear desire to better serve adult ELs, but a lack of resources specifically designed to do so. Faculty want and need more resources to support the teaching and learning process, in the form of thoughtful assessments, ma...

Test providers are increasingly moving toward exclusively administering assessments by computer. Computerized testing is becoming more desirable for test takers because of increased opportunities to test, faster turnaround of individual scores, or perhaps other factors, offering potential benefits for those who may be struggling to pass licensure e...

This report describes the results of a large field trial that was carried out to determine if it is possible to design and develop a two-stage reading assessment that will provide fair and valid scores for students with reading-based learning disabilities. The field trial included both a pretest stage and a field trial stage. The results of the fie...

When designing a reading intervention, researchers and educators face a number of challenges related to the focus, intensity, and duration of the intervention. In this paper, we argue there is another fundamental challenge-the nature of the reading outcome measures used to evaluate the intervention. Many interventions fail to demonstrate significan...

To identify the most important competencies for college graduates to succeed in the 21st century workforce, we conducted an analysis of the Occupational Information Network (O*NET) database. O*NET is a large job analysis operated and maintained by the U.S. Department of Labor. We specifically analyzed ratings of the importance of abilities (52 rati...

This paper describes the design, development, and technical adequacy of the Reading Inventory and Student Evaluation (RISE) form of the Study Aid and Reading Assessment (SARA) reading components computer-delivered assessment. The RISE form, designed for middle-school students, began through a joint project between the Strategic Education Research P...

The TOEFL Junior® Standard Test is an assessment that measures the degree to which middle school-aged students learning English as a second language have attained proficiency in the academic and social English skills representative of English-medium instructional environments. The assessment measures skills in three areas: listening comprehension,...

This study examined the scores on a state standards-based Grade 5 Science assessment obtained by a group of students without learning disabilities who took the standard form of the test and by three groups of students with learning disabilities: one taking the standard form of the test without accommodations or modifications, a second taking the te...

In this research investigation of score comparability for language minority students (English language learners [ELLs] and former English language learners), we examined 3 indicators of score comparability (reliability, internal test structure, and differential item functioning) for 4th and 8th grade students who took the NCLB-mandated content asse...

This report focuses on two aspects of teacher supply and quality: (a) the under‐representation of racial/ethnic minorities, especially African Americans in the teaching pool, and (b) teacher candidates' performance on licensure assessments, including general skills tests in reading, writing, and mathematics (known as Praxis I ® ) and selected tests...

To date, assessment validity research on non-native English speaking students in the United States has focused exclusively on those who are presently English language learners (ELLs). However, little, if any, research has been conducted on two other sizable groups of language minority students: (a) bilingual or multilingual students who were alread...

This study compared the underlying factors measured by a state standards-based grade 4 English-Language Arts (ELA) assessment given to several groups of students. The focus of the research was to gather evidence regarding whether or not the tests measured the same construct or constructs for students without disabilities who took the test under sta...

Abstract The purpose of this study was to investigate the impact of a read-aloud test change administered with the Gates-MacGinitieReading Test (GMRT) on the underlying constructs measured,by the Comprehension,subtest. The study evaluated the factor structures for the Level 4 Comprehension subtest given to a sample of New Jersey fourth-grade studen...

English language learners (ELLs) constitute one of the fastest growing subpopulations of students in the United States. It is important to determine whether the assessments used by states in determining students' proficiencies are valid and fair for ELLs. This study focused on several standards-based assessments in mathematics and science administe...