Rebecca Zwick’s research while affiliated with Educational Testing Service and other places

Today, postsecondary institutions in the US typically wish to enroll entering classes that are both academically qualified and diverse. Although the definition of diversity varies from school to school, the challenge is essentially the same: How can academic objectives be combined with goals that involve the composition of the entering class? Many schools have a commitment to facilitating access for under-represented minorities or low-income applicants, or for members of nearby communities. Incorporating these goals while maintaining academic standards can be challenging.

Higher Education Admissions Practices - edited by María Elena Oliveri January 2020

Selection decisions have a major impact on our education, occupation, and quality of life, and the role of standardized tests in selection has always been a source of controversy. Here, I consider various definitions of fairness in measurement and selection—those emerging from within educational measurement and statistics, those from philosophy, and finally, those from the public. I use examples of public challenges to selection practices to illustrate the fact that technical and philosophical definitions of fairness do not align well with public concerns. I emphasize the importance of promoting awareness of existing standards, advocating for the fair use of testing and selection practices, and communicating in a candid and straightforward way when engaging with test takers and test users.

In US colleges, the scarcity of students from low-income families is a major concern. We present a novel way of boosting the percentage of qualified low-income students using constrained optimization (CO), an operations research technique. CO allows incorporation of both academic requirements and diversity goals in college admissions. The incoming class’s academic credentials are maximized while constraints on class composition are imposed. In particular, the percentage of students in a certain demographic group can be required to exceed a minimum. In an illustrative analysis, we show how CO can be used to increase the proportion of admitted students from low-income neighborhoods.

In this article, I review the role of college admissions tests in the United States and consider the fairness issues surrounding their use. The two main tests are the SAT, first administered in 1926, and the ACT, first given in 1959. Scores on these tests have been shown to contribute to the prediction of college performance, but their role in the admissions process varies widely across colleges. Although test scores are consistently listed as one of the most important admissions factors in national surveys of postsecondary institutions, an increasing number of schools have adopted “test-optional” policies. At these institutions, test score requirements are seen as a barrier to campus diversity because of the large performance gaps among ethnic and socioeconomic groups. Fortunately, the decentralized higher education system in the United States can accommodate a wide range of admissions policies. It is essential, however, that the impact of admissions policy changes be studied and that the resource implications of these changes be thoroughly considered.

In typical differential item functioning (DIF) assessments, an item's DIF status is not influenced by its status in previous test administrations. An item that has shown DIF at multiple administrations may be treated the same way as an item that has shown DIF in only the most recent administration. Therefore, much useful information about the item's functioning is ignored. In earlier work, we developed the Bayesian updating (BU) DIF procedure for dichotomous items and showed how it could be used to formally aggregate DIF results over administrations. More recently, we extended the BU method to the case of polytomously scored items. We conducted an extensive simulation study that included four “administrations” of a test. For the single-administration case, we compared the Bayesian approach to an existing polytomous-DIF procedure. For the multiple-administration case, we compared BU to two non-Bayesian methods of aggregating the polytomous-DIF results over administrations. We concluded that both the BU approach and a simple non-Bayesian method show promise as methods of aggregating polytomous DIF results over administrations.

The goal of this study was to explore the effectiveness of a short web-based tutorial in helping teachers to better understand the portrayal of measurement error in test score reports. As described below, the short video tutorial included both verbal and graphical representations of measurement error. Results showed a significant difference in comprehension scores between each of two tutorial groups (basic and enhanced) and the control group (no tutorial), but not between the two tutorial groups. Results also provided evidence of teachers' misconceptions about the meaning of measurement error and confidence bands.

Research has shown that many educators do not understand the terminology or displays used in test score reports and that measurement error is a particularly challenging concept.We investigated graphical and verbal methods of representing measurement error associated with individual student scores. We created four alternative score reports, each constituting an experimental condition, and randomly assigned them to research participants. We then compared comprehension and preferences across the four conditions. In our main study, we collected data from 148 teachers. For comparison, we studied 98 introductory psychology students. Although we did not detect statistically significant differences across conditions, we found that participants who reported greater comfort with statistics tended to have higher comprehension scores and tended to prefer more informative displays that included variable-width confidence bands for scores. Our data also yielded a wealth of information regarding existing misconceptions about measurement error and about score-reporting conventions.

Differential item functioning (DIF) analysis is a key component in the evaluation of the fairness and validity of educational tests. Although it is often assumed that refinement of the matching criterion always provides more accurate DIF results, the actual situation proves to be more complex. To explore the effectiveness of refinement, we conducted a simulation study consisting of 40 conditions that varied in terms of amount and pattern of DIF, sample sizes, and ability distributions. We found that the effectiveness of refinement was heavily dependent on whether DIF was balanced (with positive DIF values compensating for negative DIF values) or unbalanced (all in one direction). In balanced conditions, the unrefined method generally produced better results, whereas, in unbalanced conditions, the opposite was true. In the absence of information about the pattern of DIF, it is probably best to choose the refined method because it is only slightly disadvantageous in balanced conditions, whereas the unrefined method can be substantially disadvantageous in certain unbalanced conditions.