Michael Russell

• Boston College

This poster and accompanying data dashboard explore the results of a May 2021 parent survey documenting COVID-era practices and beliefs. As part of a larger multi-year study examining longitudinal use and impacts for one of the nation’s largest community broadband initiatives, these parent survey results provide an insightful, important, and often...

Although both content alignment and standard-setting procedures rely on content-expert panel judgements, only the latter employs discussion among panel members. This paper examines how a consensus approach to content alignment can enhance the validity of panel judgements. Findings suggest that informed discussion results in more accurate judgements...

Although both content alignment and standard-setting procedures rely on content-expert panel judgements, only the latter employs discussion among panel members. This study employed a modified form of the Webb methodology to examine content alignment for twelve tests administered as part of the Massachusetts Comprehensive Assessment System (MCAS). T...

Over the past decade, large-scale testing programs have employed technology-enhanced items (TEI) to improve the fidelity with which an item measures a targeted construct. This paper presents findings from a review of released TEIs employed by large-scale testing programs worldwide. Analyses examine the prevalence with which different types of TEIs...

This document presents findings from a content alignment study conducted by members of Boston College's Department of Measurement, Evaluation, Statistics, and Assessment. The study focuses on the content alignment of the 2017 Massachusetts Comprehensive Assessment System (MCAS) Grades 3-8 English Language Arts (ELA) and Mathematics tests to the the...

This paper examines current practice regarding the use of technology-enhanced items in assessment programs worldwide, with particular focus on utility. Findings suggest that a variety of TEIs are being used by testing programs. Content-specific interactions add measurement value in the form of fidelity, while generic interactions often do not.

2017 marked a century since the development and administration of the first large-scale group administered standardized test. Since that time, both the importance of testing and the technology of testing have advanced significantly. This paper traces the technological advances that have led to the large-scale administration of educational tests in...

This paper presents findings from a descriptive study that examines current practices regarding the use of technology-enhanced items by operational K-12 testing programs worldwide. The paper aims to inform the field about current practice and the measurement utility of the current use of technology-enhanced items. The primary focus of utility in th...

The last few years have seen rapid adoption of computer-based testing across a majority of states. A variety of technology-enhanced items (TEI) have been introduced to state testing programs. Several state testing programs have experimented with automated scoring. Many testing programs and test vendors have embraced the concept of interoperability...

2017 marked a century since the development and administration of the first large-scale group administered standardized test. Since that time, both the importance of testing and the technology of testing have advanced significantly. This paper traces the technological advances that have led to the large-scale administration of educational tests in...

This paper depicts and summarizes the current practice on the use of technology-enhanced items in large-scale educational programs around the world. Through an extensive survey of items the paper discusses the utility and fidelity of these items and the implications this has on item development. Findings suggest that technology enhanced items are m...

Despite the ever-increasing number of online professional development (OPD) programs, relatively few studies have been conducted to examine the efficacy of such programs for teachers and students. This manuscript presents findings from an impact study of OPD courses infractions, algebraic thinking, and measurement on 79 fifth grade teachers’ pedago...

Analyses presented here are secondary data analyses of the Use, Support and Effect of Instructional Technology study aimed at identifying predictors of teacher-directed student use of technology (TDS) in elementary classrooms. Using data from a convenience sample of 1040 teachers nested within 81 schools in 21 Massachusetts' school districts, resea...

Online professional development (OPD) has potential to improve teacher quality by improving teachers' knowledge and instructional practices. These changes, in turn, have potential to improve student achievement. Unfortunately, there is a dearth of scientific research on the effects of OPD on teachers and, more importantly, on students. This article...

This study is a secondary data analysis of the USEIT data to inform school administrators and policymakers about the factors affecting instructional technology use in elementary classrooms. Researchers developed a predictive multilevel SEM model for teacher-directed student use of technology (TDS). The model depicts relationships between factors ac...

The presence of computers in our schools creates valuable opportunities to enhance educational testing. The number of computers in schools has changed dramatically over the last three decades. In 1983, schools had, on average, only one computer for every 125 students. Today, that ratio has dropped to one computer for every 3.8 students and nearly e...

Personalization has become a powerful approach for engaging people with products and services. Today, our children can walk into a Build-A-Bear store and in minutes create a personalized stuffed animal. We can log onto Nike.com and build a pair of shoes using our favorite color patterns and slogans. With the click of a few buttons, we can access lo...

Online professional development (OPD) has great potential to improve teacher quality by improving teachers' knowledge and instructional practices, with the ultimate goal of improving student achievement. There is a dearth of scientific research, however, on the effects of OPD. This article presents the results of a randomized control trial explorin...

In the following pages, we examine some common methodological challenges in educational technology research and highlight new data collection approaches using examples from the literature and our own work. Given that surveys and questionnaires remain widespread and dominant tools across nearly all studies of educational technology, we first discuss...

The study investigated whether online professional development courses with different levels of support have different impacts on teacher outcomes. Variations of an online course for middle school algebra teachers were created for four experimental conditions. One was a highly supported condition, with a math education instructor, an online facilit...

Computer-based diagnostic assessment systems hold potential to help teachers identify sources of poor performance and to connect teachers and students to learning activities designed to help advance students' conceptual understandings. The present article presents findings from a study that examined how students' performance in algebra and their ov...

The study compared the effects of a professional development course delivered in an online and a face-to-face format. The effects examined included changes in teachers' pedagogical beliefs, instructional practices, and understanding of teaching number-sense and related mathematical concepts. The study randomly assigned participants to either the on...

Over the past decade, standardized test results have become the primary tool used to judge the effectiveness of schools and educational programs, and today, standardized testing serves as the keystone for educational policy at the state and federal levels. This paper examines the relationship between fourth grade mathematics achievement and technol...

In spite of large expenditures on and increased access to educational technologies, a concern remains that computer-based technologies are not being integrated into regular instructional practices. While there is evidence to support the hypothesis that newer teachers' familiarity with technology leads to increased technology integration, a question...

Investment in educational technology has increased rapidly in recent years and many observers have begun to question whether, and how technology is being used as a teaching and learning tool. In order to address this issue, this research used survey data collected from 1,404 middle and high school teachers in 52 schools across 22 Massachusetts scho...

With increased emphasis on test-based accountability measures has come increased interest in examining the impact of technology use on students’ academic performance. However, few empirical investigations exist that address this issue. This paper (1) examines previous research on the relationship between student achievement and technology use, (2)...

Investment in educational technology has increased rapidly in recent years and many observers have begun to question whether, and how technology is being used as a teaching and learning tool. In order to address this issue, this research used survey data collected from 1,404 middle and high school teachers in 52 schools across 22 Massachusetts scho...

Across the nation and at nearly all levels of our educational system, efforts to hold schools accountable for student learning dominate strategies for improving the quality of education. At both the national and state level, student testing stands at the center of educational accountability programs, such that schools are effectively held accountab...

Across the nation and at nearly all levels of our educational system, efforts to hold schools accountable for student learning dominate strategies for improving the quality of education. At both the national and state level, student testing stands at the center of educational accountability programs, such that schools are effectively held accountab...

In this article, Walt Haney, Michael Russell, and Damian Bebell summarize a decade of work using student drawings as a way to both document and change education and schooling. After a brief summary of more than one hundred years of literature on children's drawings, the authors point out that drawings have been little recognized as a medium of educ...

In the last 20 years, substantial investments have been made in educational technology. Not surprisingly, in today’s zeitgeist of educational accountability there have been increasing calls for empirical, research-based evidence that these investments are affecting the lives of teachers and students. This paper examines the ways in which teachers u...

This study compares teaching and learning activities in 4th and 5th grade classrooms that were permanently equipped with one laptop for each student and classrooms that share a cart of laptops that create a 1:1 laptop environment on a temporary basis. The study originated from a question posed to us by Andover Public Schools (MA): "How does teachin...

Over the past two decades, the presence and use of technology in the workplace and in schools has increased dramatically. At the same time, the importance of test-based educational accountability has also increased. Currently, formal testing programs are used in 49 states. Some observers have raised concerns that the testing programs that make high...

Over the past two decades, the presence and use of technology in the workplace and in schools has increased dramatically. At the same time, the importance of test-based educational accountability has also increased. Currently, formal testing programs are used in 49 states. Some observers have raised concerns that the testing programs that make high...

Over the past decade, investment in technology for schools has increased at a dramatic rate. Although policy makers are eager to understand the ways in which technology use in schools is affecting student learning, we believe that a critical preliminary step toward assessing the impacts of technology on teaching and learning requires the examinatio...

To examine the impact of transitioning 4th grade reading comprehension assessments to the computer, 219 fourth graders were randomly assigned to take a one-hour reading comprehension assessment on paper, on a computer using scrolling text to navigate through passages, or on a computer using paging text to navigate through passages. This study exami...

This study replicates and extends the work of Powers et al. (1994) by examining the influence computer-print and handwriting have on raters' scores. This replication study employs an experimental design that presents the same set of responses to raters in four different formats. A second experiment is conducted to explore the extent to which the pr...

As elementary and secondary school-level testing programs begin administering tests on computers, several issues related to the validity of using test scores provided by computer-based tests to make decisions about student and school performance must be examined. In this paper, we describe how efforts to examine the validity of computer-based test...

As access to computer-based technology in schools and classrooms increases, greater emphasis has been placed on preparing teachers to use technology for instructional purposes. Survey data collected from 2,894 teachers in 22 Massachusetts districts were analyzed to examine the extent to which technology is used in and out of the classroom for instr...

Research shows that regular use of computers for writing over an extended period of time can have a positive impact on the quantity and quality of student writing. The lack of large numbers of computers in schools and in classrooms presents a major impediment to providing students with regular access to computers. The introduction of laptops and/or...

Meta-analyses were performed including 26 studies conducted between 1992–2002 focused on the comparison between K–12 students writing with computers vs. paper-and-pencil. Significant mean effect sizes in favor of computers were found for quantity of writing (d=.50, n=14) and quality of writing (d= .41, n=15). Studies focused on revision behaviors b...

Results from a national survey of teachers are reported for five types of state testing programs, those with: (1) high stakes for districts, schools, or teachers, and students; (2) high stakes for districts, schools, and teachers, and moderate stakes for students; (3) high stakes for districts, schools, and teachers, and low stakes for students; (4...

The Use, Support, and Effect of Instructional Technology (USEIT) Study was undertaken to better understand: a) how educational technologies are being used by teachers and students in the classroom, b) what factors influence these uses, and c) how the use of technology effects student learning. The three-year study began during the Spring of 2001 an...

This paper argues that the inexorable advance of technology will force fundamental changes in the format and content of assessment. Technology is infusing the workplace, leading to widespread requirements for workers skilled in the use of computers. Technology is also finding a key place in education. This is occurring not only because technology s...

This study builds on three previous studies (Russell, 1999; Russell & Haney, 1997; Russell & Plati, 2001) to examine the effect of administering extended composition test items on paper, on computer, or on a portable writing device has on student performance. This study employs writing items from the 1999 Massachusetts Comprehensive Assessment Syst...

This paper addresses problems with performance on open-ended (i.e., not multiple choice) test questions administered on paper for students accustomed to writing with computers. The first section summarizes the results of three studies that examined differences between computer and paper-and-pencil test results. The second section identifies three r...

This paper was presented as expert testimony in the Williams vs. State of California class action lawsuit. That case, filed on behalf of California public schoolchildren, charged the State with denying thousands of students the basic tools for a sound education. This paper addresses whether California's current output-based accountability system ac...

This article builds on two previous studies (Russell, 1999; Russell & Haney, 1997) that explored the effect mode of administration, that is computer versus paper-andpencil, has on student performance on open-ended items requiring written responses. Whereas the two previous studies have focused on middle school students, the study presented here foc...

The widening gap between the increased use of technology in schools and the absence of computers in state-level testing programs raises important implications for policies related to the use of both technology and testing in schools. In this article, we summarize recent developments in the use of technology in schools and in state level testing pro...

An earlier Digest described the shortcomings of three methods commonly used to summarize changes in test scores. This Digest describes two less commonly used approaches for examining changes in test scores, those of Standardized Growth Estimates and Effect Sizes. Aspects of these two approaches are combined and applied to the Iowa Test of Basic Ski...

This Digest introduces the advantages and disadvantages of three commonly used methods of reporting test score changes: (1) change in percentile rank; (2) scale or raw score change; and (3) percent change. The change in percentile rank method focuses on the increase or decrease of the mean percentile ranking for a group of students. This method has...

Russell and Haney (1997) reported that open-ended test items administered on paper may underestimate the achievement of students accustomed to writing on computers. This study builds on Russell and Haney's work by examining the effect of taking open-ended tests on computers and on paper for students with different levels of computer skill. Using it...

Russell and Haney (1997) reported that open-ended test items administered on paper may underestimate the achievement of students accustomed to writing on computers. This study builds on Russell and Haney's work by examining the effect of taking open-ended tests on computers and on paper for students with different levels of computer skill. Usi...

Computer use has grown rapidly during the past decade. Within the educational community, interest in authentic assessment has also increased. To enhance the authenticity of tests of writing, as well as of other knowledge and skills, some assessments require students to respond in written form via paper-and-pencil. However, as increasing numbers of...

Computer use has grown rapidly during the past decade. Within the educational community, interest in authentic assessment has also increased. To enhance the authenticity of tests of writing, as well as of other knowledge and skills, some assessments require students to respond in written form via paper-and-pencil. However, as increasing numbers of...

The results of a small research project that studied the effect computer administration has on student performance for writing or essay tests are presented. The introduction of computer-administered tests has raised concern about the equivalence of scores generated by computer versus paper-and-pencil test versions. For this study a sample of studen...

Many students who are deaf or hard-of-hearing are eligible for a signing accommodation for state and other standardized tests. The signing accommodation, however, presents several challenges for testing programs that attempt to administer tests under standardized conditions. One potential solution for many of these challenges is the use of computer...

This research examines how technology is being used by elementary school teachers, and examines the school and district characteristics that are associated with the use of technology in the classroom.

Over the past decade, students' use of computers has increased sharply, particularly for writing and research (Becker, 1999; Russell, O'Brien, Bebell, & O'Dwyer, 2003). At the same time, the use of large-scale tests to make decisions about schools and students has exploded. But, in addition to making decisions about students and their schools, resu...

The study presented here responds directly to the call for research-based evidence that the use of word processors has a positive impact on student writing. As is described more fully below, the study presented here employs meta-analytic techniques, commonly used in fields of medicine and economics, to integrate the findings of studies conducted be...

The widening gap between the increased use of technology in schools and the absence of computers in state-level testing programs raises important implications for policies related to the use of both technology and testing in schools. In this article, we summarize recent developments in the use of technology in schools and in state level testing pro...

Over the past decade, the presence of and access to computer-based technology in K-12 schools has increased rapidly. In turn, computer-based technologies are changing the tools with which teachers teach and students learn. As computer-based tools continue to evolve and become more prevalent in K-12 classrooms, their use provides challenges to and o...

Technology and testing have become two popular prescriptions for improving education. The technology nostrum suggests that the infusion of modern technology into schools will bolster teaching and learning and will prepare students for an increasingly technological workplace. The testing prescription holds that using standardized test scores to rate...

The studies reported here focus on the effects of mode of administration of the Massachusetts Comprehensive Assessment System (MCAS) in grades 4, 8, and 10. These studies, which draw on earlier studies, also examine the mode of administration effect at different levels of keyboarding speed and for students who received special education (SPED) acco...