Paul Douglas Deane

Educational Testing Service | ETS · CBAL Research Group

Writing is a critical skill people need to survive and prosper in the modern economy. But most people fail to become competent writers. Those who succeed (at least, in English-speaking countries) are predominantly White, upper middle class, and female. These achievement gaps are primarily the result of opportunity gaps – in other words, they repres...

This paper develops and validates a hierarchical automated writing trait model that combines features used to conduct automated essay scoring, readability analysis, and genre classification. It develops the model using NLP features extracted from 1.37 million essays submissions to Criterion, ETS' digital writing service, and validates it on a varie...

Argumentation skills are emphasized by the common core state standards (CCSS) and are viewed as essential for success in college, career, and life. Our project aims to develop formative assessment tasks measuring students’ argumentative reading and writing skills. We used the framework of the Discuss and Debate Ideas key practice (describing the ke...

Grouping individuals according to a set of measured characteristics, or profiling, is frequently used in describing, understanding, and acting on a phenomenon. The advent of computer‐based assessment offers new possibilities for profiling writing because aspects can be captured that were not heretofore observable. We explored whether writing proces...

Effectively evaluating the credibility and accuracy of multiple sources is critical for college readiness. We developed 24 source evaluation tasks spanning four predicted difficulty levels of a hypothesized learning progression (LP) and piloted these tasks to evaluate the utility of an LP-based approach to designing formative literacy assessments....

Biometrics are physical or behavioral human characteristics that can be used to identify a person. It is widely known that keystroke or typing dynamics for short, fixed texts (e.g., passwords) could serve as a behavioral biometric. In this study, we investigate whether keystroke data from essay responses can lead to a reliable biometric measure, wi...

Educators need actionable information about student progress during the school year. This paper explores an approach to this problem in the writing domain that combines three measurement approaches intended for use in interim-assessment fashion: scenario-based assessments (SBAs), to simulate authentic classroom tasks, automated writing evaluation (...

Keystroke logs provide a comprehensive record of observable writing processes. Previous studies examining the keystroke logs of young L1 English writers performing experimental writing tasks have identified writing processes features predictive of the quality of responses. Contrarily, large-scale studies on the dynamic and temporal nature of L2 wri...

A key instructional goal of English language arts instruction is teaching students to read and interpret complex literary texts. is report reviews the literature on the development and pedagogy of literary analysis skills. It analyzes literary analysis skills as a key practice -- a bundle of disciplinary skills and strategies that form a key targe...

We evaluate how higher- vs. lower-scoring middle-school students differ in their composition processes when writing persuasive essays from source materials. We examined differences on four individual process features–time taken before beginning to write, typing speed, total time spent, and number of words started. Next, we examined differences for...

This study investigates the effects of a scenario-based assessment design on students' writing processes. An experimental data set consisting of four design conditions was used in which the number of scenarios (one or two) and the placement of the essay task with respect to the lead-in tasks (first vs. last) were varied. Students' writing processes...

A theoretically driven approach to assessment of, as, and for learning led to a scenario-based assessment (SBA) design to engage and assist students in writing. The structure of the SBA simulates a condensed writing project undertaken in an order that a skilled practitioner might follow. In this study, we analyzed students' keystroke logs and inves...

One of the major goals of the English Language Arts is to teach students to read, understand, and write narratives. This report examines the ways in which the skills that support narrative develop during the school years, outlines a model of narrative as a “key practice” in which the ability to model social situations supports narrative understandi...

We used an unobtrusive approach, keystroke logging, to examine students’ cognitive states during essay writing. Based on data contained in the logs, we classified writing process data into three states: text production, long pause, and editing. We used semi-Markov processes to model the sequences of writing states and compared the state transition...

This paper presents a theoretical and empirical case for the value of scenario-based assessment (SBA) in the measurement of students’ written argumentation skills. First, we frame the problem in terms of creating a reasonably efficient method of evaluating written argumentation skills, including for students at relatively low levels of competency....

There is a growing literature on the use of process data in digitally delivered assessments. In this study, we analyzed students’ essay writing processes using keystroke logs. Using four basic writing performance indicators, writers were grouped into four clusters, representing groups from fluent to struggling. The clusters differed significantly o...

Analysis of keystroke logging data is of increasing interest, as evident from a substantial amount of recent research on the topic. Some of the research on keystroke logging data has focused on the prediction of essay scores from keystroke logging features, but linear regression is the only prediction method that has been used in this research. Dat...

The research on using event logs and item response time to study test‐taking processes is rapidly growing in the field of educational measurement. In this study, we analyzed the keystroke logs collected from 761 middle school students in the United States as they completed a persuasive writing task. Seven variables were extracted from the keystroke...

This study compared gender groups on the processes used in writing essays in an online assessment. Middle‐school students from four grades responded to essays in two persuasive subgenres, argumentation and policy recommendation. Writing processes were inferred from four indicators extracted from students’ keystroke logs. In comparison to males, on...

Writing from source text is critical for developing college-and-career readiness because it is required in advanced academic environments and many vocations. Scenario-based assessment (SBA) represents one approach to measuring this ability. In such assessment, the scenario presents an issue that the student is to read and write about. Before writin...

The purpose of the report is to explore some of the mechanisms involved in the writing process. In particular, we examine students' process data (keystroke log analysis) to uncover how students approach a knowledge‐telling task using 2 different task types. In the first task, students were asked to list as many words as possible related to a partic...

This article argues for a conceptualization of school writing that emphasizes how cognitive and sociocultural factors interact. In academic, professional, and affinity-based communities, an emphasis on standards of quality drives the adoption of such practices as revision and peer and expert review. In school, everyday writing practices center arou...

In this study (ETS Research Memorandum RM-18-10), we examine the extent to which education agencies and assessment organizations assess the writing process in existing large-scale assessments, and where they do, what aspects of the writing process they assess and what types of assessments they use. Over the past 40 years, standards for teaching and...

Writing process logs (keystroke logs) provide an excellent source of information about how writers have distributed their time and attention across the course of a writing task. However, relatively little is known about how the features that can readily be collected from such logs vary as a result of changes in writing task demands. In this study,...

The goal of this study is to model pauses extracted from writing keystroke logs as a way of characterizing the processes students use in essay composition. Low-level timing data were modeled, the interkey interval and its subtype, the intraword duration , thought to reflect processes associated with keyboarding skills and composition fluency. Heavy...

This study assessed 1,706 eighth-grade students' reasoning abilities through a critique task in an integrated argumentative reading and writing test. Results indicate that the majority of students did not detect fallacious arguments or clearly explain problems in the arguments. They encountered various challenges in critiquing an argument: (a) bein...

We evaluated whether a scenario-based assessment structure with a theoretically determined task order has an impact on the writing processes that students execute in responding to an essay task. Students’ writing processes were recorded via keystroke logs. Two testing conditions were compared, one with the original task ordering and the other with...

This project focuses on laying the foundations for automated analysis of argumentation schemes, supporting identification and classification of the arguments being made in a text, for the purpose of scoring the quality of written analyses of arguments. We developed annotation protocols for 20 argument prompts from a college-level test under the fra...

The report is the first systematic evaluation of the sentence equivalence item type introduced by the GRE revised General Test. We adopt a validity framework to guide our investigation based on Kane's approach to validation whereby a hierarchy of inferences that should be documented to support score meaning and interpretation is evaluated. We prese...

Vocabulary development is essential for learning, but conventional vocabulary assessments lack the range and flexibility to support K–12 classroom teachers in making instructional decisions. Drawing on linguistics, educational psychology, and educational measurement, this book offers a fresh perspective on word learning and describes powerful, prec...

This article exemplifies how assessment design might be grounded in theory, thereby helping to strengthen validity claims. Spanning work across multiple related projects, the article first briefly summarizes an assessment system model for the elementary and secondary levels. Next the article describes how cognitive-domain theory and principles are...

Keystroke logs are a valuable tool for writing research. Using large samples of student responses to two prompts targeting different writing purposes, we analyzed the longest 25 inter-word intervals in each keystroke log. The logs were extracted using the ETS keystroke logging engine. We found two distinct patterns of student writing processes asso...

Current educational standards call for students to engage in the skills of research and inquiry, with a focus on gathering evidence from multiple information sources, evaluating the credibility of those sources, and writing an integrated synthesis that cites evidence from those sources. Opportunities to build strong research skills are critical, ye...

In this paper, we provide a comprehensive literature review on the development of key argumentation skills to lay a foundation for a framework of the key practice, discuss and debate ideas, which is centrally involved in the expectations for academic reading and writing. Specifically, the framework includes 5 phases of core activities and related s...

In educational measurement contexts, essays have been evaluated and formative feedback has been given based on the end product. In this study, we used a large sample collected from middle school students in the United States to investigate the factor structure of the writing process features gathered from keystroke logs and the association of that...

In this report, we examine the feasibility of characterizing writing performance using process features derived from a keystroke log. Using data derived from a set of CBAL™ writing assessments, we examine the following research questions: (a) How stable are the keystroke timing and process features across testing occasions? (b) How consistent are t...

In this paper we provide the rationale and foundation for the building and sharing knowledge key practice for the CBAL™ English language arts competency model. Building and sharing knowledge is a foundational literacy activity that enables students to learn and communicate what they read in texts. It is a strategic process that involves the integra...

This paper presents a framework intended to link the following assessment development concepts into a systematic framework: evidence-centered design (ECD), scenario-based assessment (SBA), and assessment of, for, and as learning. The context within which we develop this framework is the English language arts (ELA) for K-12 students, though the fram...

This paper presents a principled approach to assessment design in which major design decisions are structured to support teaching and learning. This approach, developed as part of a long-term research initiative at ETS, Cognitively Based Assessments of, for and as Learning (CBAL), draws upon the learning and cognitive science literatures to create...

We investigate methods for studying learning progressions in English language arts using data from scenario-based assessments. Particularly, our interest lies in the empirical recovery of learning progressions in argumentation for middle school students. We collected data on three parallel assessment forms that consist of scenario-based task sets w...

This paper explores automated methods for measuring features of student writing and determining their relationship to writing quality and other features of literacy, such as reading rest scores. In particular, it uses the e-rater® automatic essay scoring system to measure product features (measurable traits of the final written text) and features e...

We expect that word knowledge accumulates gradually. This article draws on earlier approaches to assessing depth, but focuses on one dimension: richness of semantic knowledge. We present results from a study in which three distinct item types were developed at three levels of depth: knowledge of common usage patterns, knowledge of broad topical ass...

This study examined automated essay scoring for experimental tests of writing from sources. These tests (part of the CBAL research initiative at ETS) embed writing tasks within a scenario in which students read and respond to sources. Two large-scale pilots are reported: One was administered in 2009, in which four writing assessments were piloted,...

This report describes the foundation and rationale for a framework designed to measure reading literacy. The aim of the effort is to build an assessment system that reflects current theoretical conceptions of reading and is developmentally sensitive across a prekindergarten to 12th grade student range. The assessment framework is intended to docume...

Systems and methods for detecting collocation errors in a text sample using a reference database from a corpus are provided. Collocation candidates are identified within the text sample based upon syntactic patterns in the text sample. Whether a given collocation candidate contains a collocation error is detected, the detecting including: determini...

This paper examines the construct measured by automated essay scoring (AES) systems. AES systems measure features of the text structure, linguistic structure, and conventional print form of essays; as such, the systems primarily measure text production skills. In the current state-of-the-art, AES provide little direct evidence about such matters as...

This study proposes an approach to automatically score the TOEIC® Writing e-mail task. We focus on one component of the scoring rubric, which notes whether the test-takers have used particular speech acts such as requests, orders, or commitments. We developed a computational model for automated speech act identification and tested it on a corpus of...

The Fall 2007 and Spring 2008 pilot tests for the CBAL™ Writing assessment included experimental keystroke logging capabilities. This report documents the approaches used to capture the keystroke logs and the algorithms used to process the outputs. It also includes some preliminary findings based on the pilot data. In particular, it notes that the...

In educational settings, assessment targets determine the need for local validation. Instructional improvement, for example, is validated by examining the relationship between curricular innovations and improvements in criterion measures such as course grades. In such cases, as both the educational measure-ment community (Cizek, 2008; Shepard, 2006...

Proceedings of the Thirteenth Annual Meeting of the Berkeley Linguistics Society (1987), pp. 65-76

Most work on evaluation of named-entity recognition has been done in the context of competitions, as a part of Information Extraction. There has been little work on any form of extrinsic evaluation, and how one tagger compares with another on the major classes: PERSON, ORGANIZATION, and LOCATION. We report on a comparison of three state-of-the-art...

This paper presents a socio-cognitive framework for connecting writing pedagogy and writing assessment with modern social and cognitive theories of writing. It focuses on providing a general framework that highlights the connections between writing competency and other literacy skills; identifies key connections between literacy instruction, writin...

ETS has recently instituted the Cognitively Based Assessments of, for, and as Learning (CBAL) research initiative to create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math. This paper is concerned with the writi...

A particular application of corpus analysis, automated essay scoring (AES) can reveal much about students’ writing skills. In this article we present research undertaken at Educational Testing Service (ETS) as part of its ongoing commitment to developing effective AES systems. AES systems have certain advantages. They can: (a) produce scores simila...

This paper undertakes a review of the literature on writing cognition, writing instruction, and writing assessment with the goal of developing a framework and competency model for a new approach to writing assessment. The model developed is part of the Cognitively Based Assessments of, for, and as Learning (CBAL) initiative, an ongoing research pro...

For many purposes, it is useful to collect a corpus of texts all produced to the same stimulus, whether to measure performance (as on a test) or to test hypotheses about population differences. This paper examines several methods for measuring similarities in phrasing and content and demonstrates that these methods can be used to identify populatio...

This paper describes the first prototype of an automated tool for detecting collocation errors in texts written by non-native speakers of English. Candidate strings are extracted by pattern matching over POS-tagged text. Since learner texts often contain spelling and morphological errors, the tool attempts to automatically correct them in order to...

The ability to distinguish statistically dif- ferent populations of speakers or writers can be an important asset in many NLP applications. In this paper, we describe a method of using document similarity measures to describe differences in be- havior between native and non-native speakers of English in a writing task.1

This paper presents a novel method of generating word similarity scores, using a term by n-gram context matrix which is compressed using Singular Value Decomposition, a statistical data analysis method that extracts the most significant components of variation from a large data matrix, and which has previously been used in methods like Latent Seman...

One source of potential difficulty for struggling readers is the variability of texts across grade levels. This article explores the use of automatic natural language processing techniques to identify dimensions of variation within a corpus of school-appropriate texts. Specifically, we asked: Are there identifiable dimensions of lexical and discour...