Article

Exploring the potential of process-tracing technologies to support assessment for learning of L2 writing

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Assessment for learning (AfL) seeks to support instruction by providing information about students’ current state of learning, the desired end state of learning, and ways to close the gap. AfL of second-language (L2) writing faces challenges insofar as feedback from instructors tends to focus on written products while neglecting most of the processes that gave rise to them, such as planning, formulation, and evaluation. Meanwhile, researchers studying writing processes have been using keystroke logging (KL) and eye-tracking (ET) to analyze and visualize process engagement. This study explores whether such technologies can support more meaningful AfL of L2 writing. Two Chinese L1 students studying at a U.S. university who served as case studies completed a series of argumentative writing tasks while a KL-ET system traced their processes and then produced visualizations that were used for individualized tutoring. Data sources included the visualizations, tutoring-session transcripts, the participants’ assessed final essays, and written reflections. Findings showed the technologies, in combination with the assessment dialogues they facilitated, made it possible to (1) position the participants in relation to developmental models of writing; (2) identify and address problems with planning, formulation, and revision; and (3) reveal deep-seated motivational issues that constrained the participants’ learning.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... However, when faced with novel and complex academic and professional writing tasks, university students are often unable to deploy their strategies effectively (Ranalli et al., 2019). Therefore, any writing-process instruction that happens at the post-secondary level must take as a starting point each student's already-established strategies and attempt to improve those, instead of teaching writing strategies from scratch (Feng and Chukharev-Hudilainen, 2017;Ranalli et al., 2018Ranalli et al., , 2019. ...
... A different approach to providing process-based feedback was proposed by Ranalli et al. (2018). Instead of helping students adjust their writing processes so that they would lean toward process-measure benchmarks obtained from high-quality texts, . ...
... Two remediation plans were selected for this iteration based on the most common process behaviors that appeared to lead to issues in the final texts of the 40 manually analyzed sessions: "Do not edit" and "Pause sentence-initially." During this iteration, similarly to Ranalli et al. (2018), an expert decision about the optimal remediation plan was made in each case by the researchers. In the decision-making process, the authors and a trained research assistant manually reviewed visual replays of the writing processes, and noted features such as the presence/absence of final review, episodes of inscription, time that writing begins, time between the first read-through of the prompt and the beginning of writing, frequency/volume of deletions, frequency of evaluation and/or reading of the prompt, overlapping of processes, frequency/duration of pauses, location of longest pauses, and presence/absence of sentenceinitial pauses. ...
Article
Full-text available
Writing quality is dependent upon the organization and sequencing of cognitive processes during writing. College students need writing-strategy advice that is tailored to their individual needs and is cognizant of their already-established writing processes. However, there is an obstacle to providing such advice: Both writing instructors and the writers lack awareness of the moment-by-moment actions by which text was produced. This is because switching between the processes of defining the task, coming up with ideas, outputting text, evaluating, and revising is largely regulated implicitly. To address this shortcoming, the present study uses a design-based research approach to develop and evaluate a minimally viable prototype of a system called “ProWrite” that uses novel biometric technology (concurrent keystroke logging and eye tracking) for providing real-time, individualized, automated, process-focused feedback to writers. This feedback is grounded in the analysis of each writer's individual needs and is presented in the context of a learning cycle consisting of an initial diagnostic, an intervention assignment, and a final follow-up. In two iterations, eight students used the system. Effects on student behavior were determined through direct analysis of biometric writing-process data before and after remediation and through changes in writing-process and written-product measures. Semi-structured interviews revealed that students generally considered the system useful, and they would try to use the newly learned strategies in their future writing experiences. The study demonstrated that individualized, real-time feedback informed by biometric technology can effectively modify writers' processes when writing takes place.
... Although there is quite a lot of research on how to improve writing skills in students in general (López et al., 2018), for example, on selfregulation strategies and writing in particular genres (for an overview, see Fidalgo et al., 2018), none of these studies focus specifically on writing from sources and the use of sources, nor on the writing process itself. The pilot study by Ranalli et al. (2018) and the LIFT project (Vandermeulen et al., 2020) stand as the only exceptions. The former provides process feedback on students' own writing and the latter on source related behavior throughout that process. ...
... In this respect, recent research (Ranalli et al., 2018;Vandermeulen et al., 2020) shows that process-oriented feedback in source-based writing results in better quality texts. The intervention study developed by Vandermeulen et al. (2020) on process feedback through a new feature in Inputlog showed positive results on product quality. ...
Article
Full-text available
Source-based writing is a crucial skill for communication professionals. This study investigates how master students in professional communication incorporate sources in synthesis writing tasks in their L1 (Dutch), L2 (English) and FL (Spanish and French). To show the complexity that characterizes the use of sources, we will first present a case study of two students in Spanish as a foreign language. We will then link it to the results of a factor analysis that points out which variables are descriptive indicators of the way in which master’s students (N = 209) deal with sources during the writing process. There are three components that can determine source use (75% of the total variance in the data): a) initial reading time, b) interaction with sources, and c) degree of variation in source use. In L1 there seems to be a correlation between the initial reading time and interaction with sources, on the one hand, and the quality of the writing, on the other; also, the time spent consulting sources compared to writing time is considerably longer in FL than in L1 and L2. These findings allow us to reflect upon how to develop effective teaching strategies to promote writing skills from sources in a foreign language. More specifically, we will focus on developing process feedback, based on the recording data, that promotes reflection and self-regulation.
... Sin embargo, existe bastante investigación sobre cómo mejorar las habilidades de escritura en los estudiantes en general (López et al., 2018), por ejemplo, sobre estrategias de autorregulación y escritura en géneros particulares (para una visión general, véase Fidalgo et al., 2018), pero ninguno de estos estudios se centra específicamente en la escritura a partir de fuentes y el uso de las mismas, ni en el proceso de escritura propiamente dicho. El estudio piloto de Ranalli et al. (2018) y el proyecto LIFT nos constan como las únicas excepciones. El primero proporciona retroalimentación de proceso sobre la propia escritura de los estudiantes y el segundo sobre el momento adecuado para consultar fuentes a lo largo de dicho proceso. ...
... Asimismo, nuestra investigación confirma la importancia de las características individuales y la consecutiva necesidad de una retroalimentación del proceso capaz de ajustar el proceso de escritura mientras está en marcha y, por tanto, de apelar al potencial de autorregulación del escritor. Investigaciones recientes (Ranalli et al., 2018; muestran que la retroalimentación personalizada orientada al proceso en la escritura basada en fuentes da como resultado textos de mejor calidad. El estudio de intervención desarrollado por sobre la retroalimentación del proceso a través de una nueva función en Inputlog mostró resultados positivos en la calidad del producto. ...
Conference Paper
Full-text available
Este estudio investiga cómo estudiantes de máster (N = 209) incorporan las fuentes en tareas de escritura de síntesis en su L1 (neerlandés), L2 (inglés) y LE (francés y español). Los procesos de escritura y el uso de las fuentes se registraron mediante el programa de grabación de escritura Inputlog. Para mostrar la complejidad que caracteriza el uso de las fuentes, presentaremos un estudio de caso de dos estudiantes en español como lengua extranjera. Después lo vincularemos con los resultados de un análisis factorial que señala qué variables son indicadores de la forma en la que los estudiantes de máster tratan las fuentes durante el proceso de escritura. En la L1 existiría una correlación entre el tiempo de lectura inicial y la interacción con las fuentes, por un lado, y la calidad del escrito, por otro; asimismo, el tiempo de consulta de las fuentes comparado con el tiempo de escritura es considerablemente mayor en la LE que en la L1 y la L2. A partir de estos hallazgos, elaboraremos estrategias para promover las habilidades de escritura basada en fuentes en un idioma extranjero. Nos centraremos en la elaboración de una retroalimentación del proceso que promueva la reflexión y la autorregulación. Palabras clave: escritura a partir de fuentes, proceso de escritura, grabación de escritura, estrategias de retroalimentación, autorregulación.
... The manual coding was conducted as a part of a previous study, described in detail in (Conijn et al., 2021). For the manual annotations, a dataset of keystroke and eye fixation data (using Gazepoint GP3 devices with 0.5-1 degree of visual angle accuracy and a 60 Hz sampling rate) was sampled from a large database yielded by the CyWrite project (Chukharev-Hudilainen, 2019;Chukharev-Hudilainen et al., 2019;Feng et al., 2016;Ranalli et al., 2018). The dataset contained texts produced by 65 students writing with CyWrite, a web-based word-processing tool. ...
... Although the current system only works for Inputlog and CyWrite data, it can be easily adapted to other keystroke logging and eye tracking applications. In addition, the automatic annotation could also allow for the integration of revision identification in real-time systems that automatically provide information on the writer's evolving writing process, such as writing dashboards or automated writing feedback systems (see e.g., Conijn et al., 2020;Ranalli et al., 2018). ...
Article
Full-text available
Revision plays an important role in writing, and as revisions break down the linearity of the writing process, they are crucial in describing writing process dynamics. Keystroke logging and analysis have been used to identify revisions made during writing. Previous approaches include the manual annotation of revisions, building nonlinear S-notations, and the automated extraction of backspace keypresses. However, these approaches are time-intensive, vulnerable to construct, or restricted. Therefore, this article presents a computational approach to the automatic extraction of full revision events from keystroke logs, including both insertions and deletions, as well as the characters typed to replace the deleted text. Within this approach, revision candidates are first automatically extracted, which allows for a simplified manual annotation of revision events. Second, machine learning is used to automatically detect revision events. For this, 7120 revision events were manually annotated in a dataset of keystrokes obtained from 65 students conducting a writing task. The results showed that revision events could be automatically predicted with a relatively high accuracy. In addition, a case study proved that this approach could be easily applied to a new dataset. To conclude, computational approaches can be beneficial in providing automated insights into revisions in writing.
... One possible reason may have to do with the data collection methods. Traditional process-oriented writing research has relied heavily on qualitative methods including observation, think-aloud protocol, and interviews [40]. These methods provide rich data about the underlying cognitive processes used in writing, but they are inherently limited in that they interrupt the processes or produce delayed reports. ...
... To date, the application of eye tracking in writing research has centered on reading behavior during composition [17]. The method has facilitated observations on how participants read visual stimuli such as graph prompts, emerging texts, and automated feedback when producing compositions [40,46,47]. However, eye-tracking studies of the cognitive processes of writers are scarce, and to the best of our knowledge, none have explored the source-based argumentation behavior in L1 and L2 writing. ...
Article
Full-text available
Argumentative writing is the most commonly used genre in writing classroom practices and assessments. To draft an argumentative essay in authentic settings, writers are usually required to evaluate and use content knowledge from outside sources. Although source-based argumentation is a sustainable skill that is crucial for students’ academic career, this area remains under-researched. Hence, this paper presents a within-subject study that investigated Hong Kong secondary school students’ argumentation construction in L1 and L2 source-based writing from both product-oriented and process-oriented perspectives. Multiple sources of data were collected, including L1 and L2 source-based argumentative texts, eye-tracking metrics and recorded videos, and stimulated recall interviews. Findings of our study show that the L1 source-based argumentative compositions of the Hong Kong secondary student writers differed greatly from their L2 ones in terms of the argument structure, source use, and reasoning quality. Analyses on four cases further revealed a multitude of factors such as self-regulation and cultural orientations coming into play in similar and different argumentation performance between L1 and L2 source-based writing tasks. This study contributes new knowledge to better understand the argumentation in L1 and L2 source-based writing, yielding meaningful implications on pedagogy and assessment in this field.
... More recently, researchers have started to adopt advanced process-tracking technology, such as eye-tracking and keystroke logging tools (e.g. Leijten & van Waes, 2013;Michel, et al., 2020;Ranalli et al., 2018). Ranalli et al. (2018) adopted keystroke logging (KL) and eye-tracking (ET) to study two students' mental processes while writing in English. ...
... Leijten & van Waes, 2013;Michel, et al., 2020;Ranalli et al., 2018). Ranalli et al. (2018) adopted keystroke logging (KL) and eye-tracking (ET) to study two students' mental processes while writing in English. Michel et al. (2020) combined KL and ET with simulated verbal recall to record and compare students' writing processes for independent and integrated writing tasks. ...
Article
Full-text available
This research conducted diagnostic assessment of problems in first-year undergraduates’ English academic papers and tracked potential sources of the problems to the writing process and strategy use. Data collected include 339 term papers and interviews with 17 students. The samples were manually error tagged and marked against a detailed diagnostic checklist. The resultant textual features were then compared between two subgroups of Chinese students in the sample, namely, those graduating from local schools in Hong Kong (LS) and those coming from the mainland and sojourning in Hong Kong (MS). The analyses found both groups had the poorest performance in source integration and vocabulary use. LS used simpler words and made more grammatical errors, whereas MS attempted sophisticated vocabulary more successfully and used a wider variety of words and sentence structures. The difficulties they experienced, however, were rather similar, residing mainly at the researching, planning and formulating stages. Action control theory was introduced to interpret the self-regulatory strategies they adopted to cope with perceived difficulties during the writing process. Strategies to control goals, control resources, and control cognitive load were found to be the most typical. While these strategies could reduce their difficulties, only some seemed also to help with performance. A conceptual framework is proposed at the end to link writing products, process and self-regulatory control strategies as evidenced in the study. Four diagnoses are drawn with suggestions for practice and further research.
... Keystroke logging as a part of the feedback process has been used in a limited number of studies on writing. Lindgren and Sullivan (2003) , for instance, used a KSL replay function based as a prime for stimulated recall in peer discussion; Ranalli et al. (2018) , on the other hand, combined KSL with eye-tracking data as means to stimulate one-to-one discussions between teacher and student. These studies showed the potential of process feedback based on KSL, but were limited to a replay function, were rather time consuming, and required extensive teacher/peer intervention to support students in interpreting the data ( Spelman Miller, 2005 ). ...
Preprint
The importance of feedback in writing classes has long been recognised. However, most feedback centres on writing products (e.g., drafts, completed essays) with feedback on writing processes difficult to provide, particularly in online writing classes (OWC). This paper explores the potential of keystroke logging software (KSL) to provide process-oriented feedback with a secondary focus on increasing self-regulated writing capacity. Specifically, we designed a four-stage intervention using KSL to provide feedback for 34 undergraduates over a 16-week English as a foreign language OWC. Students recorded themselves writing narrative, descriptive, expository, and opinion essays (~450 words each). One group was given basic tasks and the other was given tasks designed to improve time management, revision, and source use. Results show significant improvements on first draft performance for the intervention group in text quality and words typed. There was no significant difference between groups on second draft scores. Moreover, both groups reported a significant increase in their use of peer learning strategies, whilst the intervention group reported significant increases in metacognitive strategies, particularly idea planning. Based on these findings, and students’ perceptions on process feedback, we make a number of recommendations for future studies and lessons that seek to incorporate process-oriented feedback.
... In this study, eye-tracking was used to examine writers' rereading, intense reading and error checking behaviours. In another recent study, Ranallia, Feng, and Chukharev-Hudilainena (2018) combined eye-tracking with keystroke logging and written reflections to trace L2 writers' composing processes and problems. On the other hand, Yu, He, and Isaacs (2017) made much more extensive use of eye-tracking by employing it in investigating testtakers' cognitive processes while performing the International English Language Testing System (IELTS) academic writing task one. ...
Article
Over the last few years, there has been a noticeable increase in the use of eye-tracking in second language (L2) learner process research. The previous reviews of eye-tracking language research have focused primarily either on providing guidelines for using this technological data source and/or on highlighting selected eye-tracking studies in one or two second language acquisition (SLA) areas. Thus, there is still a need for a review that covers the research and methodological issues of using eye-tracking in multiple L2 learner process areas. Addressing such gap, this paper reviews the eye-tracking L2 learner process studies published between 2011 and 2018. Specifically, the paper focuses on reviewing the eye-tracking L2 learner process research areas, the issues addressed by the studies in each area, and the data collection and analysis approaches used in these studies. It was found that eye-tracking has been used in eight L2 learner process research areas: vocabulary processing and learning, listening comprehension, syntactic processing, written text production, reading comprehension, text-based computer-mediated communication (CMC), oral communication, and data validation. The paper reveals that data collection and analysis approaches have differed from one area to another, and it provides some research and methodological suggestions.
... With the onset of technology in education, writing research employed methods like Keystroke Logging (KL) (Chukharev-Hudilainen, Saricaoglu, Torrance, & Feng, 2019;Leijten, Van Horenbeeck, & Van Waes, 2019) as a shift from paper-pencil-based exercises to electronic exercises (Akbar, Firman, & Rusyati, 2017;Ene & Upton, 2018;Zheng, Yim, & Warschauer, 2018). Various potential advantages come along with this shift, for instance, saving all data about students' writing behaviors (Ranalli, Feng, & Chukharev-Hudilainen, 2018;Ranalli, Feng, & Chukharev-Khudilaynen, 2018) in log files, making it possible to extricate the mechanisms underlying them. Many studies to date have employed think aloud protocol as a direct method of eliciting underlying cognitive processes in writing (Forbes, 2018;Tiryakioglu, Peters, & Verschaffel, 2018). ...
Article
The continuing growth of technology has led to a growing interest in online writing tools to gain an in-depth insight into students' cognitive performance. Using Inputlog 8.0, the present work explored undergraduate male and female students’ (N = 72, mean age = 19.7 years) Online Argumentative Writing Performance (OAWP). Participants were provided argumentative writing tasks in two temporal phases: a) composing and b) revision. The log-file data were analyzed using structural equation modeling (SEM) to explore any linear or quadratic relations among manifest (i.e., revision frequency, text length, mean R-bursts, total pause time, total active writing time, pause frequency, P-burst frequency, mean typed in P-burst and geometric mean of within-word, between-word, sentence, paragraph pauses) and latent (OAWP) variables simultaneously. The result showed significant linear relations between all variables and OAWP but no quadric relations were observed. Overall, findings suggest that students who employed between-word or between-sentence pause strategy (PS), active writing plus planned pause showed better OAWP, while high frequency of revision and pause at between word boundaries indicated much less successful OAWP. The result indicates that scaffolded revision, worked for L2 students' advantage in making strategic use of time for planning and improvement of written argumentation.
... Also, the 'time aspect' complicates the practical use of keystroke logging in education. A recent and innovative pilot study by Ranalli, Feng, and Chukharev-Hudilainen (2018) made use of both keystroke logging and eye-tracking in an educational setting. Chinese students wrote argumentative essays in English in a computer lab that was equipped with eye-tracking laptops. ...
Article
Full-text available
OPEN ACCESS - https://www.jowr.org/abstracts/vol12_1/Vandermeulen_et_al_2020_12_1_abstract.html Keystroke loggers facilitate researchers to collect fine-grained process data and offer support in analyzing these data. Keystroke logging has become popular in writing research, and study by study we are now paving the path to a better understanding of writing process data. However, few researchers have concentrated on how to bring keystroke logging to the classroom. Not because they are not convinced that writing development could benefit from a more process-oriented pedagogy, but because 'translating' complex and large data sets to an educational context is challenging. Therefore, we have developed a new function in Inputlog, specifically aiming to facilitate writing tutors in providing process feedback to their students. Based on an XML- logfile, the so-called 'report' function automatically generates a pdf-file addressing different perspectives of the writing process: pausing, revision, source use, and fluency. These perspectives are reported either quantitatively or visually. Brief introductory texts explain the information presented. Inputlog provides a default feedback report, but users can also customize the report. This paper describes the process report and demonstrates the use of it in an intervention. We also present some additional pedagogical scenarios to actively use this type of feedback in writing classes.
... The dashboard in this article is focused on displaying information on students' tracked writing processes, for two reasons: 1) students' self-report on their writing processes is unreliable and 2) analysis of the writing product does not always provide insight into when, where, and why students struggled [20]. The dashboard is specifically focused on revisions, because revisions play an important role in writing [9], but are not visible in the written product. ...
Conference Paper
Learning dashboards are often used to provide teachers with insight into students' learning processes. However, simply providing teachers with data on students' learning processes is not necessarily beneficial for improving learning and teaching; the data need to be action-able. Recently, human-centered learning analytics has been suggested as a solution to realize more effective and actionable dashboards. Accordingly, this study aims to identify how these human-centered approaches could be used to design an interpretable and actionable learning dashboard on students' writing processes. The design consists of three iterative steps. First, visualizations on students' revision process, created from keystroke data, were evaluated with writing researchers. Second, the updated visu-alizations were used to co-design a paper prototype of the dashboard within a focus group session with writing teachers. Finally, the paper prototype was transformed into a digital prototype and evaluated by teachers in individual user test interviews. The results showed that this approach was useful for designing an interpretable dashboard with envisioned actions, which could be further tested in classroom settings.
... In this paper, we have reported the piloting of feedback based on process-tracing through a pedagogical, practice-oriented lens. An accompanying empirical study (Ranalli, Feng, & Chukharev-Hudilainen, 2018) provided more systematic and detailed documentation of the three affordances discussed above but also highlighted two main issues that need to be addressed before this innovation can be implemented in real L2 writing classrooms. ...
Article
Full-text available
The research literature on L2 writing processes contains a multitude of insights that could inform writing instruction, but writing teachers are constrained in their capacity to make use of these insights insofar as they lack detailed information about how their students' actually engage in the processes of writing. At the same time, writing-process researchers have been using powerful technologies to trace writers' process engagement-namely, keystroke-logging and eye-tracking-that are potentially applicable in educational settings. In this article, we describe a pilot effort to integrate these technologies into L2 writing instruction with college-level ESL students. In addition to illustrating three key affordances of these technologies that emerged from the piloting, we discuss the conceptual framework that informed our efforts as well as challenges that will need to be addressed to facilitate further integration of process tracing into L2 writing pedagogy.
... To this end, learning to write in L1 is promoted and a sufficient level of L2 writing is required from the first year of university (Bychkovska & Lee, 2017;Karnal, 2013). In this regard, Eckstein and Ferris (2018), highlight that argumentative writing in L2 presents more needs for attention than in L1 and, therefore, Ranalli et al. (2018) insist on the potential of ICTs to help students to manage and acquire better writing competences in different languages. Finally, shared scenarios are proposed for teaching in L1 and L2 (Costino & Hyon, 2011). ...
Article
Full-text available
Aim/Purpose Quality in higher education assumes the challenge of developing in all citizens of the 21st century the cognitive, motivational, and socio-cultural dimensions that provide them with communication competences including the use of information and communication technologies, for the dissemination of sustainable scientific knowledge in different languages. Hence this paper evaluates a di-dactic-technological process called "Ensayo Científico Multilingüe" or ECM ("Multilingual Scientific Essay"), which guides the construction of argumentative texts in a shared didactic space in the native language (L1) and in the first foreign language (L2). Background Although the use of information and communication technologies (ICTs) in educational contexts stands out as an index of quality, some studies indicate that these technologies, by themselves, do not produce changes in learning. The possibility that ICTs can contribute to a university-quality education is by providing measures that allow verification of the effects on the real improvement of the learning of communication competences of students and, especially , in the learning of written communication for the purpose of scientific dissemination. In order to do so, this research is based on the Metasociocogni-tive Written Composition (MWC) model that explains university writing as a
... While the testing of writing skills typically only relies on the quality assessment of the finished text, considering the writing process too could result in a more fine-grained evaluation of writing performance (cf. Ranalli et al. 2018). Thus, it would make sense, as is the case for speech, to include a criterion like writing fluency, which would aim to assess how smooth the writing process is. ...
Article
Full-text available
The Process Corpus of English in Education (PROCEED) is a learner corpus of English which, in addition to written texts, consists of data that make the writing process visible in the form of keystroke log files and screencast videos. It comes with rich metadata about each learner, among which indices of exposure to the target language and cognitive measures such as working memory or fluid intelligence. It also includes an L1 component which is made up of similar data produced by the learners in their mother tongue. PROCEED opens new perspectives in the study of learner writing, by going beyond the written product. It makes it possible to investigate aspects such as writing fluency, use of online resources, cognitive phenomena like automaticity and avoidance, or theoretical modelling of the writing process. It also has applications for teaching, e.g. by showing students screencast video clips from the corpus illustrating effective writing strategies, as well as for testing, e.g. by establishing a corpus-derived standard of writing fluency for learners at a certain proficiency level.
... Keystroke logging as a part of the feedback process has been used in a limited number of studies on writing. Lindgren and Sullivan (2003), for instance, used a KSL replay function based as a prime for stimulated recall in peer discussion ;Ranalli et al. (2018), on the other hand, combined KSL with eye-tracking data as means to stimulate one-to-one discussions between teacher and student. These studies showed the potential of process feedback based on KSL, but were limited to a replay function, were rather time consuming, and required extensive teacher/peer intervention to support students in interpreting the data (Spelman Miller, 2005). ...
Article
Full-text available
The importance of feedback in writing classes has long been recognised. However, most feedback centres on writing products (e.g., drafts, completed essays) with feedback on writing processes difficult to provide. This paper explores the potential of keystroke logging software (KSL) to provide process-oriented feedback with a secondary focus on increasing self-regulated writing capacity. Specifically, we designed a four-stage intervention using KSL to provide feedback for 34 undergraduates over a 16-week English as a foreign language online writing class. Students recorded themselves writing narrative, descriptive, expository, and opinion essays (~450 words each). Both groups were given feedback tasks designed to improve time management, revision, and source use, but Group B was given targeted and more concrete tasks (i.e., increasingly other-regulated). Results show significant improvements on first draft performance for Group B in text quality and words typed. There was no significant difference between groups on second draft scores. Moreover, both groups reported a significant increase in their use of peer learning strategies, whilst Group B reported significant increases in metacognitive strategies, particularly idea planning. Based on these findings and students’ perceptions on process-oriented feedback, we make a number of recommendations for future studies and lessons that seek to incorporate process-oriented feedback.
Article
p style="text-align: justify;">The decline in student character is the result of low student learning outcomes. The common student learning outcomes are influenced by several factors, and one of them is teacher-centered, monotonous learning model. For this reason, it is deemed necessary to conduct research that aims to determine the effect of project-based assessment on values clarification technique (VCT) learning on improving students’ learning outcomes by controlling the family environment. This study uses a 2x2 factorial experimental design. The sample was selected through multistage random sampling with 120 students. The two-way ANCOVA data analysis technique was used to analyze the data. The findings obtained after controlling the family environment are: 1) civics learning outcomes from the group of students who used value clarification techniques are higher than those using conventional learning models and 2) civics learning outcomes from the group of students who were given project-based assessments are higher than the group who are given conventional assessments. Thus, it can be recommended that civics education teachers used appropriate VCT and project-based assessments to improve learning outcomes.</p
Article
Full-text available
This study explored critical thinking skills toward YouTube resources in students' argumentative texts. The study was conducted at two non-English majors of the universities in the Indonesian context. The data were in the argumentative forms that were constructed by selected students. This study used a content analysis approach that was involved in coding and categorizing to analyze the data. After the data were analyzed, the findings showed that the patterns of critical thinking elements of the student's argumentative texts were varied, and the integration of technology resources was able to foster critical thinking skills. However, it is based upon the quality of the technology itself and the instruction in the classroom. Thus, technology implementation, YouTube resources should be found on the quality of the resources and teaching in the school.
Article
This qualitative case study explores how a teacher directed himself in implementing assessment for learning (AfL) among his students in a constrained context (i.e., a lack of institutional support in offering effective teacher education programs). The data included a Chinese college English writing teacher’s reflections over two courses, interviews, and dialogues with the students, coupled with field notes. The qualitative analysis shows that in a constrained context, self-directed AfL was buttressed by the teacher's existent pedagogical knowledge. In the process, the teacher’s self-agency empowered him in synergizing existent knowledge with external resources and revamping his teaching based on the students’ needs as reflected through AfL. These student-centered practices contributed to reduced complexity of AfL implementation, and yielded a more or less beginning cycle: the students echoed the teacher’s purpose of assessment, which considered knowledge appropriation as pivotal, and the teacher felt professionally rewarded from self-directed development using assessment-based information to improve his instruction and benefit his students.
Article
Full-text available
Increased interest in the study of second language (L2) writing has resulted in an ever- increasing rate of publication in this area of study, making it challenging for L2 writing professionals to stay up to date with the relevant research. In 2018 alone, we located more than 380 publications focused exclusively on L2 writing. In order to help L2 writing professionals stay abreast of current developments in the field, we provide below an overview and synthesis of scholarship on L2 writing published last year.The types of publications we will be addressing primarily include journal articles, books (authored and edited), and dissertations. Data for this article come from a search of databases includingERIC (Educational Information Resources Center), LLBA(Linguistics and Language Behavior Abstracts), PQDT(ProQuest Dissertations and Theses), World cat, Google Scholar, and Amazon.com, as well as a regular review of more than 60 journals that, to a greater or lesser extent, typically publish articles on second language writing
Article
Keyboard-based writing has become increasingly important in everyday living and professional working. However, keyboarding literacy is presumed in most situations; there lack systematic studies on keyboarding and its association with writing processes and qualities of written products. Using the keystroke data of over 1300 middle-school students, we investigated keyboarding fluency, a key subskill of writing transcription measured by an internally reliable metric, and its association with writing process, score, and text characteristics extracted by NLP techniques. We find that: keyboarding fluency is a reliable individual-level property, subject to demographic and linguistic factors of gender, ethnicity, and grade; there exists a threshold keyboarding fluency, which varies by writing purposes and reflects the minimum level of keyboarding skill at which other writing processes are least affected; and more and less fluent typists (differentiated by the threshold) show different text production, transcription, and editing behaviors during writing processes, and distinct linguistic qualities in written products. These findings imply a keyboarding threshold hypothesis and are informative to different stakeholders including practitioners, researchers, test developers, and policy makers. They also provide practical implications for teaching and learning writing in classrooms under a digital environment and help generate personalized feedback on improving writing practice.
Article
This case study investigates the phenomenon of graduate students writing with unfamiliar digital tools in semi-public writing environments. The increase in the prevalence of writing with unfamiliar tools in semi-public environments, such as networked computer classrooms and university testing centers, makes it worthy of investigation. We use phenomenological interviews to examine the writing experiences of a group of graduate students writing in a classroom on unfamiliar computers equipped with a tool that tracked their keystrokes and eye movements. Though some of the writers had positive experiences with the tool's output and their reflective conversations about writing it prompted, they all had challenging experiences adjusting to the hardware, the physical requirements of the tool, and overcoming surveillance anxiety prompted by it. Some students who wrote in the semi-public environment using an unfamiliar tool benefitted but all were challenged by situational awareness, new hardware, new haptic interactions, surveillance anxiety, and lack of control. The study indicates a need to explore and address the factors of situation awareness, an adjustment period, and surveillance anxiety in situations where individuals are writing with unfamiliar tools in semi-public environments.
Chapter
Language tests play pivotal roles in education, research on learning, and gate-keeping decisions. The central concern for language testing professionals is how to investigate whether or not tests are appropriate for their intended purposes. This book introduces an argument-based validity framework to help with the design of research that investigates the validity of language test interpretation and use. The book presents the principal concepts and technical terms, then shows how they can be implemented successfully in practice through a variety of validation studies. It also demonstrates how argument-based validity intersects with technology in language testing research and highlights the use of validity argument for identifying research questions and interpreting the results of validation research. Use of the framework helps researchers in language testing to communicate clearly and consistently about technical issues with each other and with researchers of other types of tests.
Article
Full-text available
The Fall 2007 and Spring 2008 pilot tests for the CBAL™ Writing assessment included experimental keystroke logging capabilities. This report documents the approaches used to capture the keystroke logs and the algorithms used to process the outputs. It also includes some preliminary findings based on the pilot data. In particular, it notes that the distribution of most of the pause length is consistent with data generated from a mixture of lognormal distributions. This corresponds to a cognitive model in which some pauses are merely part of the transcription (i.e., typing) process and some are part of more involved cognitive process (e.g., attention to writing conventions, word choice, and planning). In the pilot data, many of the features extracted from the keystroke logs were correlated with human scores. Due to the small sample sizes of the pilot studies, these findings are suggestive, not conclusive; however, they suggest a line of analysis for a large sample containing keystroke logging gathered in the fall of 2009.
Article
Full-text available
Writing in the workplace: Constructing documents using multiple digital sources Leijten, M., Van Waes, L., Schriver, K., & Hayes, J.R. (2014) Journal of Writing Research, 5(3), 285-337. http://www.jowr.org/Ccount/click.php?id=76 In today’s workplaces professional communication often involves constructing documents from multiple digital sources—integrating one’s own texts/graphics with ideas based on others’ text/graphics. This article presents a case study of a professional communication designer as he constructs a proposal over several days. Drawing on keystroke and interview data, we map the professional’s overall process, plot the time course of his writing/design, illustrate how he searches for content and switches among optional digital sources, and show how he modifies and reuses others’ content. The case study reveals not only that the professional (1) searches extensively through multiple sources for content and ideas but that he also (2) constructs visual content (charts, graphs, photographs) as well as verbal content, and (3) manages his attention and motivation over this extended task. Since these three activities are not represented in current models of writing, we propose their addition not just to models of communication design, but also to models of writing in general.
Article
Full-text available
The authors argue that design-based research, which blends empirical educational research with the theory-driven design of learning environments, is an important methodology for understanding how, when, and why educational innovations work in practice. Designbased researchers’ innovations embody specific theoretical claims about teaching and learning, and help us understand the relationships among educational theory, designed artifact, and practice. Design is central in efforts to foster learning, create usable knowledge, and advance theories of learning and teaching in complex settings. Designbased research also may contribute to the growth of human capacity for subsequent educational reform.
Article
Full-text available
This paper examines the construct measured by automated essay scoring (AES) systems. AES systems measure features of the text structure, linguistic structure, and conventional print form of essays; as such, the systems primarily measure text production skills. In the current state-of-the-art, AES provide little direct evidence about such matters as strength of argumentation or rhetorical effectiveness. However, since there is a relationship between ease of text production and ability to mobilize cognitive resources to address rhetorical and conceptual problems, AES systems have strong correlations with overall performance and can effectively distinguish students in a position to apply a broader writing construct from those for whom text production constitutes a significant barrier to achievement. The papers begins by defining writing as a construct and then turns to the e-rater scoring engine as an example of AES state-of-the-art construct measurement. Common criticisms of AES are defined and explicated—fundamental objections to the construct measured, methods used to measure the construct, and technical inadequacies—and a direction for future research is identified through a socio-cognitive approach to AES.
Article
Full-text available
Keystroke logging has become instrumental in identifying writing strategies and understanding cognitive processes. Recent technological advances have refined logging efficiency and analytical outputs. While keystroke logging allows for ecological data collection, it is often difficult to connect the fine grain of logging data to the underlying cognitive processes. Multiple methodologies are useful to offset these difficulties. In this article we explore the complementarity of the keystroke logging program Inputlog with other observational techniques: thinking aloud protocols and eyetracking data. In addition, we illustrate new graphic and statistical data analysis techniques, mainly adapted from network analysis and data mining. Data extracts are drawn from a study of writing from multiple sources. In conclusion, we consider future developments for keystroke logging, in particular letter- to word-level aggregation and logging standardization.
Article
Full-text available
This article presents the results of a study carried out in Sweden to investigate the promotion of self-assessment and reflection in the adult second language (L2) classroom. A method is proposed in which the computer is used first to record a writing session, and later to replay the entire text production in retrospective peer sessions. The method provides the students with an opportunity to look into their own composing processes both linguistically and holistically, as they view and discuss the reasons behind the di¤erent actions during the writing process. Results show that after using the method, all writers experienced useful, although di¤erent, insights into their own writing behaviours. Furthermore, this method is not restricted to an L2 environment, but is likely to be e¤ective in other learning situations where reflection is useful for the acquisition process.
Article
Full-text available
Taylor and Francis Ltd CSHE_A_157196.sgm 10.1080/03075070600572132 Studies in Higher Education 0307-5079 (print)/1470-174X (online) Original Article 2006 Society for Research into Higher Education 31 2 000000April 2006 DavidCarless Faculty of EducationUniversity of Hong KongPokfulamHong Kongdcarless@hkucc.hku.hk Feedback is central to the development of effective learning, yet is comparatively underresearched. This article seeks to examine the notion of written feedback on assignments and argue that this feedback process is more complex than is sometimes acknowledged. The author illustrates the prob-lematic nature of assignment feedback by drawing on a large-scale questionnaire survey conducted across eight universities, and then analysing the issue in more depth though fine-grained data collected from students in a teacher education institute. The article is framed by the concepts of discourse, power and emotion. It highlights a number of different perceptions of students and tutors towards the assessment, marking and feedback process. The author concludes by arguing that 'assessment dialogues' are a way forward to mitigate some of the mistrust or misconceptions that may be unwanted outcomes of the assessment process.
Chapter
Full-text available
The present chapter is intended as a critical analysis of the most relevant recent research into the cognitive processes underlying second language written composition. After an introduction of the research domain, a number of relevant methodological aspects are briefly discussed. These include the data collection procedures used, the assessment of writers’ command of the second language, the evaluation of written products, the context of the research, the type and number of participants involved, the type of tasks used, and the way reliability has been reported in the different studies. The substantive part of the research has been analyzed by isolating its main theoretical frames. Each of these frames has allowed us to derive a number of research sub-domains under which the studies have been grouped: the comparison of skilled and unskilled L2 writers, the development of L2 writing skill, the comparison of L1 and L2 writing processes, and the relationship between writing ability and L2 proficiency. A systematic analysis of the findings within each category has led us to identify a number of areas in need of further research: the notion of L2 writing skill, the formulation process, the temporal character of composition, the cognitive mechanisms involved in the transfer of writing abilities across languages, and the situated nature of L2 writing. KeywordsL1 and L2 writing processes–writing skill–transfer of writing skills–skilled vs. unskilled writers–relationship writing skill and L2 proficiency
Chapter
The influence of cognitive processing on second language acquisition (SLA), and on the development of second language (SL) instruction, has always been a subject of major interest to both SLA researchers and those involved in SL pedagogy. Recent theoretical research into SLA and SL pedagogy has shown renewed interest in the role of cognitive variables such as attention, short, working, and long term memory, and automaticity of language processing. This volume first examines the theoretical foundations of research into the cognitive processes underlying SLA, and then describes various implications for pedagogically oriented research and for SL classroom practice. The blend of research from the cognitive sciences and applied linguistics make it an excellent introduction to applied linguists and language teachers interested in the psycholinguistic processes underlying SLA.
Article
Classroom assessment has always been an indispensible and integral part of any curriculum. In particular, assessment plays the role of reporting students' learning summatively (assessment of learning), providing diagnostic and formative information for teachers to inform their instruction (assessment for learning); more recently, Earl (2013) proposed the notion of assessment as learning, which puts students at the center of assessment. Students in this assessment paradigm act as critical connecters between assessment and learning through self-reflection and self-regulation. The first section of this article reconceptualizes summative and formative assessments into three assessment paradigms: assessment of, for, and as learning through incorporating Serafini's assessment models and Habermas's three human interests. In so doing, our understanding of the three paradigms is consolidated and enriched to encompass not only the pedagogical implications but also their philosophical and epistemological underpinnings. The second section of the article focuses on one particular kind of assessment method commonly used in language classrooms, which is written feedback. I summarize and categorize recent written feedback research with reference to the three assessment paradigms and suggest directions for future research.
Book
While assessment and feedback tend to be treated separately in the L2 writing literature, this book brings together these two essential topics and examines how effective classroom assessment and feedback can provide a solid foundation for the successful teaching and learning of writing. Drawing upon current educational and L2 writing theories and research, the book is the first to address writing assessment and feedback in L2 primary and secondary classrooms, providing a comprehensive, up-to-date review of key issues, such as assessment for learning, assessment as learning, teacher feedback, peer feedback, portfolio assessment, and technology enhanced classroom writing assessment and feedback. The book concludes with a chapter on classroom assessment literacy for L2 writing teachers, outlines its critical components and underscores the importance of teachers undertaking continuing professional development to enhance their classroom assessment literacy. Written in an accessible style, the book provides a practical and valuable resource for L2 writing teachers to promote student writing, and for teacher educators to deliver effective classroom writing assessment and feedback training. Though the target audience is school teachers, L2 writing instructors in any context will benefit from the thorough and useful treatment of classroom assessment and feedback in the book. "A book on L2 writing assessment that truly is useful for teachers, school administrators, and students’ learning. Educators around the world, heed these well-founded insights." Alister Cumming, University of Toronto "This book carefully synthesizes two critical aspects of writing instruction--classroom assessment and feedback from various sources--that are often relegated to "separate" chapters or even books on how to teach writing to second language (L2) students. Professor Lee shows how a formative, supportive approach to both assessment and feedback provides the foundation for successful writing instruction and student growth. In her usual clear and compelling style, Lee links current and existing research trends to specific pedagogical techniques, providing a practical and accessible resource to in-service and pre-service teachers. Though the main audience is teachers at the primary and secondary levels, L2 writing instructors in any context will benefit from this thorough and useful treatment of these essential topics." Dana Ferris, University of California, Davis
Chapter
In this chapter an instructional format, Peer-Based Intervention (PBI) using computer keystroke logging is investigated as a computer technology to enhance language acquisition. In PBI, a process-oriented approach is taken, in which noticing, reflection and language awareness are central concepts of language learning. The method aims to promote learning through individualized reflection and social interaction. A retrospective focus on the evolution of a foreign language text is used as a basis for reflection and discussion. This period of reflection and discussion opens up paths for both individual and peer-based noticing and learning.
Article
This chapter presents the use of the Geographical Information Systems (GIS) for data mining and visualising information about cognitive activities involved in writing. The information can be collected from various sources, such as keystroke logs, manual analysis of stimulated recall sessions and think-aloud protocols. After an introduction to the GIS, an English as a foreign language (EFL) writing session is used to explain how to create the various GIS layers from the different information/analysis sources, and show how they can be easily data mined using the GIS techniques to improve our understanding of the cognitive processes in writing. The illustrative graphs used to provide an insight into the methodology are based on keystroke-logged data, manual researcher-based analyses and coded stimulated recall data that were collected after the writing session. Also a tool for visualisation and data mining, the GIS technique can support analysis of the interaction of cognitive processes during writing focusing on the individual writer, differences between writers or the writing processes in general. Depending on the research question, GIS affords the possibility to aggregate data to the level of writers, de-aggregate data in any way chosen or display data as attributes of individuals. © 2007 by Elsevier Ltd. All rights of reproduction in any form reserved.
Article
Second language writing assessment has traditionally entailed a primarily summative function, where assessment information is used mainly for administrative and reporting purposes. If assessment is to make a stronger impact on student learning of writing, much has to be done to change the existing summative focus, which entails a move towards assessment for learning (AfL). This chapter discusses the principles associated with assessment for learning (AfL) particularly in the context of Hong Kong writing classrooms. After critiquing AfL and reviewing the data from Hong Kong writing classrooms, this chapter ends by highlighting the factors necessary for the successful implementation of AfL not only in the Hong Kong classroom between teachers and students, but also within the parameters of school, societal, and system contexts and wherever writing assessment is likely to be involved in changing paradigms. © Springer Science+Business Media Singapore 2014. All rights are reserved.
Article
This article is concerned pragmatically with how recent research findings in assessment for learning (AfL) can bring about higher quality learning in the day-to-day classroom. The first half of this paper reviews recent studies within Black and Wiliam's (2009) framework of formative assessment and looks for insights on how pedagogical procedures could be arranged to benefit from and resonate with research findings. In the second half, based on lessons drawn from the review, the findings were incorporated into an instructional design that is contingent on formative assessment. The concept of teacher contingency is elaborated and demonstrated to be central to the AfL pedagogy. Attempts were made to translate updated research findings into an English as a foreign language (EFL) writing instruction to illustrate how teachers may live up to promises offered by recent developments on AfL. This AfL lesson, situated in L2 writing revision, made instruction contingent on and more responsive to learner performance and learning needs. As shown in an end-of-semester survey, learner response to the usefulness of the instruction was generally quite positive.
Article
Writing is one of the central skills a student must master. Why should they be tested? How should they be tested? What tasks should be used? The answers to these questions are provided by this book, which examines the theory behind the practice of assessing a student's writing abilities.
Article
This study takes a dynamic view of transfer as reusing and reshaping previous knowledge in new writing contexts to investigate how novice Japanese as a foreign language (JFL) writers draw on knowledge across languages to construct L1 and L2 texts. We analyzed L1 English and L2 Japanese argumentation essays by the same JFL writers (N = 19) and L1 Japanese essays by Japanese university students (N = 21), along with JFL writers' reported reflections. The analysis identified both shared and contrasting L1/L2 text features, including argumentation subtypes (e.g., justification, exploration) and essay introduction/conclusion components. The findings revealed that while constructing L2 essays, the JFL writers took an active role in assessing audience, selecting appropriate text features, and transforming/reshaping selected features, influenced by contextual factors (e.g., audience expectation, purpose, topic). For example, some writers reshaped their L1 justification subtype by softening L2 assertions to meet perceived Japanese reader expectations. Results highlight the centrality of the writer's agency in deciding what previous writing knowledge to reuse or reshape when creating L2 text and also the importance of individual learning trajectories (e.g., L2 proficiency, L1/L2 writing experience) affecting writers' decisions. The study affirms that a dynamic view of transfer provides insights into the L2 writers' text construction process.
Article
What should language and writing teachers do about giving students written corrective feedback? This book surveys theory, research, and practice on the important and sometimes controversial issue of written corrective feedback, also known as “error/grammar correction,” and its impact on second language acquisition and second language writing development. Offering state-of-the-art treatment of a topic that is highly relevant to both researchers and practitioners, it critically analyzes and synthesizes several parallel and complementary strands of research — work on error/feedback (both oral and written) in SLA and studies of the impact of error correction in writing/composition courses — and addresses practical applications. Drawing from both second language acquisition and writing/composition literature, this volume is the first to intentionally connect these two separate but important lines of inquiry.
Article
Test writers sometimes complain they cannot perform to their true abilities because of time constraints. We therefore examined differences in terms of process and product between texts produced under test and non-test conditions. Ten L2 postgraduates wrote two argumentative essays, one under test conditions, with only forty minutes being allowed and without recourse to resources, and one under non-test conditions, with unlimited time as well as access to the Internet. Keystroke logging, screen capture software, and stimulated recall protocols were used, participants explaining and commenting on their writing processes. Sixteen writing process ttypes were identified. Higher proportions of the processes of translation and surface revision were recorded in the test situation, while meaningful revision and evaluation were both higher in the non-test situation. There was a statistically significant difference between time allocation for different processes at different stages. Experienced teachers awarded the non-test texts a mean score of almost one point (0.8) higher. A correlational analysis examining the relationship between writing process and product quality showed that while the distribution of writing processes can have an impact on text quality in the test situation, it had no effect on the product in the non-testt situation.
Article
While there is extensive literature on how assessment for learning (AfL) can be put into practice at the general classroom level, research examining teachers' attempts to implement AfL in writing, especially in the elementary context, is relatively less explored. The present study seeks to shed light on how four elementary teachers in Hong Kong attempt to foster change in assessment by implementing AfL in the L2 writing classroom dominated by the examination culture. Drawing on data gathered from classroom observations and interviews with administrators and teachers of two Hong Kong primary schools over the course of one year, this study uncovers the tensions that arise as a result of the introduction of AfL in writing. Using activity theory and its notion of contradiction, the study concludes that the uptake of AfL innovation in writing could be inhibited unless the contradictions in the activity systems can be resolved.
Article
The most recent research in composition has given us, important insights into the composing process. This research has revealed that composing is a non-linear, exploratory, and generative process whereby writers discover and reformulate their ideas as they attempt to approximate meaning.A study of the composing processes of advanced ESL students was undertaken to investigate the extent to which these students experience writing as a process of discovering and creating meaning and the extent to which second language factors affect this process. The findings indicate that skilled ESL writers explore and clarify ideas and attend to language-related concerns primarily after their ideas have been delineated.Since it is believed that the teaching of composition should be informed by and based upon what writing actually entails, an understanding of the composing process calls into question approaches that are prescriptive, formulaic, and overly concerned with correctness. Instead, it suggests the importance of instruction that gives students direct experiences with the composing process, that establishes a dynamic teaching/learning relationship between writers and their readers, and that enhances further linguistic development in the context of making and communicating meaning.
Article
This article discusses the use of computer logging as a means of investigating aspects of the second language (L2) writing process as writers are engaged in producing text at the keyboard. The observation of writing by means of this method provides researchers with detailed information concerning aspects of the planning, formulation, and revision processes. This function is illustrated by reference to a study in Sweden of school-age learners of English as an additional language whose written production was recorded as part of a longitudinal study, and findings from the study are presented. The discussion highlights the potential uses of logging, not only not in relation to researching writers' processes, but also as a pedagogic tool given that its replay facility allows access to information about aspects of the writers' attention and strategies as they write.
Article
The present paper reports an investigation into individual differences in writing with students of English as a foreign language (EFL) enrolled in an English for Academic Purposes (EAP) course at a university in southeast Spain. Specifically, it represents an attempt to shed light on writers’ internal process factors by examining the longitudinal development of the students’ stored beliefs on the task and goals and their contribution to the development of L2 written performance. Data for the study included reflective journals, timed essays, and language proficiency tests collected nine months apart. The results indicate the development over time of advanced L2 students’ multidimensional models of composition into more sophisticated models, and the existence of two different views on task conceptualizations in terms of processes and products that affect the activation of a hierarchical network of goals for composing. The students’ views on the task were also found to be related to their motivation, self-regulation, and levels of writing achievement. The contribution of this study lies in furthering the understanding of writer-internal factors in relation to writing development, thus helping to advance both theoretical knowledge of second language writing and to improve pedagogical practice.
Article
In Section 1 of this article, the author discusses the succession of models of adult writing that he and his colleagues have proposed from 1980 to the present. He notes the most important changes that differentiate earlier and later models and discusses reasons for the changes. In Section 2, he describes his recent efforts to model young children’s expository writing. He proposes three models that constitute an elaboration of Bereiter and Scardamalia’s knowledge-telling model. In Section 3, he describes three running computer programs that simulate the action of the models described in Section 2.
Article
Feedback is one of the most powerful influences on learning and achievement, but this impact can be either positive or negative. Its power is frequently mentioned in articles about learning and teaching, but surprisingly few recent studies have systematically investigated its meaning. This article provides a conceptual analysis of feedback and reviews the evidence related to its impact on learning and achievement. This evidence shows that although feedback is among the major influences, the type of feedback and the way it is given can be differentially effective. A model of feedback is then proposed that identifies the particular properties and circumstances that make it effective, and some typically thorny issues are discussed, including the timing of feedback and the effects of positive and negative feedback. Finally, this analysis is used to suggest ways in which feedback can be used to enhance its effectiveness in classrooms.
Article
While there has been a proliferation of research on assessment for learning (AFL) over the past two decades, L2 writing assessment has tended to focus much more on assessment of learning (AOL) than AFL. This study seeks to investigate the implementation of AFL for EFL writing within an examination-driven AOL system in Hong Kong, its possible impact on students’ motivation and writing performance, as well as the factors that might facilitate or inhibit its uptake. Multiple sources of data were collected, including questionnaires, interviews, pre- and post-tests, and lesson observations. The findings showed that while teachers strengthened planning and pre-assessment instruction, sharing learning goals with students and using feedback forms to provide feedback and align assessment with instruction, they had to adhere to conventional practices that required detailed attention to errors and summative scores, and were unable to engage students in multiple drafting and peer evaluation on a regular basis. Results about students’ motivation are mixed, suggesting some tension between innovative and traditional assessment practices. Pre- and post-tests indicated some improvement in students’ writing performance, with teachers believing that AFL did have a role to play. The factors that facilitated and constrained teachers’ attempts to implement assessment for learning are discussed, and implications drawn.
Article
The idea that assessment is intrinsic to effective instruction is traced from early experiments in the individualization of learning through the work of Benjamin Bloom to reviews of the impact of feedback on learners in classrooms. While many of these reviews detailed the adverse impact of assessment on learning, they also indicated that under certain conditions assessment had considerable potential to enhance learning. It is shown that understanding the impact that assessment has on learning requires a broader focus than the feedback intervention itself, particularly the learner's responses to the feedback, and the learning milieu in which the feedback operates. Different definitions of the terms “formative assessment” and “assessment for learning” are discussed, and subsumed within a broad definition that focuses on the extent to which instructional decisions are supported by evidence. The paper concludes by exploring some of the consequences of this definition for classroom practice.
Article
Progression analysis (PA) is a computer-based, multilevel method for research on writing in the workplace. PA focuses on the situational context of the writing process (macro level), the movement of writing throughout the growing text (meso level), and the writer's consciously applied revising strategies (micro level). In this article, the innovative procedures of PA are presented on the basis of a pedagogically motivated case study from a national research project. Interconnections of the writing situation, movements, and strategies with the text product are made clear. The comparison with 17 other case studies reveals a regularity: experienced writers align their writing strategies more closely to text function and process control than inexperienced writers do.
Article
With a paradigm shift from a focus on product to one on process in language assessment, assessment for learning (AfL) has been gaining currency in educational policy in different parts of the world. While AfL emphasizes the use of assessment for improving learning and teaching, assessment of learning (AoL) focuses on using assessment for administrative and reporting purposes. In L2 writing, assessment has traditionally been characterized by AoL. Although AfL strategies like process pedagogy, formative feedback, peer response, and conferences have been promoted in L2 writing, these strategies are not widely adopted outside North American educational contexts. In English as a foreign language (EFL) contexts, there is scanty research that investigates writing teachers' attempts to bring innovation to their assessment practices through a focus on AfL. Using data from four Secondary 1 (i.e. Grade 7) classrooms in a Hong Kong school, the study aimed to investigate how the teachers' determination to implement AfL in writing influenced their instructional and assessment practices and impacted on students' attitudes and beliefs regarding writing. Results show that the implementation of AfL resulted in a significant change in teachers' instructional and assessment practices, and students improved their motivation in writing. The paper concludes with a few implications for EFL writing.
Article
This article is a review of the literature on classroom formative assessment. Several studies show firm evidence that innovations designed to strengthen the frequent feedback that students receive about their learning yield substantial learning gains. The perceptions of students and their role in self‐assessment are considered alongside analysis of the strategies used by teachers and the formative strategies incorporated in such systemic approaches as mastery learning. There follows a more detailed and theoretical analysis of the nature of feedback, which provides a basis for a discussion of the development of theoretical models for formative assessment and of the prospects for the improvement of practice.
Article
Preparing a written outline during prewriting and composing a rough 1st draft are strategies that may ease attentional overload and consequently enhance writing performance. The present research examined how these strategies affect the efficiency of the writing process and the quality of the written product. The processing time and cognitive effort given to planning ideas, translating ideas into text, and reviewing ideas and text were monitored by using directed retrospection and comparing secondary-task reaction times. The results of Exp 1 indicated that preparing a written outline, compared with not doing so, led to higher quality documents as indexed by ratings of judges. Composing a rough draft, as opposed to a polished draft, had no beneficial effect on writing quality. Exp 2 showed that a mental outline improved the quality of the documents as much as a written outline, indicating that the written outline was not serving as an external memory aid. Also, both mental and written outlines eased attentional overload by allowing the writer to focus processing time on the single process of translating ideas into text. (PsycINFO Database Record (c) 2012 APA, all rights reserved)
Article
The writing process has long been a subject for investigation. Until recently researchers have been restricted to written protocols for the analysis of writing sessions. These provide vast amounts of information from which it is impossible to create detailed mental representations of the writer’s movements around the text, revision activity, or pause behavior. Computer keystroke –logging programs, which record all keystrokes and mouse actions, facilitate the collection of quantitative data about text creation. This article presents the LS graph, a novel way of graphically representing and summarizing the quantitative data collected when keystroke logging. Further, the graph can be combined with a detailed manual analysis of the individual revisions that can be undertaken by playing back the logged writing session.
Article
The second-language writing performance of 23 young adults on three composition tasks was assessed in relation to their writing expertise and second-language proficiency. Both factors accounted for large proportions of variance in the qualities of written texts and problem-solving behaviors in the second language. But the factors exerted independent effects, suggesting they are psychologically distinct. Writing expertise proved to relate to: qualities of discourse organization and content in the compositions produced; attention to complex aspects of writing during decision making; problem-solving behaviors involving heuristic searches; and well-differentiated control strategies. Second-language proficiency proved to be an additive factor, enhancing the overall quality of writing produced, and interacting with the attention that participants devoted to aspects of their writing. But second-language proficiency did not visibly affect the processes of composing. In all analyses, more cognitively demanding argument and summary tasks produced significantly different behaviors from a less cognitively demanding letter task.
Article
This cross-sectional study used verbal protocol analysis to compare the temporal distribution of formulation processes of Spanish EFL writers composing L1 and L2 argumentative texts. We studied three groups at different levels of L2 proficiency. Results showed the same total formulation time regardless of whether participants wrote in L1 or L2. L2 proficiency, however, affected times. Higher-proficiency participants devoted less time to formulation, concentrated formulation in the central stages of composing, and increased the interaction between formulation and other subprocesses. Fluent formulation was twice as common as problem-solving formulation when writing in L2 and five times as common in L1; these ratios appeared to be invariant across proficiency. Theoretical and methodological implications for the study of L2 writing processes are discussed.
Article
Different ethical issues arise according to how the construct of writing is defined for assessment purposes. Most formal assessments assume a pragmatic, functional definition of second-language (L2) writing in which an examinee’s text production is judged normatively in respect to conventions for a discourse type or domain (Santos, 1992). This raises ethical concerns that have conventionally been addressed in high stakes language tests such as confidentiality, prior orientation, fairness, or equality of opportunity. But other, alternative definitions of writing are prominent in the pedagogical literature and in educational practices, particularly for users of second languages, for example, defining writing as a mode of learning, the expression of identity, or a medium for political action (e.g., reviewed in Cumming, 2001a; Hamp-Lyons & Kroll, 1997). Such constructs are rarely assessed in formal tests, seemingly because they raise moral issues, for example, concerning definitions of personal development, inter-cultural negotiation, knowledge, and emancipation, which necessarily have to be denied to fulfill the ethical criteria (e.g., confidentiality, prior orientation, fairness, and equality of opportunity) associated with the assessment of functional definitions of writing. I discuss these issues in respect to efforts to develop new task types for a new TOEFL, particularly in distinguishing between the constructs of writing tasks that solicit either independent expression or responses to substantive reading or listening material (Cumming, Kantor, Powers, Santos & Taylor, 2000).
Article
Although composing has long been recognised as recursive, so far there have been few studies on the temporal dimension of writing processes. This is regrettable given that one might expect the predominance of certain processes at different stages of writing at the expense of others and/or differences among writers with respect to the duration and distribution of the processes throughout the composing act. To shed light on these issues, we report on a study in which we (i) used protocol data to investigate whether the writer's proficiency level influences the total processing time devoted to writing processes and (ii) compared the differential distribution of the time allocated by different proficiency groups to different writing processes at each stage by dividing the writing session into three different stages. Two main findings emerged from the data: (i) formulation took up the largest percentage of composition time for all groups and (ii) writing processes are differentially distributed across the three periods depending on the writer's proficiency level. These findings are discussed with respect to their relevance for model building and suggestions for future research are advanced.
Article
A traditional writing pedagogy, learning-by-doing exercises, is criticised for its lack of focus on balancing writing and learning processes. Three variants of learning-by-observation are investigated as possible alternatives: observing writers as models (OW), observing both writers and readers as models (OWR), and observing readers as feedback on writing performance (FW).Observations were made by means of authentic video-tape recordings of student writers or readers (model conditions), or by live confrontations between writers and their readers (feedback condition). Training focused on argumentative text. Participants were pre- and post-tested on reading skill and writing skill in order to measure learning and transfer effects.Results show that all observation conditions were more effective than the learning-by-doing condition: OW and FW showed larger learning effects (on writing skill) and larger transfer effects (on reading skill). Condition OWR only showed larger transfer effects. It is concluded that the effective components of learning-by-observation deserve to be studied in more detail.
Article
It is widely assumed that the constraints of timed essay exams will make it virtually impossible for students to engage in the major hallmarks of the writing process, especially revision, in testing situations. This paper presents the results of a study conducted at Washington State University in the Spring of 2008. The study examined the occurrence of prewriting and revision in 890 timed essay responses as well as the impact of writing process on student scores. It was found that both prewriting and revision occur more frequently in timed essays than was previously realized. While prewriting corresponded to higher scores, revision corresponded to lower scores. These results encourage composition scholars to reevaluate their assumptions about both the validity of timed writing exams and the efficacy of current practices in teaching the writing process.
Article
In CSCL research, collaboration through chat has primarily been studied in dyadic settings. This article discusses three issues that emerged during the development of a multi-dimensional coding procedure for small-group chat communication: (a) the unit of analysis and unit fragmentation, (b) the reconstruction of the response structure and (c) determining reliability without overestimation. Threading, i.e. connections between analysis units, proved essential to handle unit fragmentation, to reconstruct the response structure and for reliability of coding. In addition, a risk for reliability overestimation was identified. Implications for analysis methodology in CSCL are discussed.
Article
This study compares second language learners' L1 writing, L2 writing, and translation from L1 into L2, focusing on writing and translating processes, attention patterns, and quality of language use. Thinking aloud, 22 Japanese ESL students studying at a Canadian college performed 3 tasks individually. These think-aloud protocols were analyzed, supplemented by observational notes and interviews, and the writing samples were evaluated. The data were analyzed with attention to theories of composing processes (Bereiter & Scardamalia, 1987), Schmidt's “conscious attention” (1990), and Swain's “i + 1 output” hypothesis (1985).It was found that (a) most students used a “what-next” approach both in the L1 and L2 writing tasks and a “sentence-by-sentence” approach in the translation task, (b) attention patterns in the L1 and L2 writing tasks were very similar, but quite different in the translation task. Attention to language use in the translation task was significantly higher than in the L1 and L2 writing tasks and, (c) scores on language use in the L1 and L2 writing tasks were similar, but scores on language use in the translation task were significantly better than in the L2 writing task.
Article
The present study throws some light into an area that has been relatively untouched: it analyzes how differences in the beliefs or metacognitive knowledge (MK) held about writing relates to differences in English as a Foreign Language (EFL) writing skills. Data were collected from four undergraduate university Spanish students, two good writers and two poor writers, enrolled in EFL classes at the University of Barcelona. They were first required to take an English test and write an argumentative essay to assess their language and writing proficiency. Subsequently, they were interviewed and required to think aloud as they wrote another argumentative essay. This study revealed a number of areas in which the knowledge of the two pairs clearly differed. On the whole, these differences pointed to a more appropriate and comprehensive view of the writing process, which they were able to apply more flexibly. In contrast, the less successful writers' MK was limited and inadequate. Furthermore, the case studies also revealed the clear relationship that exists between the MK of the writers and the strategies they deployed, underscoring the major role played by MK in providing a rationale for the learners' approach to writing and researchers with a more thorough understanding of the learners' writing process.
Article
The present study describes the way one institution operationalized the notion of process-oriented writing assessment in its entry placement testing context. In an attempt to enhance how it assesses academic writing ability, a workshop-based essay test was proposed in place of the existing, traditional timed essay test. The study looked at how the tests from the two different approaches in writing assessment (product-oriented and process-oriented) affected examinees’ test performance by comparing both the textual quality of the test essays and the placement results. Content-area faculty evaluations of the examinees’ writing ability and the examinees’ view on the tests were also used in evaluating the quality of the tests. The results suggested that assessment methods have an impact on the examinees’ test performance. Even though further evidence is needed for a better understanding of the result, a traditional method of direct writing assessment seemed to put examinees at a disadvantage. The new test was implemented in the given research context and implications of writing assessment in general are discussed.