ArticlePDF Available

Sources of text difficulty: Across genres and grades

Authors:
... While the nature of genre is debated, it is generally agreed upon that different text genres have different purposes, are marked by different linguistic features, and can be reliably distinguished by readers. For instance, social studies and science texts have more challenging words and sentence structures compared to narrative texts (McNamara, Graesser & Louwerse, 2012). These aspects of text difficulty further translate to differences in text cohesion. ...
... For example, narratives tend to contain less referential cohesion, but more connectives compared to science texts. This linguistic variation drives differences in the coherence-building processes that afford comprehension (McNamara et al., 2012). ...
... These features of texts also have top-down effects on individuals' coherencebuilding processes and subsequent comprehension (McNamara et al., 2012;Zwaan, 1994). For example, readers' genre expectations have been shown to affect the cognitive top-down processes that occur while they construct and update their mental representations during reading (Kintsch, 1992;Parodi, 2014;Schmitz, Gräesel & Rothstein, 2017). ...
Article
Full-text available
Successful text comprehension requires readers to engage in a number of coherence-building processes. This study examined how analyzing the cohesion of students’ constructed responses can be used to evaluate these coherence-building processes and the extent to which they vary across readers’ individual differences and across types of texts. We posed two primary research questions: 1) Can we predict individual differences in working memory and reading skill based on the cohesion of students’ constructed responses to text? 2) Do the relations between individual differences and cohesion vary as a function of genre? Participants (n = 119) generated constructed responses while reading history and science texts and completed reading skill and working memory assessments. The current study leveraged natural language processing (NLP) techniques to analyze the cohesion of readers’ constructed responses, using cohesion as a proxy for assessing the coherence of their mental representations of the texts. Cohesion was measured at the sentence, paragraph, and synonym levels. Machine learning models showed that linguistic indices related to cohesion were significant predictors of both working memory and reading skill. Additional quantitative and qualitative inspection revealed that the relations between individual differences and coherence-building processes varied depending on the text’s genre. These findings indicate that the interaction between genre and individual differences may be used to model coherence-building processes during reading. This study has important implications for the realm of educational technology such as in the implementation of stealth assessments to predict students’ cognitive abilities.
... The principal difference in this study was the added focus on multimodal elements of the text segments in comprehension assessment. Because there is evidence that multiple representations can increase cognitive demands (Cromley et al., 2016;McNamara et al., 2012), we wanted to test their effects with more competent readers who are likely more equipped to deal with the added complexity than younger or struggling readers. Further, we incorporated an additional calibration measure in this study that centered on topic knowledge. ...
Article
The comprehension and calibration of 54 undergraduates were investigated as they read excerpts from an introductory geology textbook on weather and soil in print and digitally. All excerpts were approximately 1600 words in length and contained a graph, a diagram, and three photographs that complemented or extended the written text. Each student read two texts with medium and topic counterbalanced. Prior to reading, the students completed a demographic survey, rated their topic familiarity, and completed two topic knowledge pretests. They next read one chapter on either weather or soil in print or digitally and then answered a series of short-answer questions. The questions drew on content from the written text only, visuals only, or both. The same procedure was then repeated in the other medium. Analyses indicated processing multimodal texts in print was significantly more advantageous than processing those same texts digitally, and this difference was more pronounced for questions focused on visuals only. Students' self-rated topic familiarity was compared to their demonstrated topic knowledge for weather and soil and their predicted comprehension performance was compared to actual comprehension performance. Results showed that undergraduates' calibration was poor overall , but comprehension was overestimated more often when students read multimodal texts digitally.
... Ils peuvent combiner par exemple la description, l'énumération, la comparaison, l'exposition de chaines causales ou de raisonnements visant à résoudre un problème (Bianco, 2017). Ces deux types de textes se distinguent donc par leur structure, mais aussi par leur composition lexicale et morphosyntaxique (McNamara et al., 2012). Des mots plus familiers mais une syntaxe plus compliquée sont privilégiés dans les textes narratifs. ...
Thesis
Full-text available
L’apprentissage de la lecture est une activité complexe qui requiert, au CP, un enseignement explicite et structuré, souvent guidé par une méthode de lecture (i.e., un ensemble d’outils pour l’enseignant et les élèves) éditée. Une équipe pluridisciplinaire, composée d’enseignants, de chercheurs et d’un éditeur (les Éditions Hatier), a choisi de proposer une nouvelle méthode de lecture pour le CP, basée sur les preuves : la méthode Lili CP. Une telle méthode se doit d’être utile (efficace) pour les apprentissages des élèves, mais elle doit aussi être utilisable (facile à prendre en main) et acceptable (compatible avec la classe) pour les enseignants et les élèves, afin de pouvoir être largement adoptée. L’objectif principal de cette thèse était d’évaluer l’utilité, l’utilisabilité et l’acceptabilité de certains outils et de certaines séquences de la méthode en cours de conception, afin d’identifier des pistes concrètes d’amélioration.Notre recherche a débuté par une analyse des pratiques des potentiels futurs utilisateurs. Un questionnaire diffusé à large échelle a mis en évidence la grande diversité des pratiques enseignantes au CP et les principaux critères de choix d’une méthode de lecture. Une étude a révélé l’excellent niveau d’utilisabilité et d’acceptabilité du matériel original d’entrainement à la combinatoire prévu dans Lili CP. Deux interfaces différentes du guide pédagogique au format web ont été comparées, en terme d’utilisabilité et d’acceptabilité également, permettant de dégager la pertinence de certains choix de présentation pour la future méthode. Dans une étude expérimentale, nous avons évalué l’efficacité d’une séquence d’enseignement explicite de la compréhension conçue pour Lili CP, sur les acquis des élèves dans ce domaine. La comparaison à un groupe contrôle actif (i.e., ayant suivi une autre séquence, plus classique, d’enseignement de la compréhension) a démontré l’intérêt de ce type de séquence pour la compétence entrainée.Enfin, deux versions (basique et gamifiée) de l’application numérique ECRIMO, développée pour Lili CP et visant à entrainer l’écriture de mots en autonomie, ont été évaluées sur les trois dimensions d’utilité, d’utilisabilité et d’acceptabilité. L’application, dans ses deux versions, obtient d’excellents scores d’utilisabilité et d’acceptabilité. Les entrainements avec ECRIMO, dans ses deux versions, se sont révélés aussi efficaces qu’un entrainement à l’encodage sous forme d’exercices classiques de dictée dirigés par l’enseignant. Dans tous les groupes entrainés, les progrès en encodage sont plus importants que dans le groupe contrôle et sont visibles surtout chez les élèves ayant déjà un bon niveau d’encodage en début de CP. Enfin, pour ces élèves, la version basique a engendré un progrès plus important que la version gamifiée.Ce travail doctoral apporte une démonstration de la possibilité et de l’intérêt de conduire une évaluation intégrée des outils éducatifs qui doivent être étudiés dans les trois dimensions d’utilité, d’utilisabilité et d’acceptabilité, avant leur diffusion à grande échelle sur le terrain. Il se conclut par la proposition d’une nouvelle démarche intégrée de conception et d’évaluation d’outils pédagogiques.
... .001), which has been found to be more prominent within the texts composed for children and for narrative texts [59]. The use of causal words in integrating ideas are linked to more concise explanations [60]. ...
... Thus, coherence building and monitoring in expository texts is more demanding for readers. Unlike expository texts, narratives often follow a standardized structure, engage readers emotionally, refer to familiar situations, and contain shorter and more common words (Appel et al., 2021;McNamara et al., 2012). Accordingly, younger adolescents have greater difficulty detecting inconsistencies in expository than in narrative texts (Currie et al., 2021;Zabrucky & Ratner, 1992). ...
... A limitation in RQ2 was that text difficulty was only manipulated through word frequency, and larger differences in reading level could have been achieved by increasing sentence length and/or syntactic complexity (e.g., McNamara, Graesser, & Louwerse, 2012). But the more relevant change, in an applied context, would likely be to reduce text difficulty. ...
Article
Text highlighting is a novel method for measuring consumer attitudes where participants read information about a focal topic and use highlighting functions to select aspects of the text that they like and dislike. The present research contributed methodological knowledge about text highlighting by investigating how responses are influenced by two aspects of the texts — length and degree of reading difficulty. A case study pertaining to biodynamic agriculture was used to assess the research questions and empirical data were collected from 3718 consumers across four countries (United Kingdom, Australia, Germany, and Singapore). Results showed that both text length and reading difficulty influenced responses — overall frequency of highlighting, frequency of ‘like’ highlighting, frequency of ‘dislike’ highlighting, and sentiment scores — leading to recommendations about future implementations of the methodology. Specifically, a single highlighting task on a longer text is recommended less than consecutive highlighting tasks on shorter texts. Implementation of the latter increases highlighting frequency and is expected to be associated with greater participant task engagement. Text length also influenced sentiment scores but did so in a manner that was topic and content specific. Regarding text difficulty, significant differences were established for all types of highlighting responses, although the differences were smaller than found for text length. The recommendation is to use simple and familiar language that is suited to the groups of participants taking part in the study. A general recommendation is to interpret findings in the context of the presented information.
Article
Full-text available
Communicating clearly about their socially responsible activities is becoming increasingly important for companies, as a growing number of stakeholders with different goals, knowledge, and language skills seek information on corporate social responsibility (CSR). Furthermore, the ability to communicate clearly is particularly appreciated in the workplace. To fill a gap in CSR communication training, this article describes the development and preliminary evaluation of an interdisciplinary and multimodal online module whose goal is to train Dutch-speaking business students in the production of accessible CSR content in English. After presenting our module, we discuss its implications for future training and for corporate communication.
Thesis
Full-text available
The lack of integration of cognitive science and psychometrics is commonly deplored - and ignored. In the assessment of reading, one manifestation of this problem is a theoretical avoidance regarding sources of text difficulty and cognitive processes underlying text comprehension. To facilitate the desired integration of cognitive science and psychometrics, we adopt a computational approach. By considering computational procedures as simplified and partial representations of cognitivist models, a computational approach facilitates the integration of theoretical elements in psychometrics, as well as the development of theories in cognitive psychology. This thesis studies the contribution of a computational perspective to the measurement of two facets of linguistic complexity, using complementary perspectives. Intrinsic text complexity is approached from the perspective of natural language processing, with the goal of identifying and measuring text features that best model text difficulty. Paper 1 introduces ISLA (Integrated Lexico-Syntactic Analyzer), a new natural language processing tool that extracts a variety of linguistic features from French text, primarily taken from research in psycholinguistics and computational linguistics. We then evaluate the features’ potential to estimate text difficulty. Paper 2 uses ISLA and statistical learning methods to estimate difficulty of texts used in primary and secondary education in Quebec. In the second part of the thesis, complexity associated with reading processes is addressed using eye-tracking, which allows inferences to be made about cognitive load and visual attention allocation strategies in reading. Paper 3 describes a methodology for analyzing mobile eye-tracking recordings using computer vision techniques (a branch of artificial intelligence); this methodology is then tested on simulated data. Paper 4 deploys the same methodology in the context of an eye-tracking pilot experiment comparing reading processes in novices and experts during an argumentative text iv comprehension test. Overall, our work demonstrates that it is possible to obtain convincing results by combining theoretical contributions with a computational approach using statistical learning techniques. The tools created or perfected in the context of this thesis constitute a significant advance in the development of digital technologies for the measurement and evaluation of reading, with easy-to-identify applications in both academic and research contexts.
Article
Full-text available
In this paper, we highlight the importance of distilling the computational assessments of constructed responses to validate the indicators/proxies of constructs/trins using an empirical illustration in automated summary evaluation. We present the validation of the Inbuilt Rubric (IR) method that maps rubrics into vector spaces for concepts’ assessment. Specifically, we improved and validated its scores’ performance using latent variables, a common approach in psychometrics. We also validated a new hierarchical vector space, namely a bifactor IR. 205 Spanish undergraduate students produced 615 summaries of three different texts that were evaluated by human raters and different versions of the IR method using latent semantic analysis (LSA). The computational scores were validated using multiple linear regressions and different latent variable models like CFAs or SEMs. Convergent and discriminant validity was found for the IR scores using human rater scores as validity criteria. While this study was conducted in the Spanish language, the proposed scheme is language-independent and applicable to any language. We highlight four main conclusions: (1) Accurate performance can be observed in topic-detection tasks without hundreds/thousands of pre-scored samples required in supervised models. (2) Convergent/discriminant validity can be improved using measurement models for computational scores as they adjust for measurement errors. (3) Nouns embedded in fragments of instructional text can be an affordable alternative to use the IR method. (4) Hierarchical models, like the bifactor IR, can increase the validity of computational assessments evaluating general and specific knowledge in vector space models. R code is provided to apply the classic and bifactor IR method.
Article
Full-text available
Students often struggle with developing understanding from expository science texts. This study explored whether training students to engage in a POE (Predict-Observe-Explain) study strategy might be beneficial when learning from texts that introduce theories by describing experiments and empirical results, a common style in social science textbooks. The main questions tested in this experiment were if training students how to use a POE study strategy while reading textbook excerpts would support better comprehension and comprehension monitoring outcomes when students engaged in future learning attempts for an introductory psychology class. In one condition students were trained to use the POE study strategy, while in a comparison condition students were simply trained to use an explanation study strategy. Analyses suggested that students in the POE strategy training condition may have become preoccupied with whether or not their experimental predictions were correct, prohibiting them from engaging with the POE strategy as intended. Although both POE and explanation strategy training helped students to improve their comprehension monitoring on a new set of texts, students in the explanation condition displayed better comprehension on those new texts than students in the POE condition.
ResearchGate has not been able to resolve any references for this publication.