Conference Paper

Graph-based Local Coherence Modeling

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

We propose a computationally efficient graph-based approach for local coherence modeling. We evaluate our system on three tasks: sentence ordering, summary coherence rating and readability assessment. The performance is comparable to entity grid based approaches though these rely on a computationally expensive training phase and face data sparsity problems.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... In general, a coherent discourse generally has many similar components (lexical overlap or coreference) across sentences within a text, while incoherent discourse is the other one. Therefore, the traditional cohesion theory of Centering [10] driven and entity-based model [11] [12][13] [14] was proposed to capture the syntactic or semantic distribution of discourse entities (nouns) between two adjacent sentences in a text. Thereafter, many extension works were presented such as Feng and Hirst [15] 's multiple ranking model, Lin et al. [16] 's discourse relationbased approach, Louis and Nenkova [17] 's syntactic patterns-based model. ...
... Evaluation Metric: We report system's performance using accuracy, which is the ratio of the number of the selected original text/translation document divided by the total number of texts/translation document. Baseline System 1: Entity graph based model [14] which has been demenstrated as a simple but effective implementation of the entity-based coherence model. We re-implement their method in this paper based on publicly available code 3 . ...
... The performance decreases with the increment of the window size, and the best performance yields at the window size with 3. It is mostly caused by the local entity distribution characteristic demenstrated by Barzilay and Lapata [11] [12] ,Guinaudeau and Strube [14] . As the increment of the number of the window size, the entity co-occurance decreases accordingly. ...
Preprint
Chinese discourse coherence modeling remains a challenge taskin Natural Language Processing field.Existing approaches mostlyfocus on the need for feature engineering, whichadoptthe sophisticated features to capture the logic or syntactic or semantic relationships acrosssentences within a text.In this paper, we present an entity-drivenrecursive deep modelfor the Chinese discourse coherence evaluation based on current English discourse coherenceneural network model. Specifically, to overcome the shortage of identifying the entity(nouns) overlap across sentences in the currentmodel, Our combined modelsuccessfully investigatesthe entities information into the recursive neural network freamework.Evaluation results on both sentence ordering and machine translation coherence rating task show the effectiveness of the proposed model, which significantly outperforms the existing strong baseline.
... Typically this topic transition is approximated by extracting salient discourse entities from a document (for instance, the subject and object of each sentence) and measuring their occurrence (and distance) through sentences in an entity grid [1] (see example in Table 1). Recently the elements of such an entity grid (i.e., the discourse entities and the sentences in which they occur) have been represented as a graph, the topology of which has been used to approximate document coherence, for instance as the average out-degree [14], pagerank, clustering coefficient, or betweenness [34] computed over the whole graph (each graph representing a single document). This type of graph-based coherence modelling, despite being completely unsupervised, performs comparably to equivalent supervised approaches, thus showing great promise. ...
... Existing graph-based computations of text coherence [14,34] represent each document as a bipartite graph of sentences and their discourse entities. A bipartite graph is a particular class of graph also known as two-mode graph. ...
... In addition, our bipartite metrics incur no additional efficiency cost over existing one-mode graph metrics. One of our metrics is shown to be a much more accurate approximation of document coherence than the state of the art computed from one-mode projections [14,34] (Section 4). All three coherence metrics are shown to be useful to retrieval effectiveness (Section 5). ...
Preprint
Document coherence describes how much sense text makes in terms of its logical organisation and discourse flow. Even though coherence is a relatively difficult notion to quantify precisely, it can be approximated automatically. This type of coherence modelling is not only interesting in itself, but also useful for a number of other text processing tasks, including Information Retrieval (IR), where adjusting the ranking of documents according to both their relevance and their coherence has been shown to increase retrieval effectiveness [34,37]. The state of the art in unsupervised coherence modelling represents documents as bipartite graphs of sentences and discourse entities, and then projects these bipartite graphs into one-mode undirected graphs. However, one-mode projections may incur significant loss of the information present in the original bipartite structure. To address this we present three novel graph metrics that compute document coherence on the original bipartite graph of sentences and entities. Evaluation on standard settings shows that: (i) one of our coherence metrics beats the state of the art in terms of coherence accuracy; and (ii) all three of our coherence metrics improve retrieval effectiveness because, as closer analysis reveals, they capture aspects of document quality that go undetected by both keyword-based standard ranking and by spam filtering. This work contributes document coherence metrics that are theoretically principled, parameter-free, and useful to IR.
... The subgraph matching algorithm mines the frequent subgraph patterns in the sentence semantic graph to capture specific coherence patterns in the English text and subdivides the weights of the edges in the graph to measure the specific degree of coherence between sentences. Using the above approach, we designed a semantic coherence analysis model for English texts based on the work of Guinaudeau and Strube et al. [5]. The main contributions of this paper are as follows: (1) To address the problem of insufficient semantic information between sentences in the traditional entity graph model, under the guidance of semantic coherence theory, the Word2Vec word embedding model is used to represent the English text in the semantic space and combine the semantic similarity information between sentences with the entity information in the entity graph to construct semantic associations between sentences, thus representing the English text as a semantic graph of sentences containing rich semantic information. ...
... Rangjun Li [16] suggested that considering both intrinsic attribute information and inter-sample structural information simultaneously enhances the feature recognition capability of the model. Guinaudeau and Strube et al. [5] extended the entity grid into an entity graph model to represent a text in a graph, thus analysing the coherence of the text in its entirety. Takenobu Tokunaga et al. [17] constructed semantic similarity graphs by means of word embedding, so that the degree of semantic relevance of different sentences could be distinguished. ...
... In summary, researchers have proposed research methods for the study of text coherence quality in a number of ways, with one of the more popular ideas being a series of improvements and extensions based on the solid grid model. Based on Guinaudeau and Strube et al. [5], this paper designs a semantic coherence analysis model for English text by fusing entity-based construction of entity graphs with semantic similarity graphs based on semantic similarity into a sentence semantic graph and capturing the unique coherence patterns in English text by mining frequent subgraph patterns in the sentence semantic graph with a subgraph matching algorithm. ...
Article
Full-text available
With the reform of China's education industry, more and more universities are using computers to conduct examinations. For the automatic correction of essays as subjective questions, existing automatic English text scoring systems suffer from insufficient extraction of coherence information and low accuracy when analysing text coherence. Therefore, this paper proposes an unsupervised semantic coherence analysis model for English texts based on sentence semantic graphs, taking Chinese students' English compositions as the research context. Guided by the semantic coherence theory, the English text is represented as a sentence semantic graph, and an improved VF2 subgraph matching algorithm is used to mine the frequently occurring subgraph patterns in the sentence semantic graph. After that, the set of frequent subgraphs is generated by filtering the subgraph patterns according to their frequencies, and the subgraph frequency of each frequent subgraph is calculated separately. Finally, the distribution characteristics of frequent subgraphs and the semantic values of subgraphs in the sentence semantic graphs are extracted to quantify the overall coherence quality of English texts. The experimental results show that the model proposed in this paper has higher accuracy and practical value compared with the current methods of coherence analysis.
... Motivated by the Centering theory (Joshi and Weinstein, 1981), many approaches to local coherence modeling rely on entity relations between sentences. The entity grid Lapata, 2005, 2008) and the entity graph (Guinaudeau and Strube, 2013) are two well-studied frameworks for representing entity relations in a text. Entity grid-based models use grids while entity graph-based models use graphs to capture entity relations between sentences. ...
... We compare any of these texts with 20 permutations. For SCR, we use the dataset proposed by Barzilay and Lapata (2008) and used by prior work for coherence evaluation (Guinaudeau and Strube, 2013;Tien Nguyen and Joty, 2017). The dataset comprises texts from the DUC-2003 corpus, which contains English summaries produced by human experts and extractive summarization systems. ...
... Settings We compare our model (Section 2) with the following coherence models: EntGraph (Guinaudeau and Strube, 2013), Neural EntGrid (Tien Nguyen and Joty, 2017), Lex. Neural EntGrid (Joty et al., 2018) results on our machines. ...
... Later, Guinaudeau and Strube extend the entity-grid model to an entity-graph model and argue for the proposed model's computational efficiency and comparable performance. [30]. ...
... The most common evaluation task is sentence ordering (shuffle test), where the goal is to identify the most coherent order from the set of jumbled sentences. Sentence ordering is designed either as a generative task, where the model aims to find an optimal ordering of sentences that maximizes the coherence [9], [15]- [17], [33], or as a discriminative task, where the model learns to correctly rank the original document higher than some random permutations of the same sentences [5], [10], [18], [19], [30]. Recently, Farag et al. [32] extended the sentence ordering task to sentence pair discrimination task, where the goal is to identify the original sentence pair among its linguistically perturbed variations. ...
... In this experiment, we use two well-explored datasets -Accidents and Earthquake [5]. Both datasets have been extensively used in previous studies for local and sequential coherence analysis [5], [16], [30], [42], [43]. ...
Article
Full-text available
In the era of heavy emphasis on deep neural architectures with well-proven and impressive competencies albeit with massive carbon footprints, we present a simple and inexpensive solution founded on BERT Next Sentence Prediction (NSP) task to localize sequential discourse coherence in a text. We propose Fast and Frugal Coherence Detection (FFCD) method, which is an effective tool for the author to incarcerate regions of weak coherence at the sentence level and reveal the extent of overall coherence of the document in near real-time. The mixed performance of our solution compared to state-of-the-art methods for coherence detection invigorates efforts to design explainable and inexpensive solutions downstream of the existing upscale language models.
... Coherence measurement has been studied across various tasks, such as the document discrimination task (Barzilay and Lapata, 2005;Elsner et al., 2007;Barzilay and Lapata, 2008;Elsner and Charniak, 2011;Li and Jurafsky, 2017;Putra and Tokunaga, 2017), sentence insertion (Elsner and Charniak, 2011;Putra and Tokunaga, 2017;Xu et al., 2019), paragraph reconstruction (Lapata, 2003;Elsner et al., 2007;Li and Jurafsky, 2017;Xu et al., 2019;Prabhumoye et al., 2020), summary coherence rating (Barzilay and Lapata 2005;Pitler et al., 2010;Guinaudeau and Strube, 2013;Tien Nguyen and Joty, 2017), readability assessment (Guinaudeau and Strube, 2013;Mesgar andStrube, 2016, 2018), and essay scoring (Mesgar and Strube, 2018;Somasundaran et al., 2014;Tay et al., 2018). These tasks differ from our task of intruder sentence detection as follows. ...
... Coherence measurement has been studied across various tasks, such as the document discrimination task (Barzilay and Lapata, 2005;Elsner et al., 2007;Barzilay and Lapata, 2008;Elsner and Charniak, 2011;Li and Jurafsky, 2017;Putra and Tokunaga, 2017), sentence insertion (Elsner and Charniak, 2011;Putra and Tokunaga, 2017;Xu et al., 2019), paragraph reconstruction (Lapata, 2003;Elsner et al., 2007;Li and Jurafsky, 2017;Xu et al., 2019;Prabhumoye et al., 2020), summary coherence rating (Barzilay and Lapata 2005;Pitler et al., 2010;Guinaudeau and Strube, 2013;Tien Nguyen and Joty, 2017), readability assessment (Guinaudeau and Strube, 2013;Mesgar andStrube, 2016, 2018), and essay scoring (Mesgar and Strube, 2018;Somasundaran et al., 2014;Tay et al., 2018). These tasks differ from our task of intruder sentence detection as follows. ...
... To assess local coherence, traditional studies have used entity matrices, for example, to represent entity transitions across sentences Lapata, 2005, 2008). Guinaudeau and Strube (2013) and Mesgar and Strube (2016) use a graph to model entity transition sequences. Sentences in a document are represented by nodes in the graph, and two nodes are connected if they share the same or similar entities. ...
Article
Full-text available
While pretrained language models (LMs) have driven impressive gains over morpho-syntactic and semantic tasks, their ability to model discourse and pragmatic phenomena is less clear. As a step towards a better understanding of their discourse modeling capabilities, we propose a sentence intrusion detection task. We examine the performance of a broad range of pretrained LMs on this detection task for English. Lacking a dataset for the task, we introduce INSteD, a novel intruder sentence detection dataset, containing 170,000+ documents constructed from English Wikipedia and CNN news articles. Our experiments show that pretrained LMs perform impressively in in-domain evaluation, but experience a substantial drop in the cross-domain setting, indicating limited generalization capacity. Further results over a novel linguistic probe dataset show that there is substantial room for improvement, especially in the cross- domain setting.
... The Entity Grid (EGrid) model Lapata, 2005, 2008) is one of the key coherence models that spurred from Centering theory; it creates an abstract representation of text that tracks entity distribution and the transition of the syntactic roles entities take across sentences. The EGrid approach has been adapted and further enhanced in numerous coherence models (Elsner et al., 2007;Filippova and Strube, 2007;Burstein et al., 2010;Cheung and Penn, 2010a;Elsner and Charniak, 2011b;Feng and Hirst, 2012;Guinaudeau and Strube, 2013). ...
... In addition, coherence modeling has been frequently paired with information insertion and information ordering tasks. In information insertion, a sentence is pulled out of a text and the model is tasked with inserting it back in its original place (Chen et al., 2007;Elsner andCharniak, 2008, 2011b;Guinaudeau and Strube, 2013;Tien Nguyen and Joty, 2017); this is useful in community edited web resources such as Wikipedia that require continuous update and insertion of new information (Chen et al., 2007). In information ordering, a model is asked to organise a given set of sentences to form a coherent text (Lapata, 2003;Barzilay and Lee, 2004;Bollegala et al., 2006;Gong et al., 2016;Li and Jurafsky, 2017;Cui et al., 2018;Logeswaran et al., 2018;Yin et al., 2019a;Wang and Wan, 2019;Oh et al., 2019;Kumar et al., 2020a), which has utility in text generation applications such as ordering the sentences produced by multi-document summarisers (Lapata, 2003). ...
... Their enhanced EGrid version outperformed the basic one in coherence discrimination and sentence insertion tasks. Guinaudeau and Strube (2013) adapted EGrids in a graph-based framework where there are two sets of nodes: sentences and entities. An entity is linked to the sentence it appears in via an edge weighted by its grammatical role, and entity transitions between sentences are modeled by a projection graph, where two sentences are connected if they share the same entity. ...
Thesis
Discourse coherence is an important aspect of text quality that refers to the way different textual units relate to each other. In this thesis, I investigate neural approaches to modeling discourse coherence. I present a multi-task neural network where the main task is to predict a document-level coherence score and the secondary task is to learn word-level syntactic features. Additionally, I examine the effect of using contextualised word representations in single-task and multi-task setups. I evaluate my models on a synthetic dataset where incoherent documents are created by shuffling the sentence order in coherent original documents. The results show the efficacy of my multi-task learning approach, particularly when enhanced with contextualised embeddings, achieving new state-of-the-art results in ranking the coherent documents higher than the incoherent ones (96.9%). Furthermore, I apply my approach to the realistic domain of people’s everyday writing, such as emails and online posts, and further demonstrate its ability to capture various degrees of coherence. In order to further investigate the linguistic properties captured by coherence models, I create two datasets that exhibit syntactic and semantic alterations. Evaluating different models on these datasets reveals their ability to capture syntactic perturbations but their inadequacy to detect semantic changes. I find that semantic alterations are instead captured by models that first build sentence representations from averaged word embeddings, then apply a set of linear transformations over input sentence pairs. Finally, I present an application for coherence models in the pedagogical domain. I first demonstrate that state of-the-art neural approaches to automated essay scoring (AES) are not robust to adversarially created, grammatical, but incoherent sequences of sentences. Accordingly, I propose a framework for integrating and jointly training a coherence model with a state-of-the-art neural AES system in order to enhance its ability to detect such adversarial input. I show that this joint framework maintains a performance comparable to the state-of-the-art AES system in predicting a holistic essay score while significantly outperforming it in adversarial detection.
... Coherence measurement has been studied across various tasks, such as the document discrimination task (Barzilay and Lapata, 2005;Elsner et al., 2007;Barzilay and Lapata, 2008;Elsner and Charniak, 2011;Li and Jurafsky, 2017; Putra and Tokunaga, 2017), sentence insertion (Elsner and Charniak, 2011;Putra and Tokunaga, 2017;Xu et al., 2019), paragraph reconstruction (Lapata, 2003;Elsner et al., 2007;Li and Jurafsky, 2017;Xu et al., 2019;Prabhumoye et al., 2020), summary coherence rating (Barzilay and Lapata, 2005;Pitler et al., 2010;Guinaudeau and Strube, 2013;Tien Nguyen and Joty, 2017), readability assessment (Guinaudeau and Strube, 2013;Strube, 2016, 2018), and essay scoring (Mesgar and Strube, 2018;Somasundaran et al., 2014;Tay et al., 2018). These tasks differ from our task of intruder sentence detection as follows. ...
... Coherence measurement has been studied across various tasks, such as the document discrimination task (Barzilay and Lapata, 2005;Elsner et al., 2007;Barzilay and Lapata, 2008;Elsner and Charniak, 2011;Li and Jurafsky, 2017; Putra and Tokunaga, 2017), sentence insertion (Elsner and Charniak, 2011;Putra and Tokunaga, 2017;Xu et al., 2019), paragraph reconstruction (Lapata, 2003;Elsner et al., 2007;Li and Jurafsky, 2017;Xu et al., 2019;Prabhumoye et al., 2020), summary coherence rating (Barzilay and Lapata, 2005;Pitler et al., 2010;Guinaudeau and Strube, 2013;Tien Nguyen and Joty, 2017), readability assessment (Guinaudeau and Strube, 2013;Strube, 2016, 2018), and essay scoring (Mesgar and Strube, 2018;Somasundaran et al., 2014;Tay et al., 2018). These tasks differ from our task of intruder sentence detection as follows. ...
... To assess local coherence, traditional studies have used entity matrices, e.g. to represent entity transitions across sentences Lapata, 2005, 2008). Guinaudeau and Strube (2013) and Mesgar and Strube (2016) use a graph to model entity transition sequences. Sentences in a document are represented by nodes in the graph, and two nodes are connected if they share the same or similar entities. ...
Preprint
While pretrained language models ("LM") have driven impressive gains over morpho-syntactic and semantic tasks, their ability to model discourse and pragmatic phenomena is less clear. As a step towards a better understanding of their discourse modelling capabilities, we propose a sentence intrusion detection task. We examine the performance of a broad range of pretrained LMs on this detection task for English. Lacking a dataset for the task, we introduce INSteD, a novel intruder sentence detection dataset, containing 170,000+ documents constructed from English Wikipedia and CNN news articles. Our experiments show that pretrained LMs perform impressively in in-domain evaluation, but experience a substantial drop in the cross-domain setting, indicating limited generalisation capacity. Further results over a novel linguistic probe dataset show that there is substantial room for improvement, especially in the cross-domain setting.
... The second is the Jaccard similarity index between the sets of words in the essay and the prompt, as well as the keywords from both. • Local Coherence: We extracted six features using the TRUNAJOD library [Palma and Atkinson 2018], which is based on the entity grid model proposed in [Guinaudeau and Strube 2013]. This model is designed to compute the overlap of entities between subsequent sentences. ...
... N refers to the number of essays. Sentences and Words are shown as Mean (Standard Deviation).Considering the previous works in the literature on essay scoring and cohesion analysis[Guinaudeau and Strube 2013, Ferreira-Mello et al. 2019, Ferreira Mello et al. 2022, Oliveira et al. 2023, Oliveira et al. 2023], we decided to use linguistic features based on state-of-the-art tools such as Coh-Metrix. It follows a brief description of the features used. ...
Conference Paper
Full-text available
While Thematic Coherence is a fundamental aspect of essay writing, scoring it is labor-intensive. This issue is often addressed using machine learning algorithms to estimate the score. However, related work is mostly limited to the English language or argumentative essays. Consequently, there is a lack of research on other widely used languages and essay types, such as Brazilian Portuguese and narrative essays. Hence, this paper reports on the findings of a study that aimed to evaluate the value of machine learning algorithms to automatically score the Thematic Coherence of both narratives (n = 400) and argumentative (n = 6567) essays written in Brazilian Portuguese. Expanding on previous studies, this paper evaluated regression models using conventional, feature-based algorithms according to essays’ linguistic features. Overall, we found that Extra Trees was the best performing algorithm, yielding predictions with moderate to strong correlations with human-generated scores. Mainly, those findings expand the literature with evidence on the potential of machine learning to estimate the Thematic Coherence of narrative and argumentative essays, suggest an improved performance for the former type.
... Unlike the Entity Grid model [10], which is a method for evaluating coherence at the local level, in the work of Guinaudeau and Strube (2013) a graphical model called Entity Graph [12] was proposed to measure text coherence at the global level. This bipartite graph allows relating non-adjacent sentences of a text. ...
... They also created 2 generative models that produce coherent text, one is based on SEQ2SEQ and the other is a Markovian model. These models capture the latent discourse dependencies of a given text [14] Based on the foundations of the Entity Graph model [12], a semantic similarity graph model was proposed in the work of Putra and Tokunaga (2017) to address coherence from a cohesion perspective [15]. They argue that the coherence of a text is built by the cohesion between its sentences. ...
... entity correlations among sentences can be used to create coherence patterns for text. Also, it has been widely recognized that entity graphs (or subgraphs) are a powerful form to characterize the coherent structure in natural language [14]. Compared with the traditional entity-grid method [6] which is restricted to capturing coherent transitions between adjacent sentences, the KG-based entity graphs can easily span the entire text and capture the semantic correlations between different sentences. ...
... However, these methods still require considerable experience or domain expertise to define or extract features. Other related approaches include global graph model [14] which projects entities into a global graph and HMM system [26] in which the coherence between adjacent sentences is modeled by a hidden Markov framework and captured by the transition rules of topics. ...
Preprint
As a natural language generation task, it is challenging to generate informative and coherent review text. In order to enhance the informativeness of the generated text, existing solutions typically learn to copy entities or triples from knowledge graphs (KGs). However, they lack overall consideration to select and arrange the incorporated knowledge, which tends to cause text incoherence. To address the above issue, we focus on improving entity-centric coherence of the generated reviews by leveraging the semantic structure of KGs. In this paper, we propose a novel Coherence Enhanced Text Planning model (CETP) based on knowledge graphs (KGs) to improve both global and local coherence for review generation. The proposed model learns a two-level text plan for generating a document: (1) the document plan is modeled as a sequence of sentence plans in order, and (2) the sentence plan is modeled as an entity-based subgraph from KG. Local coherence can be naturally enforced by KG subgraphs through intra-sentence correlations between entities. For global coherence, we design a hierarchical self-attentive architecture with both subgraph- and node-level attention to enhance the correlations between subgraphs. To our knowledge, we are the first to utilize a KG-based text planning model to enhance text coherence for review generation. Extensive experiments on three datasets confirm the effectiveness of our model on improving the content coherence of generated texts.
... In [89], authors use a bipartite graph to model the text as a graph of sentences. They presum that sentences in a document are represented by one set of nodes, while entities are represented by the other set. ...
Article
Full-text available
Creating systems that can interpret and manage the ambiguity and subjectivity of the representation and retrieval of information is one of the issues facing information systems researchers. With the emergence of social networks, improvement methods have been developed in both traditional and social information research while taking into account the specificity of the information. The most main features of language that directly impact the results of information systems are ambiguity and uncertainty. We show through this article several approaches have been applied in order to take into account the ambiguity of language, especially in social networks. We thus notice a tendency to apply a variety of aggregation tools in order to overcome the weaknesses of social information retrieval systems. In what follows, we will give an overview on other levels of aggregation allowing to solve certain problems of social information analysis such as credibility evaluation, profile categorization, opinion mining, influencer detection, etc. Then, we held a discussion on the ability of uncertainty theory to consider the different degrees of feature importance as well as the heterogeneity of information resources.
... The first is a collection of aviation accident reports written by officials from the National Transportation Safety Board and the second contains Associated Press articles from the North American News Corpus on the topic of earthquakes. [12], HMM+Content [12], Conference+Syntax [1], and Graph [9]. ...
Preprint
In this paper, we propose a novel deep coherence model (DCM) using a convolutional neural network architecture to capture the text coherence. The text coherence problem is investigated with a new perspective of learning sentence distributional representation and text coherence modeling simultaneously. In particular, the model captures the interactions between sentences by computing the similarities of their distributional representations. Further, it can be easily trained in an end-to-end fashion. The proposed model is evaluated on a standard Sentence Ordering task. The experimental results demonstrate its effectiveness and promise in coherence assessment showing a significant improvement over the state-of-the-art by a wide margin.
... Concretely, RC and LC [46] measure the number and proportion of words that serve the connecting role. EntityGraph [47] evaluates text coherence through graphs. LexicalChain [48] measures the overlap of lexical chains between the hypothesis and reference texts. ...
Preprint
Full-text available
Offering rich contexts to Large Language Models (LLMs) has shown to boost the performance in various tasks, but the resulting longer prompt would increase the computational cost and might exceed the input limit of LLMs. Recently, some prompt compression methods have been suggested to shorten the length of prompts by using language models to generate shorter prompts or by developing computational models to select important parts of original prompt. The generative compression methods would suffer from issues like hallucination, while the selective compression methods have not involved linguistic rules and overlook the global structure of prompt. To this end, we propose a novel selective compression method called PartPrompt. It first obtains a parse tree for each sentence based on linguistic rules, and calculates local information entropy for each node in a parse tree. These local parse trees are then organized into a global tree according to the hierarchical structure such as the dependency of sentences, paragraphs, and sections. After that, the root-ward propagation and leaf-ward propagation are proposed to adjust node values over the global tree. Finally, a recursive algorithm is developed to prune the global tree based on the adjusted node values. The experiments show that PartPrompt receives the state-of-the-art performance across various datasets, metrics, compression ratios, and target LLMs for inference. The in-depth ablation studies confirm the effectiveness of designs in PartPrompt, and other additional experiments also demonstrate its superiority in terms of the coherence of compressed prompts and in the extreme long prompt scenario.
... There are limitations to grid-based approaches, despite their advantages, including data sparsity, dependence on specific domains, and computational complexity [72]. It is suggested to model local coherence using a graph-based approach for representing entities, followed by applying centrality measures to nodes within the graph: [73] provides an unsupervised, computationally efficient method of modeling local coherence based on graphs. Entity grids are used as the incidence matrix of bipartite graphs capturing text structure. ...
Article
Full-text available
Automatic text summarization is the process of shortening a large document into a summary text that preserves the main concepts and key points of the original document. Due to the wide applications of text summarization, many studies have been conducted on it, but evaluating the quality of generated summaries poses significant challenges. Selecting the appropriate evaluation metrics to capture various aspects of summarization quality, including content, structure, coherence, readability, novelty, and semantic relevance, plays a crucial role in text summarization application. To address this challenge, the main focus of this study is on gathering and investigating a comprehensive set of evaluation metrics. Analysis of various metrics can enhance the understanding of the evaluation method and leads to select appropriate evaluation text summarization systems in the future. After a short review of various automatic text summarization methods, we thoroughly analyze 42 prominent metrics, categorizing them into six distinct categories to provide insights into their strengths, limitations, and applicability.
... Coherence modeling has been studied in various tasks in literature, including the most representative two: Sentence Ordering in NLP and Visual Storytelling in multimedia understanding. Sentence ordering is representative coherence modeling task in the area of NLP, with early-stage methods mostly built based on domain knowledge and language-based features [4,8,19,27,39]. For example, these methods use vectors of linguistic features in probabilistic transition models. In recent years, with the development of deep learning and NLP technologies, more advanced methods apply an encoder-decoder framework and retrieve the final order using pointer networks [14,26,43,53,58]. ...
Preprint
Cross-modal coherence modeling is essential for intelligent systems to help them organize and structure information, thereby understanding and creating content of the physical world coherently like human-beings. Previous work on cross-modal coherence modeling attempted to leverage the order information from another modality to assist the coherence recovering of the target modality. Despite of the effectiveness, labeled associated coherency information is not always available and might be costly to acquire, making the cross-modal guidance hard to leverage. To tackle this challenge, this paper explores a new way to take advantage of cross-modal guidance without gold labels on coherency, and proposes the Weak Cross-Modal Guided Ordering (WeGO) model. More specifically, it leverages high-confidence predicted pairwise order in one modality as reference information to guide the coherence modeling in another. An iterative learning paradigm is further designed to jointly optimize the coherence modeling in two modalities with selected guidance from each other. The iterative cross-modal boosting also functions in inference to further enhance coherence prediction in each modality. Experimental results on two public datasets have demonstrated that the proposed method outperforms existing methods for cross-modal coherence modeling tasks. Major technical modules have been evaluated effective through ablation studies. Codes are available at: \url{https://github.com/scvready123/IterWeGO}.
... browncoherence Entity Graph (EGr). (Guinaudeau & Strube, 2013) Models a text as a graph of sentences with edges connecting sentences that have at least one noun in common. Following W. Zhao, Strube, and Eger (2023), averaged adjacency matrix is reported as a proxy for cohesion. ...
Article
Extractive summaries are usually presented as lists of sentences with no expected cohesion between them and with plenty of redundant information if not accounted for. In this paper, we investigate the trade-offs incurred when aiming to control for inter-sentential cohesion and redundancy in extracted summaries, and their impact on their informativeness. As case study, we focus on the summarization of long, highly redundant documents and consider two optimization scenarios, reward-guided and with no supervision. In the reward-guided scenario, we compare systems that control for redundancy and cohesiveness during sentence scoring. In the unsupervised scenario, we introduce two systems that aim to control all three properties --informativeness, redundancy, and cohesiveness-- in a principled way. Both systems implement a psycholinguistic theory that simulates how humans keep track of relevant content units and how cohesiveness and non-redundancy constraints are applied in short-term memory during reading. Extensive automatic and human evaluations reveal that systems optimizing for --among other properties-- cohesiveness are capable of better organizing content in summaries compared to systems that optimize only for redundancy, while maintaining comparable informativeness. We find that the proposed unsupervised systems manage to extract highly cohesive summaries across varying levels of document redundancy, although sacrificing informativeness in the process. Finally, we lay evidence as to how simulated cognitive processes impact the trade-off between the analysed summary properties.
... These methods provide insights into how information is connected at the local level but often fall short of capturing global coherence. With recent advancements in neural models such as the Transformer (Vaswani et al. 2017), efforts have been made to improve these methods using deep network architectures (Guinaudeau and Strube 2013;Tien Nguyen and Joty 2017;Adewoyin, Dutta, and He 2022), integrating various lexical information to enhance performance. Previous studies have revealed that Transformer language models struggle to effectively capture coherence structures (Deng, Kuleshov, and Rush 2022). ...
Article
Measuring the coherence of text is a vital aspect of evaluating the quality of written content. Recent advancements in neural coherence modeling have demonstrated their efficacy in capturing entity coreference and discourse relations, thereby enhancing coherence evaluation. However, many existing methods heavily depend on static embeddings or focus narrowly on nearby context, constraining their capacity to measure the overarching coherence of long texts. In this paper, we posit that coherent texts inherently manifest a sequential and cohesive interplay among sentences, effectively conveying the central theme, purpose, or standpoint. To explore this abstract relationship, we introduce the "BB Score," a novel reference-free metric grounded in Brownian bridge theory for assessing text coherence. Our findings showcase that when synergized with a simple additional classification component, this metric attains a performance level comparable to state-of-the-art techniques on standard artificial discrimination tasks. We also establish in downstream tasks that this metric effectively differentiates between human-written documents and text generated by large language models within specific domains. Furthermore, we illustrate the efficacy of this approach in detecting written styles attributed to various large language models, underscoring its potential for generalizability. In summary, we present a novel Brownian bridge coherence metric capable of measuring both local and global text coherence, while circumventing the need for end-to-end model training. This flexibility allows for its application in various downstream tasks.
... Traditional approaches to sentence ordering used hand-engineered features to capture document coherence (Barzilay and Lapata, 2005;Elsner et al., 2007;Barzilay and Lapata, 2008;Elsner and Charniak, 2011;Mesgar and Strube, 2016), e.g. using an entity matrix Lapata, 2005, 2008) or graph (Guinaudeau and Strube, 2013) to represent entity transitions across sentences, and maximising transition probabilities between adjacent sentences. ...
... This included heuristics with Markov models (Barzilay and Lee, 2004;Bollegala et al., 2005;Ji and Pulman, 2006), K-means clustering (Ji and Nie, 2008;Zhang, 2011), support vector machines (Bollegala et al., 2006;Nahnsen, 2009;Peng et al., 2009;Yanase et al., 2015) and others like latent semantic analysis (Zhang et al., 2010) and conditional random fields (Gella and Duong Thanh, 2012). We also note the representation of sentence ordering as a graph in many past works as well (Elsner and Charniak, 2011;Li et al., 2011;Guinaudeau and Strube, 2013). However, with advances in neural modeling, other approaches have developed. ...
... In 2008, a model for assessing text coherence called Entity Grid was proposed [4]. The main idea of this work is to assume that the distribution of key text entities (noun groups present in sentences) follows a certain pattern. ...
Article
The subject of the article is to determine the degree of scientific and technical text connectedness using statistical calculations. The aim of the scientific investigation is to study the possibilities of using the coherence of fluctuations in the relative frequencies of keywords in paragraphs to determine the lexical coherence and thematic unity of scientific and technical texts. The task is to develop a method for determining the thematic unity of a text at the set of paragraphs level; to develop a method for determining the coherence of a text at the set of paragraphs level; and to test the developed methods on a collection of documents. The methods used are statistical analysis and computational experiment methods. The following results were obtained. The study has shown that it is advisable to cluster paragraphs as points in the keyword space to determine the degree of scientific and technical text coherence at the level of paragraphs. This opens up the possibility of calculating the degree of thematic unity within the clusters and in the entire text. The degree of text fragments and the whole text coherence is determined by analyzing the sequence of paragraph numbers in the clusters. This makes it possible to formally determine the quality of the material presented in a scientific and technical article or in a textbook. Conclusions. The scientific novelty of the study is as follows: there was refined on the method for determination of the connectedness degree (coherence and thematic unity) of scientific and technical texts at the level of paragraphs by implementation of paragraphs clustering in the keywords space, using the calculation of thematic unity degree inside the clusters and in the overall text, as well as through analysis of paragraphs numbers sequence in clusters in order to determine the degree of text fragments and the overall text coherence. The methods are language-independent, based on clear hypotheses, and complement each other. The methods have an adjusting element that can be used to adapt it to different thematic and stylistic areas. It has been experimentally proved that the proposed methods for the determination of scientific and technical text connectedness are efficient and can provide the framework for information technology of content analysis of scientific and technical texts. The proposed methods do not use WEB resources for syntactic and semantic analysis, providing the possibility to use them autonomously.
... ii) Sentence ordering involves positional reasoning on textual contents, which aims to order sentences into a coherent narration. Early works solved the task by modeling local coherence using language-based features [20,2,9,14]. Recent works leverage deep learning to encode sentences and retrieve the final order using pointer networks, which compare sentences in a pairwise fashion [40,11,25,6,43,42]. ...
Preprint
Full-text available
Positional reasoning is the process of ordering unsorted parts contained in a set into a consistent structure. We present Positional Diffusion, a plug-and-play graph formulation with Diffusion Probabilistic Models to address positional reasoning. We use the forward process to map elements' positions in a set to random positions in a continuous space. Positional Diffusion learns to reverse the noising process and recover the original positions through an Attention-based Graph Neural Network. We conduct extensive experiments with benchmark datasets including two puzzle datasets, three sentence ordering datasets, and one visual storytelling dataset, demonstrating that our method outperforms long-lasting research on puzzle solving with up to +18% compared to the second-best deep learning method, and performs on par against the state-of-the-art methods on sentence ordering and visual storytelling. Our work highlights the suitability of diffusion models for ordering problems and proposes a novel formulation and method for solving various ordering tasks. Project website at https://iit-pavis.github.io/Positional_Diffusion/
... For the last linguistic feature set investigated, we look at discourse in the form of averages of unique and non-unique entity presence that can affect working memory load as well as local coherence distance measures which captures distribution and transitions of entities in a passage (Barzilay and Lapata, 2008;Guinaudeau and Strube, 2013). Feng et al. (2009) previously applied these cognitivelymotivated features for assessing reading difficulty in the case of adults with intellectual disabilities. ...
Preprint
Full-text available
Powerful language models such as GPT-2 have shown promising results in tasks such as narrative generation which can be useful in an educational setup. These models, however, should be consistent with the linguistic properties of triggers used. For example, if the reading level of an input text prompt is appropriate for low-leveled learners (ex. A2 in the CEFR), then the generated continuation should also assume this particular level. Thus, we propose the task of uniform complexity for text generation which serves as a call to make existing language generators uniformly complex with respect to prompts used. Our study surveyed over 160 linguistic properties for evaluating text complexity and found out that both humans and GPT-2 models struggle in preserving the complexity of prompts in a narrative generation setting.
... One of the subtasks in coherence modeling, called sentence ordering, refers to organizing shuffled sentences into an order that maximizes coherence (Barzilay and Lapata, 2008). Several downstream applications benefit from this task to assemble sound and easy-to-understand texts, such as extraction-based multi-document summarization (Barzilay and El-hadad, 2002;Galanis et al., 2012;Nallapati et al., 2017;Logeswaran et al., 2018), natural language generation (Reiter and Dale, 1997), retrieval-based question answering , concept-to-text generation (Konstas and Lapata, 2012), storytelling (Fan et al., 2019;Hu et al., 2020;Zhu et al., 2020), opinion generation (Yanase et al., 2015), conversational analysis (Zeng et al., 2018), image captioning (Anderson et al., 2018), recipe generation (Chandu et al., 2019), and discourse coherence (Elsner et al., 2007;Barzilay and Lapata, 2008;Guinaudeau and Strube, 2013). ...
Preprint
Full-text available
Text coherence is a fundamental problem in natural language generation and understanding. Organizing sentences into an order that maximizes coherence is known as sentence ordering. This paper is proposing a new approach based on the graph neural network approach to encode a set of sentences and learn orderings of short stories. We propose a new method for constructing sentence-entity graphs of short stories to create the edges between sentences and reduce noise in our graph by replacing the pronouns with their referring entities. We improve the sentence ordering by introducing an aggregation method based on majority voting of state-of-the-art methods and our proposed one. Our approach employs a BERT-based model to learn semantic representations of the sentences. The results demonstrate that the proposed method significantly outperforms existing baselines on a corpus of short stories with a new state-of-the-art performance in terms of Perfect Match Ratio (PMR) and Kendall's Tau (Tau) metrics. More precisely, our method increases PMR and Tau criteria by more than 5% and 4.3%, respectively. These outcomes highlight the benefit of forming the edges between sentences based on their cosine similarity. We also observe that replacing pronouns with their referring entities effectively encodes sentences in sentence-entity graphs.
... Entity coherence Various local entity coherence metrics have been suggested in literature, such as distance-based clustering and linkage coefficients (Lioma et al., 2016) and local entity coherence (Barzilay & Lapata, 2008;Mesgar & Strube, 2014;Guinaudeau & Strube, 2013). However, current LMs present high local coherence when compared with human-written stories, giving the impression that coherence has been achieved. ...
Preprint
Full-text available
Large pre-trained language models (LMs) have demonstrated impressive capabilities in generating long, fluent text; however, there is little to no analysis on their ability to maintain entity coherence and consistency. In this work, we focus on the end task of narrative generation and systematically analyse the long-range entity coherence and consistency in generated stories. First, we propose a set of automatic metrics for measuring model performance in terms of entity usage. Given these metrics, we quantify the limitations of current LMs. Next, we propose augmenting a pre-trained LM with a dynamic entity memory in an end-to-end manner by using an auxiliary entity-related loss for guiding the reads and writes to the memory. We demonstrate that the dynamic entity memory increases entity coherence according to both automatic and human judgment and helps preserving entity-related information especially in settings with a limited context window. Finally, we also validate that our automatic metrics are correlated with human ratings and serve as a good indicator of the quality of generated stories.
... Early studies mainly focused on exploring humandesigned features for sentence ordering (Lapata, 2003;Barzilay and Lee, 2004;Lapata, 2005, 2008;Elsner and Charniak, 2011;Guinaudeau and Strube, 2013). Recently, neural network based sentence ordering models have become dominant , consisting of the following two kinds of models: ...
... Graph. e graph approach [27] extends the entity grid model to a bipartite graph to represent the text and computes the local coherence of the entity transition in the bipartite graph. ...
Article
Full-text available
Discourse coherence is strongly associated with text quality, making it important to natural language generation and understanding. However, existing coherence models focus on measuring individual aspects of coherence, such as lexical overlap, entity centralization, rhetorical structure, etc., lacking measurement of the semantics of text. In this paper, we propose a discourse coherence analysis method combining sentence embedding and the dimension grid, we obtain sentence-level vector representation by deep learning, and we introduce a coherence model that captures the fine-grained semantic transitions in text. Our work is based on the hypothesis that each dimension in the embedding vector is exactly assigned a stated certainty and specific semantic. We take every dimension as an equal grid and compute its transition probabilities. The document feature vector is also enriched to model the coherence. Finally, the experimental results demonstrate that our method achieves excellent performance on two coherence-related tasks.
... Early studies mainly focused on exploring humandesigned features for sentence ordering (Lapata, 2003;Barzilay and Lee, 2004;Lapata, 2005, 2008;Elsner and Charniak, 2011;Guinaudeau and Strube, 2013). Recently, neural network based sentence ordering models have become dominant , consisting of the following two kinds of models: ...
Preprint
Full-text available
Dominant sentence ordering models can be classified into pairwise ordering models and set-to-sequence models. However, there is little attempt to combine these two types of models, which inituitively possess complementary advantages. In this paper, we propose a novel sentence ordering framework which introduces two classifiers to make better use of pairwise orderings for graph-based sentence ordering. Specially, given an initial sentence-entity graph, we first introduce a graph-based classifier to predict pairwise orderings between linked sentences. Then, in an iterative manner, based on the graph updated by previously predicted high-confident pairwise orderings, another classifier is used to predict the remaining uncertain pairwise orderings. At last, we adapt a GRN-based sentence ordering model on the basis of final graph. Experiments on five commonly-used datasets demonstrate the effectiveness and generality of our model. Particularly, when equipped with BERT and FHDecoder, our model achieves state-of-the-art performance.
... To solve this issue, other methodologies have been proposed based on discourse theory, in particular the centering theory. One such approach is entity grids (Barzilay & Lapata, 2008) and entity graphs (Guinaudeau & Strube, 2013) that treat coherence as to how are entities take different roles between sentences and how are they connected in the text. TRUNAJOD implements all these models, and thus TRUNAJOD can compute coherence based on sentence similarities using word vectors. ...
Article
Full-text available
We present TRUNAJOD, a text complexity analysis tool that includes a wide variety of linguistics measurements that can be extracted from texts as an approximation for readability, coherence, and cohesion. The features that TRUNAJOD can extract from the text are based on the literature and can be separated into the following categories: discourse markers, emotions, entity grid-based measurements, givenness, lexical-semantic norms, semantic measures, surface proxies, etc. In this first version of TRUNAJOD, we mainly support the Spanish language, but several features support any language that has proper natural language processing POS tagging and dependency parsing capabilities. Finally, we show how TRUNAJOD could be used in applied research
... The output of the model interprets a prediction whether the input text is coherent or not. This idea of the investigation of the syntactic role changes was used in the further connected researches [7,8]. In contrast to the Entity Grid approach, these methods are based on the representation of a text as a graph structure. ...
Article
Full-text available
Coherence evaluation of texts falls into a category of natural language processing tasks. The evaluation of texts’ coherence implies the estimation of their semantic and logical integrity; such a feature of a text can be utilized during the solving of multidisciplinary tasks (SEO analysis, medicine area, detection of fake texts, etc.). In this paper, different state-of-the-art coherence evaluation methods based on machine learning models have been analyzed. The investigation of the effectiveness of different methods for the coherence estimation of Polish texts has been performed. The impact of text’s features on the output coherence value has been analyzed using different approaches of a semantic similarity graph. Two neural networks based on LSTM layers and a pre-trained BERT model correspondingly have been designed and trained for the coherence estimation of input texts. The results obtained may indicate that both lexical and semantic components should be taken into account during the coherence evaluation of Polish documents; moreover, it is advisable to analyze corresponding documents in a sentence-by-sentence manner taking into account word order. According to the retrieved accuracy of the proposed neural networks, it can be concluded that suggested models may be used in order to solve typical coherence estimation tasks for a Polish corpus.
... Putra and Tokunaga (2017) design a unsupervised graph method from the perspective of sentence similarities and coherence inside a text. They claim superiority on the supervised Entity Grid (Barzilay and Lapata, 2008) and the unsupervised Entity Graph (Guinaudeau and Strube, 2013). But it is unlearnable and needs to build all graphs with potential positions of the taken-out sentence. ...
Preprint
Sentence insertion is a delicate but fundamental NLP problem. Current approaches in sentence ordering, text coherence, and question answering (QA) are neither suitable nor good at solving it. In this paper, We propose InsertGNN, a simple yet effective model that represents the problem as a graph and adopts the graph Neural Network (GNN) to learn the connection between sentences. It is also supervised by both the local and global information that the local interactions of neighboring sentences can be considered. To the best of our knowledge, this is the first recorded attempt to apply a supervised graph-structured model in sentence insertion. We evaluate our method in our newly collected TOEFL dataset and further verify its effectiveness on the larger arXivdataset using cross-domain learning. The experiments show that InsertGNN outperforms the unsupervised text coherence method, the topological sentence ordering approach, and the QA architecture. Specifically, It achieves an accuracy of 70%, rivaling the average human test scores.
... To distinguish coherence from incoherence, n-gram sub-sequences of transitions per term in the discourse role matrix are used. In Guinaudeau and Strube (2013), Guineaudeau and Strube modeled the text into a graph of sentences by using a bipartite graph. They suppose that one set of nodes represents entities and the other set represents sentences of a document. ...
Article
Full-text available
Nowadays, social medias are very popular among their users. One of the most well-known social networks is Twitter. It is a micro-blog that enables its users to send short messages called tweets. A tweet is a 280 characters long message that is rarely self-content. Hence, additional information is necessary to allow better readability of the tweet. This new task has attracted a great deal of attention recently. Given a tweet, the aim of tweet contextualization is to produce an informative paragraph, called a context, from a set of documents in response to topics treated by the tweet. Furthermore of being informative, a summary should be coherent, i.e., well-written to be readable and grammatically compact. Hence, coherence is an essential characteristic in order to produce comprehensible texts. In this paper, we propose a new approach of tweet contextualization based on graphs by combining sentence extraction, sentence aggregation and sentence reordering to enhance informativeness and readability in order to build a relevant and coherent context. The main idea of our proposed method is to select relevant, informative coherent and semantically related sentences from a document that best describes themes expressed by the tweet, and aggregate relevant phrases in the same graph to filter more informative ones. We proposed a novel algorithm called CSA algorithm to achieve our aim and to construct a concise extract. We also proposed to invest in a reordering phase to improve the coherence of the obtained context.
... Discourse analysis, which takes a more holistic view of conversations and brand texts (Jørgensen & Phillips, 2002), can use artificial intelligence techniques for textual analysis and can analyze beyond single words to assess words in context (Pandey & Pandey, 2019). The NPL approach to discourse analysis is built from the Rhetorical Structure Theory (RST) (Carlson, Okurowski, & Marcu, 2002;Subba & Di Eugenio, 2009) and homes in on coherence models using entity grids (Barzilay & Lapata, 2008;Elsner & Charniak, 2011b;Guinaudeau & Strube, 2013), discourse relation-based model (Lin, Ng, & Kan, 2011;Pitler & Nenkova, 2008), neural coherence models (Li & Jurafsky, 2016;Mesgar & Strube, 2018;Mohiuddin, Joty, & Nguyen, 2018), and coherence models for conversations (Elsner & Charniak, 2011a;Mohiuddin et al., 2018). Another valuable methodology for studying brand legitimacy can be the Q methodology. ...
Article
In a hyperconnected world, branding is moving away from single, organization-driven ownership to shared ownership. Motivated by the continuous advancements in digital technologies for B2B brands, this research uses a rhetorical and discursive approach to build a framework of B2B brands legitimacy. This conceptual framework integrates multiple levels of social discourse, from brand texts to stories and narratives, to illustrate how the connections between these and brand legitimacy are made through language. We describe how B2B brands can use the rhetorical elements of logos, pathos, and ethos to achieve an advantageous position within their field's discourse. We identify B2B brand reputation, awareness, and credibility among the benefits of having a brand legitimacy within a network of actors and provide propositions and empirical guides to substantiate the proposed framework.
... This model considers the distribution of entities over sentences. Guinaudeau and Strube (2013) convert the supervised entity grid into an unsupervised graph-based model. Li and Jurafsky (2017) propose a neural model which uses cliques, sets of adjacent sentences, to discriminate the difference of sentences extracted from original articles and randomly permutated ones. ...
Preprint
Modeling the structure of coherent texts is a key NLP problem. The task of coherently organizing a given set of sentences has been commonly used to build and evaluate models that understand such structure. We propose an end-to-end unsupervised deep learning approach based on the set-to-sequence framework to address this problem. Our model strongly outperforms prior methods in the order discrimination task and a novel task of ordering abstracts from scientific articles. Furthermore, our work shows that useful text representations can be obtained by learning to order sentences. Visualizing the learned sentence representations shows that the model captures high-level logical structure in paragraphs. Our representations perform comparably to state-of-the-art pre-training methods on sentence similarity and paraphrase detection tasks.
Article
Full-text available
Text readability assessment aims to automatically evaluate the degree of reading difficulty of a given text for a specific group of readers. Most of the previous studies considered it as a classification task and explored a wide range of linguistic features to express the readability of a text from different aspects, such as semantic-based and syntactic-based features. Intuitively, when the external form of a text becomes more complex, individuals will experience more reading difficulties. Based on this motivation, our research attempts to separate the textual external form from the text and investigate its efficiency in determining readability. Specifically, in this paper, we introduce a new concept, namely textual form complexity, to provide a novel insight into text readability. The main idea is that the readability of a text can be measured by the degree to which it is challenging for readers to overcome the distractions of external textual form and obtain the text’s core semantics. To this end, we propose a set of textual form features to express the complexity of the outer form of a text and characterize its readability. Findings show that the proposed external textual form features can be used as effective evaluation indexes to indicate the readability of text. It brings a new perspective to the existing research and provides a new complement to the existing rich features.
Chapter
This paper presents an approach for evaluating coherence in Chinese middle school student essays, addressing the challenges of time-consuming and inconsistent essay assessment. Previous approaches focused on linguistic features, but coherence, crucial for essay organization, has received less attention. Recent works utilized neural networks, such as CNN, LSTM, and transformers, achieving good performance with labeled data. However, labeling coherence manually is costly and time-consuming. To address this, we propose a method that pretrains RoBERTa with whole word masking (WWM) on a low-resource dataset of middle school essays, followed by finetuning for coherence evaluation. The WWM pretraining is unsupervised and captures general characteristics of the essays, adding little cost to the low-resource setting. Experimental results on Chinese essays demonstrate that this strategy improves coherence evaluation compared to naive finetuning on limited data. We also explore variants of their method, including pseudo labeling and additional neural networks, providing insights into potential performance trade-offs. The contributions of this work include the collection and curation of a substantial dataset, the proposal of a cost-effective pretraining method, and the exploration of alternative approaches for future research.
Article
Argumentation is ubiquitous in everyday discourse, and it is a skill that can be learned. In our society, it is also one that must be learned: education systems all over the world agree on the importance of argumentation skills. However, writing effective argumentation is difficult, and even more so if it has to be expressed in a foreign language. Existing artificial intelligence systems for language learning can help learners: they can provide objective feedback (e.g., concerning grammar and spelling), as well as providing learners with opportunities to identify errors and subsequently improve their texts. Even so, systems aiming at higher discourse-level skills, such as persuasiveness and content organisation, are still limited. In this article, we propose the novel task of sentence reordering for improving the logical flow of argumentative essays. To train such a computational system, we present a new corpus called ICNALE-AS2R, containing essays written by English-as-foreign-language learners from various Asian countries, that have been annotated with argumentative structure and sentence reordering. We also propose a novel method to automatically reorder sentences in imperfect essays, which is based on argumentative structure analysis. Given an input essay and its corresponding argumentative structure, we cast the reordering task as a traversal problem. Our sentence reordering system first determines the pairwise ordering relation between pairs of sentences that are connected by argumentative relations. In the second step, the system traverses the argumentative structure that has been augmented with pairwise ordering information, in order to generate the final output text. Empirical evaluation shows that in the task of reconstructing the final reordered essays in the dataset, our reordering system achieves .926 and .879 in longest common subsequence ratio and Kendall's Tau metrics, respectively. The system is also able to perform the reordering operation selectively, that is, it reorders sentences when necessary and retains the original input order when it is already optimal.
Chapter
Coherence between various enterprise facets is essential for optimal performance. A quantitative expression for enterprise coherence has therefore the opportunity to function as a leading indicator for enterprise performance, but is currently lacking. This research focuses on the quantification of ‘enterprise coherence’, in order to aid enterprise architecture governance and realize more sustainable enterprises. The Enterprise Coherence Index (the EC-index) to measure enterprise coherence is proposed. Design development of the EC-index is guided by a well-established design science methodology. One of the identified components of the EC-index is an enterprise coherence calculation engine. The enterprise coherence calculation engine requires a quantification method in order to calculate coherence. Quantification models from different domains exist. Four different candidate quantification models are selected, and the selection is made plausible through a literature overview based on key search terms. All approaches are based on a graph model. For the domain of the enterprise a bipartite network of ‘direction statements’ versus enterprise decisions is chosen. To aid in developing the EC-index, quantification methods are compared with data from two historical cases. It is shown that some models can already be eliminated based on these cases, and that other methods can be unified. It will be shown that coherence contribution of individual decisions can be expressed as a number, based on their supportiveness of the enterprise’s purpose. This paper aims to contribute to the domain of governance and policy modeling as well as to organizational decision making.KeywordsArchitecture governanceEnterprise architectureEnterprise coherenceEnterprise coherence indexGeneral Enterprise architecting
Article
The category of coreference plays an important role in the creation of texts, as it is based on fundamental principles of text organization. The linearity of a text and the non-linearity of a situation described in the text are closely related to the mention of elements of a situation with another nomination of this object. Therefore, chains of coreferential names appear in the text and, having a binary relation, partly cause the emergence of a secondary nomination, which is an anaphoric relation with the primary nomination. The purpose of this article is to study the category of coreference in the poetic speech of Lesia Ukrainka based on the corpus of the Ukrainian language represented on mova.info. The object of the research is the idiostyle of Lesia Ukrainka’s poetry, and the subject is the functioning of the category of coreference. The categorical essence of coreference is characterized, the types of representatives of real objects, their functional and stylistic varieties and typical objects of representation in a poetic text are determined, and the most frequent structures are considered. Software for working with coreference has been developed and implemented. It has a user-friendly interface, which allows searching, sort, and doing the quantitative processing of the collected information according to the needs of researchers. On the material of 153 texts, 1520 referentially identical pairs are established. The coreferentiality index and the index of coreference coverage of the text proposed in this article helped to quantitatively assess the saturation of the text with the category of coreference. In addition, the classification of coreferential relations is composed: 1) identity; 2) collection gap; 3) part-whole; 4) predicative identity; 5) metonymy; 6) association. A dependency tree of a sentence made it possible to determine the location of coreferential pairs in the sentence, the position relative to each other, the belonging of the segment with the coreferent or the referent to the complicators of the sentence, i.e., a subordinate clause or phrases. This representation also helped to establish the type of syntactic relationship between the coreferent and the referent, the type of syntactic-semantic relationship between them, the presence of referent-dependent words, the intersection of the coreferent and the referent in gender and number. Automatic extraction of coreferential pairs from dependency trees allowed building intersentential coreference.
Article
Automatic generation of long texts containing multiple sentences has many applications in the field of Natural Language Processing (NLP) including question answering, machine translation, and paraphrase generation, etc. However, in terms of readability, the long texts generated by machines are not comparable to those organized by human beings. Through statistics, we observed that human-organized texts generally have a special property: one or more of the words (particularly nouns and pronouns) appeared in one sentence will reappear in the next one in the same or a different form. This repetition of words in consecutive sentences can greatly improve the readability. Based on this observation, we propose CMST, a deep neural network model for generating Coherent Multi-Sentence Texts. CMST explicitly incorporates a training strategy of coherence mechanism to evaluate the repetition of words in consecutive sentences. We evaluate the performance of the CMST on the CNN/Daily Mail dataset. The experimental results show that, compared with the baseline models, CMST not only improves the readability of the generated texts, but achieves higher METEOR and ROUGE values.
Chapter
Text coherence is a fundamental problem in natural language generation and understanding. Organizing sentences into an order that maximizes coherence is known as sentence ordering. This paper is proposing a new approach based on the graph neural network approach to encode a set of sentences and learn orderings of short stories. We propose a new method for constructing sentence-entity graphs of short stories to create the edges between sentences and reduce noise in our graph by replacing the pronouns with their referring entities. We improve the sentence ordering by introducing an aggregation method based on majority voting of state-of-the-art methods and our proposed one. Our approach employs a BERT-based model to learn semantic representations of the sentences. The results demonstrate that the proposed method significantly outperforms existing baselines on a corpus of short stories with a new state-of-the-art performance in terms of Perfect Match Ratio (PMR) and Kendall’s Tau (τ\tau ) metrics. More precisely, our method increases PMR and τ\tau criteria by more than 5% and 4.3%, respectively. These outcomes highlight the benefit of forming the edges between sentences based on their cosine similarity. We also observe that replacing pronouns with their referring entities effectively encodes sentences in sentence-entity graphs.
Article
Automatic evaluation of summary is one of the basic problems of automatic text summarization research. The commonly used approaches evaluate the informativeness, conciseness and coherence of summary separately based on different representations with weak semantics such as n-gram and vector of words. By transforming text into semantic link network, this paper proposes a general framework for automatically evaluating summary from the following aspects: (1) fidelity, the extent of a summary that conveys the core of its source text; (2) conciseness, the extent of non-redundancy within summary and, (3) coherence, the extent of relatedness between representations within summary, which are based on a uniform semantic representation of text. Experiments on the evaluation of the summarization of scientific papers and news show that the proposed framework achieves comparable or higher correlations with human judgments than the popular evaluation models. This research also verifies the role of the semantic link network in representing and analysing texts.
Conference Paper
Full-text available
The main technology used in e-commerce is Internet technology. The advent of the Internet significantly reduced the cost of e-commerce due to the low cost of transmitting information and led to the emergence of its qualitatively new forms, where the key is the interaction with the consumer through the website. Companies doing business over the Internet basically operate as follows. The seller company places on its web server an interface with which the end-user or partner can, for example, form and place an order in the information system of the seller company. After that, the special software of the seller company processes the order.
Article
Full-text available
Existing software systems for automated essay scoring can provide NLP researchers with opportunities to test certain theoretical hypotheses, including some derived from Centering Theory. In this study we employ the Educational Testing Service's e-rater essay scoring system to examine whether local discourse coherence, as defined by a measure of Centering Theory's Rough-Shift transitions, might be a significant contributor to the evaluation of essays. Rough-Shifts within students' paragraphs often occur when topics are short-lived and unconnected, and are therefore indicative of poor topic development. We show that adding the Rough-Shift based metric to the system improves its performance significantly, better approximating human scores and providing the capability of valuable instructional feedback to the student. These results indicate that Rough-Shifts do indeed capture a source of incoherence, one that has not been closely examined in the Centering literature. They not only justify Rough-Shifts as a valid transition type, but they also support the original formulation of Centering as a measure of discourse continuity even in pronominal-free text. Finally, our study design, which used a combination of automated and manual NLP techniques, highlights specific areas of NLP research and development needed for engineering practical applications.
Article
Full-text available
Centering theory is the best-known framework for theorizing about local coherence and salience; however, its claims are articulated in terms of notions which are only partially specified, such as “utterance,” “realization,” or “ranking.” A great deal of research has attempted to arrive at more detailed specifications of these parameters of the theory; as a result, the claims of centering can be instantiated in many different ways. We investigated in a systematic fashion the effect on the theory's claims of these different ways of setting the parameters. Doing this required, first of all, clarifying what the theory's claims are (one of our conclusions being that what has become known as “Constraint 1” is actually a central claim of the theory). Secondly, we had to clearly identify these parametric aspects: For example, we argue that the notion of “pronoun” used in Rule 1 should be considered a parameter. Thirdly, we had to find appropriate methods for evaluating these claims. We found that while the theory's main claim about salience and pronominalization, Rule 1—a preference for pronominalizing the backward-looking center (CB)—is verified with most instantiations, Constraint 1–a claim about (entity) coherence and CB uniqueness—is much more instantiation-dependent: It is not verified if the parameters are instantiated according to very mainstream views (“vanilla instantiation”), it holds only if indirect realization is allowed, and is violated by between 20% and 25% of utterances in our corpus even with the most favorable instantiations. We also found a trade-off between Rule 1, on the one hand, and Constraint 1 and Rule 2, on the other: Setting the parameters to minimize the violations of local coherence leads to increased violations of salience, and vice versa. Our results suggest that “entity” coherence—continuous reference to the same entities—must be supplemented at least by an account of relational coherence.
Article
Full-text available
In this article we discuss several metrics of coherence defined using centering theory and investigate the usefulness of such metrics for information ordering in automatic text generation. We estimate empirically which is the most promising metric and how useful this metric is using a general methodology applied on several corpora. Our main result is that the simplest metric (which relies exclusively on NOCB transitions) sets a robust baseline that cannot be outperformed by other metrics which make use of additional centering-based features. This baseline can be used for the development of both text-to-text and concept-to-text generation systems.
Article
Full-text available
This article describes an implemented system which uses centering theory for planning of coherent texts and choice of referring expressions. We argue that text and sentence planning need to be driven in part by the goal of maintaining referential continuity and thereby facilitating pronoun resolution: Obtaining a favorable ordering of clauses, and of arguments within clauses, is likely to increase opportunities for nonambiguous pronoun use. Centering theory provides the basis for such an integrated approach. Generating coherent texts according to centering theory is treated as a constraint satisfaction problem. The well-known Rule 2 of centering theory is reformulated in terms of a set of constraints—cohesion, salience, cheapness, and continuity—and we show sample outputs obtained under a particular weighting of these constraints. This framework facilitates detailed research into evaluation metrics and will therefore provide a productive research tool in addition to the immediate practical benefit of improving the fluency and readability of generated texts. The technique is generally applicable to natural language generation systems, which perform hierarchical text structuring based on a theory of coherence relations with certain additional assumptions.
Conference Paper
The CoNLL-2012 shared task involved predicting coreference in three languages -- English, Chinese and Arabic -- using OntoNotes data. It was a follow-on to the English-only task organized in 2011. Until the creation of the OntoNotes corpus, resources in this subfield of language processing have tended to be limited to noun phrase coreference, often on a restricted set of entities, such as ACE entities. OntoNotes provides a large-scale corpus of general anaphoric coreference not restricted to noun phrases or to a specified set of entity types and covering multiple languages. OntoNotes also provides additional layers of integrated annotation, capturing additional shallow semantic structure. This paper briefly describes the OntoNotes annotation (coreference and other layers) and then describes the parameters of the shared task including the format, pre-processing information, evaluation criteria, and presents and discusses the results achieved by the participating systems. Being a task that has a complex evaluation history, and multiple evalation conditions, it has, in the past, been difficult to judge the improvement in new algorithms over previously reported results. Having a standard test set and evaluation parameters, all based on a resource that provides multiple integrated annotation layers (parses, semantic roles, word senses, named entities and coreference) that could support joint models, should help to energize ongoing research in the task of entity and event coreference.
Conference Paper
This paper considers the problem of automatic assessment of local coherence. We present a novel entity-based representation of discourse which is inspired by Centering Theory and can be computed automatically from raw text. We view coherence assessment as a ranking learning problem and show that the proposed discourse representation supports the effective learning of a ranking function. Our experiments demonstrate that the induced model achieves significantly higher accuracy than a state-of-the-art coherence model.
Conference Paper
In this paper we develop a story generator that leverages knowledge inherent in corpora without requiring extensive manual involvement. A key feature in our approach is the reliance on a story planner which we acquire automatically by recording events, their participants, and their precedence relationships in a training corpus. Contrary to previous work our system does not follow a generate-and-rank architecture. Instead, we employ evolutionary search techniques to explore the space of possible stories which we argue are well suited to the story generation task. Experiments on generating simple children's stories show that our system outperforms previous data-driven approaches.
Conference Paper
We present a novel model to represent and assess the discourse coherence of text. Our model assumes that coherent text implicitly favors certain types of discourse relation transitions. We implement this model and apply it towards the text ordering ranking task, which aims to discern an original text from a permuted ordering of its sentences. The experimental results demonstrate that our model is able to significantly outperform the state-of-the-art coherence model by Barzilay and Lapata (2005), reducing the error rate of the previous approach by an average of 29% over three data sets against human upper bounds. We further show that our model is synergistic with the previous approach, demonstrating an error reduction of 73% when the features from both models are combined for the task.