No file available
To read the file of this research,
you can request a copy directly from the authors.
On the Feasibility of Automated Detection of Allusive Text Reuse
Preprints and early-stage research may not have been peer reviewed yet.
Abstract
The detection of allusive text reuse is particularly challenging due to the sparse evidence on which allusive references rely---commonly based on none or very few shared words. Arguably, lexical semantics can be resorted to since uncovering semantic relations between words has the potential to increase the support underlying the allusion and alleviate the lexical sparsity. A further obstacle is the lack of evaluation benchmark corpora, largely due to the highly interpretative character of the annotation process. In the present paper, we aim to elucidate the feasibility of automated allusion detection. We approach the matter from an Information Retrieval perspective in which referencing texts act as queries and referenced texts as relevant documents to be retrieved, and estimate the difficulty of benchmark corpus compilation by a novel inter-annotator agreement study on query segmentation. Furthermore, we investigate to what extent the integration of lexical semantic information derived from distributional models and ontologies can aid retrieving cases of allusive reuse. The results show that (i) despite low agreement scores, using manual queries considerably improves retrieval performance with respect to a windowing approach, and that (ii) retrieval performance can be moderately boosted with distributional semantics.
ResearchGate has not been able to resolve any citations for this publication.
This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder.
We show how to consider similarity between features for calculation of similarity of objects in the Vector Space Model (VSM) for machine learning algorithms and other classes of methods that involve similarity between objects. Unlike LSA, we assume that similarity between features is known (say, from a synonym dictionary) and does not need to be learned from the data.We call the proposed similarity measure soft similarity.
Similarity between features is common, for example, in natural language processing: words, n-grams, or syntactic n-grams can be somewhat different (which makes them different features) but still have much in common: for example, words “play” and “game” are different but related. When there is no similarity between features then our soft similarity measure is equal to the standard similarity. For this, we generalize the well-known cosine similarity measure in VSM by introducing what we call “soft cosine measure”. We propose various formulas for exact or approximate calculation of the soft cosine measure. For example, in one of them we consider for VSM a new feature space consisting of pairs of the original features weighted by their similarity. Again, for features that bear no similarity to each other, our formulas reduce to the standard cosine measure. Our experiments show that our soft cosine measure provides better performance in our case study: entrance exams question answering task at CLEF. In these experiments, we use syntactic n-grams as features and Levenshtein distance as the similarity between n-grams, measured either in characters or in elements of n-grams.
Tesserae is a web-based tool for automatically detecting allusions in Latin poetry. Although still in the start-up phase,
it already is capable of identifying significant numbers of known allusions, as well as similar numbers of allusions previously
unnoticed by scholars. In this article, we use the tool to examine allusions to Vergil’s Aeneid in the first book of Lucan’s Civil War. Approximately 3,000 linguistic parallels returned by the program were compared with a list of known allusions drawn from
commentaries. Each was examined individually and graded for its literary significance, in order to benchmark the program’s
performance. All allusions from the program and commentaries were then pooled in order to examine broad patterns in Lucan’s
allusive techniques which were largely unapproachable without digital methods. Although Lucan draws relatively constantly
from Vergil’s generic language in order to maintain the epic idiom, this baseline is punctuated by clusters of pointed allusions,
in which Lucan frequently subverts Vergil’s original meaning. These clusters not only attend the most significant characters
and events but also play a role in structuring scene transitions. Work is under way to incorporate the ability to match on
word meaning, phrase context, as well as metrical and phonological features into future versions of the program.
The TREC-8 Question Answering track was the first large-scale evaluation of domain-independent question answering systems. This paper summarizes the results of the track by giving a brief overview of the different approaches taken to solve the problem. The most accurate systems found a correct response for more than 2/3 of the questions. Relatively simple bag-of-words approaches were adequate for finding answers when responses could be as long as a paragraph (250 bytes), but more sophisticated processing was necessary for more direct responses (50 bytes).
We propose two novel model architectures for computing continuous vector representations of words from very large data sets. The quality of these representations is measured in a word similarity task, and the results are compared to the previously best performing techniques based on different types of neural networks. We observe large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set. Furthermore, we show that these vectors provide state-of-the-art performance on our test set for measuring syntactic and semantic word similarities.
This chapter touches upon several issues in the calculation and assessment of inter-annotator agreement. It gives an introduction to the theory behind agreement coefficients and examples of their application to linguistic annotation tasks. Specific examples explore variation in annotator performance due to heterogeneous data, complex labels, item difficulty, and annotator differences, showing how global agreement coefficients may mask these sources of variation, and how detailed agreement studies can give insight into both the annotation process and the nature of the underlying data. The chapter also reviews recent work on using machine learning to exploit the variation among annotators and learn detailed models from which accurate labels can be inferred. I therefore advocate an approach where agreement studies are not used merely as a means to accept or reject a particular annotation scheme, but as a tool for exploring patterns in the data that are being annotated.
Continuous word representations, trained on large unlabeled corpora are useful for many natural language processing tasks. Many popular models to learn such representations ignore the morphology of words, by assigning a distinct vector to each word. This is a limitation, especially for morphologically rich languages with large vocabularies and many rare words. In this paper, we propose a new approach based on the skip-gram model, where each word is represented as a bag of character n-grams. A vector representation is associated to each character n-gram, words being represented as the sum of these representations. Our method is fast, allowing to train models on large corpus quickly. We evaluate the obtained word representations on five different languages, on word similarity and analogy tasks.
Texts propagate through many social networks and provide evidence for their structure. We describe and evaluate efficient algorithms for detecting clusters of reused passages embedded within longer documents in large collections. We apply these techniques to two case studies: analyzing the culture of free reprinting in the nineteenth-century United States and the development of bills into legislation in the U.S. Congress. Using these divergent case studies, we evaluate both the efficiency of the approximate local text reuse detection methods and the accuracy of the results. These techniques allow us to explore how ideas spread, which ideas spread, and which subgroups shared ideas.
In literary study, intertextuality refers to the reuse of text, where new meaning or novel stylistic effects have been generated. Most typically in the digital humanities, algorithms for intertextual analysis search for approximate lexical correspondence that can be described as paraphrase. In this article, we look at a complimentary approach that more closely captures the behavior of the reader when faced with meaningful connections between texts in the absence of words that have the same form or stem, which constrains the match to semantics. The technique we employ for identifying such semantic intertextuality is the popular natural language processing strategy of semantic analysis. Unlike the typical scenario for semantic analysis, where a corpus of long form documents is available, we examine the far more limited textual fragments that embody intertextuality. We are primarily concerned with texts from antiquity, where small phrases or passages often form the locus of comparison. In this vein, we look at a specific case study of established parallels between book 1 of Lucan’s Civil War and all of Vergil’s Aeneid. Applying semantic analysis over these texts, we are able to recover parallels that lexical matching cannot, as well as discover new and interesting thematic matches between the two works.
The study of intertextuality, or how authors make artistic use of other texts in their works, has a long tradition, and has in recent years benefited from a variety of applications of digital methods. This article describes an approach for detecting the sorts of intertexts that literary scholars have found most meaningful, as embodied in the free Tesserae website http://tesserae.caset.buffalo.edu/. Tests of Tesserae Versions 1 and 2 showed that word-level n-gram matching could recall a majority of parallels identified by scholarly commentators in a benchmark set. But these versions lacked precision, so that the meaningful parallels could be found only among long lists of those that were not meaningful. The Version 3 search described here adds a second stage scoring system that sorts the found parallels by a formula accounting for word frequency and phrase density. Testing against a benchmark set of intertexts in Latin epic poetry shows that the scoring system overall succeeds in ranking parallels of greater significance more highly, allowing site users to find meaningful parallels more quickly. Users can also choose to adjust both recall and precision by focusing only on results above given score levels. As a theoretical matter, these tests establish that lemma identity, word frequency, and phrase density are important constituents of what make a phrase parallel a meaningful intertext.
Text re-use describes the spoken and written repetition of information. Historical text re-use, with its longer time span, embraces a larger set of morphological, linguistic, syntactic, semantic and copying variations, thus adding a complication to text-reuse detection. Furthermore, it increases the chances of redundancy in a Digital Library. In Natural Language Processing it is crucial to remove these redundancies before applying any kind of machine learning techniques to the text. In Humanities, these redundancies foreground textual criticism and allow scholars to identify lines of transmission. This chapter investigates two aspects of the historical text re-use detection process, based on seven English editions of the Holy Bible. First, we measure the performance of several techniques. For this purpose, when considering a verse—such as book Genesis, Chapter 1, Verse 1—that is present in two editions, one verse is always understood as a paraphrase of the other. It is worth noting that paraphrasing is considered a hyponym of text re-use. Depending on the intention with which the new version was created, verses tend to differ significantly in the wording, but not in the meaning. Secondly, this chapter explains and evaluates a way of extracting paradigmatic relations. However, as regards historical languages, there is a lack of language resources (for example, WordNet) that makes non-literal text re-use and paraphrases much more difficult to identify. These differences are present in the form of replacements, corrections, varying writing styles, etc. For this reason, we introduce both the aforementioned and other correlated steps as a method to identify text re-use, including language acquisition to detect changes that we call paradigmatic relations. The chapter concludes with the recommendation to move from a ”single run” detection to an iterative process by using the acquired relations to run a new task.
We present here a method for automat-ically discovering several classes of text reuse across different languages, from the most similar (translations) to the most oblique (literary allusions). Allusions are an important subclass of reuse because they involve the appropriation of isolated words and phrases within otherwise unre-lated sentences, so that traditional methods of identifying reuse including topical sim-ilarity and translation models do not ap-ply. To evaluate this work we have created (and publicly released) a test set of liter-ary allusions between John Milton's Par-adise Lost and Vergil's Aeneid; we find that while the baseline discovery of trans-lations (55.0% F-measure) far surpasses the discovery of allusions (4.8%), its abil-ity to expedite the traditional work of hu-manities scholars makes it a line of re-search strongly worth pursuing.
A new method for automatic indexing and retrieval is described. The approach is to take advantage of implicit higher-order structure in the association of terms with documents (“semantic structure”) in order to improve the detection of relevant documents on the basis of terms found in queries. The particular technique used is singular-value decomposition, in which a large term by document matrix is decomposed into a set of ca. 100 orthogonal factors from which the original matrix can be approximated by linear combination. Documents are represented by ca. 100 item vectors of factor weights. Queries are represented as pseudo-document vectors formed from weighted combinations of terms, and documents with supra-threshold cosine values are returned. initial tests find this completely automatic method for retrieval to be promising.
We describe here a method for automatically identifying word sense variation in a dated collection of historical books in a large digital library. By leveraging a small set of known translation book pairs to induce a bilingual sense inventory and labeled training data for a WSD classifier, we are able to automatically classify the Latin word senses in a 389 million word corpus and track the rise and fall of those senses over a span of two thousand years. We evaluate the performance of seven different classifiers both in a tenfold test on 83,892 words from the aligned parallel corpus and on a smaller, manually annotated sample of 525 words, measuring both the overall accuracy of each system and how well that accuracy correlates (via mean square error) to the observed historical variation.
We propose a computational model of text reuse tailored for ancient literary texts, avail- able to us often only in small and noisy sam- ples. The model takes into account source alternation patterns, so as to be able to align even sentences with low surface similarity. We demonstrate its ability to characterize text reuse in the Greek New Testament.
Categorization and taxonomy are topical issues in intertextuality studies. Instead of increasing the number of overlapping or contradictory definitions (often established with reference to limited databases) which exist even for key concepts such as "allusion" or "quotation", we propose an electronically implemented data-driven approach based on the isolation, analysis and description of a number of relevant parameters such as general text relation, marking for quotation, modification etc. If a systematic parameter analysis precedes discussions of possible correlations and the naming of features bundles as composite categories, a dynamic approach to categorization emerges which does justice to the varied and complex phenomena in this field. The database is the HyperHamlet corpus, a chronologically and generically wide-ranging collection of Hamlet references that confront linguistic and literary researchers with a comprehensive range of formal and stylistic issues. Its multi-dimensional encodings and search facilities provide the indispensable 'freedom from the analytic limits of hardcopy', as Jerome McGann put it. The methodological and heuristic gains include a more complete description of possible parameter settings, a clearer recognition of multiple parameter settings (as implicit in existing genre definitions), a better understanding of how parameters interact, descriptions of disregarded literary phenomena that feature unusual parameter combinations and, finally, descriptive labels for the most polysemous areas that may clarify matters without increasing taxonomical excess.
Class-tested and coherent, this textbook teaches classical and web information retrieval, including web search and the related areas of text classification and text clustering from basic concepts. It gives an up-to-date treatment of all aspects of the design and implementation of systems for gathering, indexing, and searching documents; methods for evaluating systems; and an introduction to the use of machine learning methods on text collections. All the important ideas are explained using examples and figures, making it perfect for introductory courses in information retrieval for advanced undergraduates and graduate students in computer science. Based on feedback from extensive classroom experience, the book has been carefully structured in order to make teaching more natural and effective. Slides and additional exercises (with solutions for lecturers) are also available through the book's supporting website to help course instructors prepare their lectures.
The logic and discovery of textual allusion
- David Bamman
- Gregory Crane
David Bamman and Gregory Crane. 2008. The logic
and discovery of textual allusion. In Proceedings of
the Second Workshop on Language Technology for
Cultural Heritage Data.
Informationstechnische Aspekte des Historical Text Re-use
- Marco Büchler
Marco Büchler. 2013. Informationstechnische Aspekte
des Historical Text Re-use. Ph.D. thesis, Universität
Leipzig.
Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures
- Alexander Budanitsky
- Graeme Hirst
Alexander Budanitsky and Graeme Hirst. 2001. Semantic distance in WordNet: An experimental,
application-oriented evaluation of five measures. In
Workshop on WordNet and other lexical resources,
volume 2, page 2.
From word embeddings to document distances
- Matt Kusner
- Yu Sun
- Nicholas Kolkin
- Kilian Weinberger
Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian
Weinberger. 2015. From word embeddings to document distances. In International Conference on Machine Learning, pages 957-966.
Python implementation of Empirical Mode Decomposition algorithm
- Dawid Laszuk
Dawid Laszuk. 2017. Python implementation of Empirical Mode Decomposition algorithm.
The Latin WordNet project
- Stefano Minozzi
Stefano Minozzi. 2010. The Latin WordNet project.
In Akten des 15. Internationalen Kolloquiums zur
Lateinischen Linguisti, pages 707-716, Innsbruck.
Institut für Sprachen und Literaturen der Universität
Innsbruck Bereich Sprachwissenschaft.
A Method for Human-Interpretable Paraphrasticality Prediction
- Maria Moritz
- Johannes Hellrich
- Sven Buechel
Maria Moritz, Johannes Hellrich, and Sven Buechel.
2018. A Method for Human-Interpretable Paraphrasticality Prediction. In Proceedings of the Second Joint SIGHUM Workshop on Computational
Linguistics for Cultural Heritage, Social Sciences,
Humanities and Literature, pages 113-118.
- Waleed Ammar
- George Mulcaire
- Yulia Tsvetkov
- Guillaume Lample
- Chris Dyer
- Noah A Smith
Waleed Ammar, George Mulcaire, Yulia Tsvetkov,
Guillaume Lample, Chris Dyer, and Noah A Smith.
2016. Massively multilingual word embeddings.
arXiv preprint arXiv:1602.01925.