Figure 5 - uploaded by Felix Naumann
Content may be subject to copyright.
Error reduction in rule confidence estimation using an increasing number of training instances. Light blue lines indicate individual rules error.
Source publication
Contexts in source publication
Context 1
... first determine the correct confidence value for each rule based on its annotated facts and use it to calculate the average estimation error over all rules. The continuous red line (SCF) in Figure 5 shows that the average SCF estimate is 19.6% away from the correct confidence value. To calculate the estimation error of Colt-GP, we use an increasing number of instances to train a model for each rule and report the average error of these models. ...
Context 2
... this calculation, we only use rules from Ta- ble 1, that produce at least 100 instances. The blue line (Colt-GP) in Figure 5 shows that, from the beginning, Colt-GP estimates the average rule confidence more accurately than AMIE's SCF estimate. As a result, 10 to 20 training instances are sufficient to achieve an average estimated confidence error of 12.9 to 10.8 percent, which corresponds to an improvement of 6.7 and 8.8 percentage points over the SCF estimate. ...
Context 3
... the number of training data increases, the estimation error decreases and reaches its lowest value of 4.5 percentage points when using 100 training instances. Figure 5 also shows that, although the estimated error between 50-100 training instances improves by only 1.3 percentage points, the variance decreases by 3.5%, indicating that Colt-GP gains more confidence in its predictions. ...
Similar publications
Citations
... And in case of KGs where issues like inaccuracy and incompleteness widely exist, new evaluation metrics are being proposed, such as those in [170], [171]. Many works suggest to check the extracted rules by domain experts before applying, and a latest work [172] introduces the thought of human-in-the-loop and designs a few-shot knowledge validation framework for interactive quality assessment of rules, which takes the rule validation forward one step. ...
... It also tries to sovle the possible conflicts when using various rules. In [172], embedding techniques and user feedback are used interactively to assess the quality of a particular rule, which leads to better estimation of the confidence score than simple statistical measures. ...
As a powerful expression of human knowledge in a structural form, knowledge graph (KG) has drawn great attention from both the academia and the industry and a large number of construction and application technologies have been proposed. Large-scale knowledge graphs such as DBpedia, YAGO and Wikidata are published and widely used in various tasks. However, most of them are far from perfect and have many quality issues. For example, they may contain inaccurate or outdated entries and do not cover enough facts, which limits their credibility and further utility. Data quality has a long research history in the field of traditional relational data and recently attracts more knowledge graph experts. In this paper, we provide a systematic and comprehensive review of the quality management on knowledge graphs, covering overall research topics about not only quality issues, dimentions and metrics, but also quality management processes from quality assessment and error detection, to error correction and KG completion. We categorize existing works in terms of target goals and used methods for better understanding. In the end, we discuss some key issues and possible directions on knowledge graph quality management for further research.
... In addition, rules can be validated with a prediction quality of 75%, requiring as little as 5% of the rule instances to be annotated. The work by Loster et al. [2020b], which was produced in collaboration with Davide Mottin, Paolo Papotti, Jan Ehmueller, and Benjamin Feldmann, constitutes the basis of this chapter. Davide Mottin contributed to the design and formalization of the approach, while Paolo Paotti assisted in creating the rules and the use of RuDik. ...
... The content of this chapter is based on the work of Loster et al. [2020b] and is organized as follows: In Section 4.1, we give a comprehensive introduction to the topic of this chapter. We discuss related work in Section 4.2 and formally define the problem in Section 4.3. ...
Modern knowledge bases contain and organize knowledge from many different topic areas. Apart from specific entity information, they also store information about their relationships amongst each other. Combining this information results in a knowledge graph that can be particularly helpful in cases where relationships are of central importance. Among other applications, modern risk assessment in the financial sector can benefit from the inherent network structure of such knowledge graphs by assessing the consequences and risks of certain events, such as corporate insolvencies or fraudulent behavior, based on the underlying network structure. As public knowledge bases often do not contain the necessary information for the analysis of such scenarios, the need arises to create and maintain dedicated domain-specific knowledge bases. This thesis investigates the process of creating domain-specific knowledge bases from structured and unstructured data sources. In particular, it addresses the topics of named entity recognition (NER), duplicate detection, and knowledge validation, which represent essential steps in the construction of knowledge bases. As such, we present a novel method for duplicate detection based on a Siamese neural network that is able to learn a dataset-specific similarity measure which is used to identify duplicates. Using the specialized network architecture, we design and implement a knowledge transfer between two deduplication networks, which leads to significant performance improvements and a reduction of required training data. Furthermore, we propose a named entity recognition approach that is able to identify company names by integrating external knowledge in the form of dictionaries into the training process of a conditional random field classifier. In this context, we study the effects of different dictionaries on the performance of the NER classifier. We show that both the inclusion of domain knowledge as well as the generation and use of alias names results in significant performance improvements. For the validation of knowledge represented in a knowledge base, we introduce Colt, a framework for knowledge validation based on the interactive quality assessment of logical rules. In its most expressive implementation, we combine Gaussian processes with neural networks to create Colt-GP, an interactive algorithm for learning rule models. Unlike other approaches, Colt-GP uses knowledge graph embeddings and user feedback to cope with data quality issues of knowledge bases. The learned rule model can be used to conditionally apply a rule and assess its quality. Finally, we present CurEx, a prototypical system for building domain-specific knowledge bases from structured and unstructured data sources. Its modular design is based on scalable technologies, which, in addition to processing large datasets, ensures that the modules can be easily exchanged or extended. CurEx offers multiple user interfaces, each tailored to the individual needs of a specific user group and is fully compatible with the Colt framework, which can be used as part of the system. We conduct a wide range of experiments with different datasets to determine the strengths and weaknesses of the proposed methods. To ensure the validity of our results, we compare the proposed methods with competing approaches.
Property graphs serve as unifying abstractions for encoding, inspecting, and updating interconnected data with greater expressive power. They are increasingly popular across various application domains involving real users. However, graph data often contains inconsistencies that need proper transformations to address underlying constraint violations and often require specific domain knowledge. In this paper, we propose an interactive and user-centric approach to repair property graphs under denial constraints. Our approach includes a novel theoretical framework comprising a query-based inconsistency detection mechanism, a dependency graph for tracking violations, and an assignment algorithm facilitating multi-user property graph repairs by leveraging independent sets. We evaluate our approach through several experiments on real-world and synthetic datasets, considering different levels of user expertise and comparing against various baselines. Even with multiple non-oracle users, our approach outperforms existing interactive and non-interactive baselines by 30% on average in terms of repair quality. Additionally, we conduct a user study to assess real user performance in property graph repairs.
Knowledge Graphs (KGs) represent heterogeneous domain knowledge on the Web and within organizations. There exist shapes constraint languages to define validating shapes to ensure the quality of the data in KGs. Existing techniques to extract validating shapes often fail to extract complete shapes, are not scalable, and are prone to produce spurious shapes. To address these shortcomings, we propose the Quality Shapes Extraction (QSE) approach to extract validating shapes in very large graphs, for which we devise both an exact and an approximate solution. QSE provides information about the reliability of shape constraints by computing their confidence and support within a KG and in doing so allows to identify shapes that are most informative and less likely to be affected by incomplete or incorrect data. To the best of our knowledge, QSE is the first approach to extract a complete set of validating shapes from WikiData. Moreover, QSE provides a 12x reduction in extraction time compared to existing approaches, while managing to filter out up to 93% of the invalid and spurious shapes, resulting in a reduction of up to 2 orders of magnitude in the number of constraints presented to the user, e.g., from 11,916 to 809 on DBpedia.
Supervised machine learning has several drawbacks that make it difficult to use in many situations. Drawbacks include heavy reliance on massive training data, limited generalizability, and poor expressiveness of high-level semantics. Low-shot Learning attempts to address these drawbacks. Low-shot learning allows the model to obtain good predictive power with very little or no training data, where structured knowledge plays a key role as a high-level semantic representation of human. This article will review the fundamental factors of low-shot learning technologies, with a focus on the operation of structured knowledge under different low-shot conditions. We also introduce other techniques relevant to low-shot learning. Finally, we point out the limitations of low-shot learning, the prospects and gaps of industrial applications, and future research directions.
