Detection error trade-off curves for all considered learning methods based on GOLD-NEW data. The full curves are shown on the left, and a zoomed plot focusing on low values of false negatives is shown on the right. The enlarged dots indicate the false positives and false negatives corresponding to the classification threshold of 0.5.

Detection error trade-off curves for all considered learning methods based on GOLD-NEW data. The full curves are shown on the left, and a zoomed plot focusing on low values of false negatives is shown on the right. The enlarged dots indicate the false positives and false negatives corresponding to the classification threshold of 0.5.

Source publication
Article
Full-text available
In the two recent decades various security authorities around the world acknowledged the importance of exploiting the ever-growing amount of information published on the web on various types of events for early detection of certain threats, situation monitoring and risk analysis. Since the information related to a particular real-world event might...

Context in source publication

Context 1
... Section provides comparative information on errors made by the different learning methods classifiers. Fig. 7 shows the detection error trade-off curves for all learners evaluated, while the ROC curves are provided in Fig. 8. We can observe from these diagrams the superiority of the LSTM-based approach in terms of lower false positive rates, i.e., classifying event pairs as related that are in fact ...

Similar publications

Article
Full-text available
With the increasing popularity of social media, people has changed the way they access news. News online has become the major source of information for people. However, much information appearing on the Internet is dubious and even intended to mislead. Some fake news are so similar to the real ones that it is difficult for human to identify them. T...
Article
Full-text available
In recent years, the consumption of social media content to keep up with global news and to verify its authenticity has become a considerable challenge. Social media enables us to easily access news anywhere, anytime, but it also gives rise to the spread of fake news, thereby delivering false information. This also has a negative impact on society....
Chapter
Full-text available
This article proposes a method to classify atrial fibrillation signals using time-frequency characteristics through a BiLSTM network. The experiment was performed with the ECG signals, which are part of the PhysioNet CinC 2017 database. In addition to the BiLSTM network, machine learning algorithms such as k Nearest Neighbors, Linear SVM, RBF SVM,...
Article
Full-text available
Manual investigation is warranted in traditional approaches for estimating the bug severity level, which adds to the effort and time required. For bug severity report prediction, numerous automated strategies have been proposed in addition to manual ones. However, the current bug report predictors by facing several issues, such as overfitting and w...
Article
Full-text available
Purpose: However, from the variety of uses of these algorithms, in general, accuracy problems are still a concern today, even accuracy problems related to multi-class classification still require further research.Methods: This study proposes a stacking ensemble classifier method to produce better accuracy by combining Logistic Regression, Random Fo...

Citations

... The second way is to use deep learning methods. Shallow and deep learning are both used for event-relatedness classification [15]; they compare the performance of shallow learning methods and deep learning approaches based on long short-term memory (LSTM) recurrent neural network. Other studies are mainly about the implicit relation of discourse (IDRR), which is shorter than news data. ...
Article
Full-text available
With more and more news articles appearing on the Internet, discovering causal relations between news articles is very important for people to understand the development of news. Extracting the causal relations between news articles is an inter-document relation extraction task. Existing works on relation extraction cannot solve it well because of the following two reasons: (1) most relation extraction models are intra-document models, which focus on relation extraction between entities. However, news articles are many times longer and more complex than entities, which makes the inter-document relation extraction task harder than intra-document. (2) Existing inter-document relation extraction models rely on similarity information between news articles, which could limit the performance of extraction methods. In this paper, we propose an inter-document model based on storytree information to extract causal relations between news articles. We adopt storytree information to integer linear programming (ILP) and design the storytree constraints for the ILP objective function. Experimental results show that all the constraints are effective and the proposed method outperforms widely used machine learning models and a state-of-the-art deep learning model, with F1 improved by more than 5% on three different datasets. Further analysis shows that five constraints in our model improve the results to varying degrees and the effects on the three datasets are different. The experiment about link features also suggests the positive influence of link information.
... Thus, we verify whether it is possible to exploit knowledge and tools proposed to address similar problems. With respect to conventional machine learning methods, the deep ones have unique advantages in feature extraction and semantic mining [11], and have achieved excellent results in text classification tasks [12,13,14,15,16,17]. Thus, from the analysis of user stories the deep learning models can infer individual privacy information and privacy rules, which can be used to recognize privacy-related entities for individual user stories. ...
... The comparison between shallow and deep learning methods is often performed when evaluating text classification tools (e.g., [13,14,15]), mainly due to the possible noise in the data that can lead to substantial changes in the accuracy of decisions [13]. In particular, in some studies, shallow learning methods outperformed the deep ones in text classification tasks [15]. ...
Preprint
Full-text available
To provide privacy-aware software systems, it is crucial to consider privacy from the very beginning of the development. However, developers do not have the expertise and the knowledge required to embed the legal and social requirements for data protection into software systems. Objective: We present an approach to decrease privacy risks during agile software development by automatically detecting privacy-related information in the context of user story requirements, a prominent notation in agile Requirement Engineering (RE). Methods: The proposed approach combines Natural Language Processing (NLP) and linguistic resources with deep learning algorithms to identify privacy aspects into User Stories. NLP technologies are used to extract information regarding the semantic and syntactic structure of the text. This information is then processed by a pre-trained convolutional neural network, which paved the way for the implementation of a Transfer Learning technique. We evaluate the proposed approach by performing an empirical study with a dataset of 1680 user stories. Results: The experimental results show that deep learning algorithms allow to obtain better predictions than those achieved with conventional (shallow) machine learning methods. Moreover, the application of Transfer Learning allows to considerably improve the accuracy of the predictions, ca. 10%. Conclusions: Our study contributes to encourage software engineering researchers in considering the opportunities to automate privacy detection in the early phase of design, by also exploiting transfer learning models.
... A variety of machine learning models and NLP-related tasks are implemented in almost various programming languages. Shallow and Deep learning techniques in NLP give better results as compared to other techniques [51]. These models cannot depend upon classical taskspecific methodology and work as a single end-to-end model. ...
Article
Nowadays, plagiarism became very common in many fields of life, such as research and educational fields. Due to the advancement in plagiarism techniques adopted by plagiarists, it is challenging to detect plagiarism accurately by the existing techniques. Different features are observed while checking plagiarism, such as syntactic, lexical, semantic, and structural features. This research explores new and modern plagiarism detection tasks especially, text-based plagiarism detection including monolingual plagiarism detection. We proposed a four-stage novel framework for plagiarism detection. This framework uses natural Language Processing (NLP) instead of focusing on traditional string-matching approaches. This framework explores text similarity by combining two metrics as Skip-Gram and Dice Coefficient, on the corpus-based approach. The Deep meaning of the text is explored by the use of the Deep and shallow NLP technique. Our results conclude that Heavy revision is identifying quickly through Deep NLP. Shallow NLP prepares text very well that is processed further quickly. Word2vec results are close to simple Deep NLP methods, but word2vec also highlights those documents that other techniques may not highlight. Synonym and phrase changes are also captured through Deep NLP.
Article
Discourse structure is generally represented as hierarchical structure, the two most well known representations are rhetorical structure theory (RST) and Penn discourse treebank (PDTB). The main problem of the hierarchical structure is that it can not describe the direct semantic relevance between the elementary discourse units (EDU), especially the non-adjacent and cross-level EDUs. Discourse dependency structure (DDS) has been put forward in recent years to describe the head-dependent relation between the EDUs. However, the judgment process of the head can not be answered theoretically. This problem is particularly serious in Chinese discourse analysis, because Chinese lacks the form differences between the main clauses and the subordinate clauses. In this paper, we propose clause relevance structure to represent the discourse structure. Compared with the hierarchical discourse structure and DDS, the clause relevance structure can effectively describe the direct semantic association between discontinuous and cross-level clauses in a text, and the construction of structure is not presupposed by the head recognition. We propose the judgment criteria and formal constraints of the clause relevance structure, and built a human-annotated corpus on Chinese text. Based on the Chinese corpus, we explore the automatic recognition of clause relevance structure. The clause relevance recognition task is formalized as a classification problem and performed by the BERT-based model. A bidirectional LSTM layer is added on the top of the BERT to improve the performance, and the recognition accuracy (90.77%) is achieved by the BERT-LSTM model. Experimental results show that the long distance clause pairs are the main difficulties in the clause relevance recognition, and these difficulties mainly focus on the positive examples, while the clause pairs with short distance are especially difficult to be correctly recognized as negative relevance.
Article
Rumor refutation is a common method to control rumors to address potential risks. This paper studies the social media rumor refutation effectiveness lifecycle (SMRREL), focusing on three important characteristics (i.e., lifespan, peak value, and distribution) to provide support for (1) enhancing the persistence and intensity of rumor refutation effectiveness and (2) investigating the changing law of rumor refutation effectiveness. In total, 77,080 comment records, 55,847 forward records, and other pertinent data of 251 rumor refutation microblogs from an official rumor refutation platform are collected to perform analysis. To explore how the lifespan and peak value of SMRREL are influenced by the possible affecting factors, five regressors (i.e., RFRegressor, AdaBoostRegressor, XGBoostRegressor, LGBMRegressor, and CatBoostRegressor) are trained based on the collected data. The XGBoostRegressor shows the best performance, and the results are shown and explained using SHapley Additive exPlanations (SHAP). To investigate the distribution of SMRREL, lifecycle graphs of rumor refutation effectiveness are summarized and divided into three types, i.e., Outburst, Multiple Peaks, and Steep Slope. Finally, based on the results of the SMRREL analysis, corresponding decision-making recommendations are proposed to make better persistence and intensity of rumor refutation effectiveness.
Article
Random forest (RF) and MaxEnt models are shallow machine learning approaches that perform well in predicting species' potential distributions. RF models can produce robust results with the default automatic configuration in most cases, but it is necessary for MaxEnt to optimize the model settings to improve the performance, and the predictive performance difference between optimized MaxEnt and RF is uncertain. To explore this issue, the potential distribution of the endangered amphibian Quasipaa boulengeri in China was predicted using optimized MaxEnt and RF models. A total of 408 occurrence data were selected, 1000 locations were generated as pseudo-absence data by the geographic distance method, and 10,000 sites were selected as background data by creating a bias file. Partial ROC at different thresholds and success rate curves were used to compare the predictive performances between optimized MaxEnt and RF. Our results showed that the RF and optimized MaxEnt models both had good performance in predicting the potential distribution of Q. boulengeri, with the RF model performing slightly better whether based on partial ROC or success rate curves. Furthermore, the core suitable habitat regions of Q. boulengeri identified by RF and MaxEnt were similar and were all located in the Sichuan, Chongqing, Hubei, Hunan, and Guizhou provinces. However, the RF model produced a habitat suitability map with higher discrimination and greater heterogeneity. Temperature annual range, mean temperature of the driest quarter, and annual precipitation were the vital environmental variables limiting the distribution of Q. boulengeri. The RF model is the stronger machine learner. We believe it may be more applicable in predicting the native potential distributions of species with sufficient occurrence data, given the additional predictive detail, the simplicity of use, the computational time involved, and the operational complexity.
Article
Context To provide privacy-aware software systems, it is crucial to consider privacy from the very beginning of the development. However, developers do not have the expertise and the knowledge required to embed the legal and social requirements for data protection into software systems. Objective We present an approach to decrease privacy risks during agile software development by automatically detecting privacy-related information in the context of user story requirements, a prominent notation in agile Requirement Engineering (RE). Methods The proposed approach combines Natural Language Processing (NLP) and linguistic resources with deep learning algorithms to identify privacy aspects into User Stories. NLP technologies are used to extract information regarding the semantic and syntactic structure of the text. This information is then processed by a pre-trained convolutional neural network, which paved the way for the implementation of a Transfer Learning technique. We evaluate the proposed approach by performing an empirical study with a dataset of 1680 user stories. Results The experimental results show that deep learning algorithms allow to obtain better predictions than those achieved with conventional (shallow) machine learning methods. Moreover, the application of Transfer Learning allows to considerably improve the accuracy of the predictions, ca. 10%. Conclusions Our study contributes to encourage software engineering researchers in considering the opportunities to automate privacy detection in the early phase of design, by also exploiting transfer learning models.
Article
Natural language processing (NLP) based approaches have recently received attention for legal systems of several countries. It is of interest to study the wide variety of legal systems that have so far not received any attention. In particular, for the legal system of the Republic of Turkey, codified in Turkish, no works have been published. We first review the state-of-the-art of NLP in law, and then study the problem of predicting verdicts for several different courts, using several different algorithms. This study is much broader than earlier studies in the number of different courts and the variety of algorithms it includes. Therefore it provides a reference point and baseline for further studies in this area. We further hope the scope and systematic nature of this study can set a framework that can be applied to the study of other legal systems. We present novel results on predicting the rulings of the Turkish Constitutional Court and Courts of Appeal, using only fact descriptions, and without seeing the actual rulings. The methods that are utilized are based on Decision Trees (DTs), Random Forests (RFs), Support Vector Machines (SVMs) and state-of-the-art deep learning (DL) methods; specifically Gated Recurrent Units (GRUs), Long Short-Term Memory networks (LSTMs) and bidirectional LSTMs (BiLSTMs), with the integration of an attention mechanism for each model. The prediction results for all algorithms are given in a comparative and detailed manner. We demonstrate that outcomes of the courts of Turkish legal system can be predicted with high accuracy, especially with deep learning based methods. The presented results exhibit similar performance to earlier work in the literature for other languages and legal systems.
Article
Extracting events from texts using neural networks has gained increasing research focus in recent years. However, existing methods prepare candidate arguments in a separate classifier suffering from the error propagation problem and fail to model correlations between entity mentions and event structures. To improve the performance of both entity recognition and event extraction, we propose a transition-based joint neural model for the tasks by converting graph structures to a set of transition actions. In particular, we design ten types of novel actions and introduce a global normalization strategy to alleviate the label bias issue. We conduct experiments based on the widely used Automatic Content Extraction (ACE) corpora and the results show that our model achieves 88.7% F1-score on entities and 75.3% F1-score on event triggers, outperforming the baseline neural networks by a large margin. Further in-depth analysis shows the effectiveness of our model in capturing structural dependencies in long sentences. The proposed model can be used for facilitating a range of downstream tasks.