Content uploaded by Riyaz Ahamed Ariyaluran Habeeb
Author content
All content in this area was uploaded by Riyaz Ahamed Ariyaluran Habeeb on Mar 04, 2019
Content may be subject to copyright.
A preview of the PDF is not available
Real-time big data processing for anomaly detection: A Survey
Abstract and Figures
The advent of connected devices and omnipresence of Internet have paved way for intruders to attack networks, which leads to cyber-attack, financial loss, information theft in healthcare, and cyber war. Hence, network security analytics has become an important area of concern and has gained intensive attention among researchers, off late, specifically in the domain of anomaly detection in network, which is considered crucial for network security. However, preliminary investigations have revealed that the existing approaches to detect anomalies in network are not effective enough, particularly to detect them in real time. The reason for the inefficacy of current approaches is mainly due the amassment of massive volumes of data though the connected devices. Therefore, it is crucial to propose a framework that effectively handles real time big data processing and detect anomalies in networks. In this regard, this paper attempts to address the issue of detecting anomalies in real time. Respectively, this paper has surveyed the state-of-the-art real-time big data processing technologies related to anomaly detection and the vital characteristics of associated machine learning algorithms. This paper begins with the explanation of essential contexts and taxonomy of real-time big data processing, anomalous detection, and machine learning algorithms, followed by the review of big data processing technologies. Finally, the identified research challenges of real-time big data processing in anomaly detection are discussed.
Figures - uploaded by Riyaz Ahamed Ariyaluran Habeeb
Author content
All figure content in this area was uploaded by Riyaz Ahamed Ariyaluran Habeeb
Content may be subject to copyright.
... The study on real-time intrusion detection, however, is still in its infancy and is insufficient [10]. Three categories describe the primary real-time intrusion detection studies. ...
... Anomaly detection is a critical component of fraud prevention, as it involves identifying patterns in data that do not conform to expected behavior. In the context of financial transactions, anomalies can indicate fraudulent activities that deviate from normal transaction patterns (Habeeb, et. al., 2019, Thudumu, et. al., 2020. The importance of anomaly detection lies in its ability to; Traditional methods often rely on known fraud patterns, but anomaly detection can uncover previously unknown or emerging fraud tactics. By identifying anomalies early, financial institutions can prevent fraud before significant losses occur. Anomaly det ...
In the rapidly evolving landscape of cyber financial fraud, traditional detection methods are increasingly inadequate to counter sophisticated fraudulent activities. This study examines the potential of deep learning techniques, specifically focusing on neural networks and anomaly detection, to enhance cyber financial fraud detection. Neural networks, with their ability to model complex patterns and relationships in data, offer a robust framework for identifying fraudulent transactions. The study examines the application of various neural network architectures, including Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), which are adept at processing sequential data and identifying anomalies that signify fraudulent behavior. Anomaly detection, a critical aspect of this research, leverages unsupervised learning techniques to identify outliers in financial transactions that do not conform to established patterns. By employing autoencoders and generative adversarial networks (GANs), the study demonstrates how these models can effectively differentiate between legitimate and suspicious activities without the need for labeled datasets. This is particularly beneficial in the financial sector, where fraudulent patterns constantly evolve, and labeled data may be scarce or outdated. The integration of these deep learning techniques into existing fraud detection frameworks is explored, highlighting the benefits of real-time analysis and predictive capabilities. The study also addresses the challenges associated with implementing deep learning models, such as the need for high-quality data, computational resources, and the interpretability of model outputs. Furthermore, the research underscores the importance of continuous model training and adaptation to keep pace with emerging fraud tactics. By leveraging advanced neural network architectures and anomaly detection methods, financial institutions can significantly enhance their fraud detection capabilities, leading to reduced financial losses and increased security for customers. In conclusion, this study provides a comprehensive analysis of how deep learning techniques, particularly neural networks and anomaly detection, can transform cyber Website: https://www.eajournals.org/ Published of the European Centre for Research Training and Development UK 91 financial fraud detection. It emphasizes the need for ongoing research and development in this field to stay ahead of fraudsters and protect the integrity of financial systems. The findings suggest that deep learning not only enhances the accuracy and efficiency of fraud detection but also offers a scalable solution adaptable to the dynamic nature of cyber financial fraud.
... Seiring dengan upaya berbagai industri untuk memproses data secara real-time, meskipun model komputasi awan tradisional sudah cukup untuk penyimpanan dan pemrosesan data dalam jumlah besar (Cheng et al., 2018), tantangan terkait latensi dan kebutuhan pembaruan data yang cepat menjadi poin utama, terutama pada aplikasi yang memerlukan pengambilan keputusan cepat berdasarkan analisis data yang selalu diperbarui (Lu et al., 2020;Shukla et al., 2023). Dalam konteks ini, dibutuhkan sistem yang efisien dan otomatis untuk menangani transfer data menjadi sangat penting (Ariyaluran et al., 2019;Bakken et al., 2011). Salah satu solusi yang dapat diterapkan adalah cron job yang mengotomatisasi operasi berulang pada interval yang telah ditentukan sebelumnya sehingga manajemen data dapat dilakukan dengan tepat waktu dan konsisten (Bourne & Fox, 1984). ...
Perkembangan pesat perangkat Internet of Things (IoT) telah menghasilkan peningkatan signifikan dalam volume data, yang menciptakan tantangan dalam pemrosesan data secara real-time. Meskipun sistem cloud-fog dapat mengurangi latensi dengan memproses data lebih dekat ke sumbernya, pengelolaan dan transfer data tetap bergantung pada tugas terjadwal yang efisien. Penelitian ini bertujuan untuk mengurangi intervensi manual dan meningkatkan efisiensi sistem melalui penggunaan cron job yang dikonfigurasi dengan logika skrip PHP dan diimplementasikan di Cloud Panel. Sistem ini dirancang untuk menjadwalkan transfer data secara otomatis dengan interval yang disesuaikan dan memastikan pembaruan data yang tepat waktu dalam lingkungan cloud-fog. Hasil penelitian menunjukkan bahwa penerapan otomatisasi transfer data berhasil mengurangi kesalahan, meningkatkan efisiensi, dan memastikan pembaruan data yang lebih tepat waktu dalam sistem cloud-fog
... Ariyaluran Habeeb et al. (2019) -Розглядають підходи до обробки великих даних у реальному часі для виявлення аномалій на основі автоенкодерів. Автори визначають ключові проблеми, пов'язані зі швидкістю обробки даних та складністю адаптації моделей до нових процесів [4]. Bezerra et al. (2008) -Спеціалізуються на алгоритмах виявлення аномалій у бізнес-процесах, та наголошують на необхідності адаптації нових підходів, таких як DNN [5]. ...
This work evaluates the effectiveness of various deep neural network (DNN) architectures for anomaly detection in business processes, which is a critical task in the context of digital transformation and automation of managerial decisions. The study examines and compares different architectures: graph neural networks (GNN) for detecting structural anomalies, recurrent neural networks (LSTM, RNN) and transformers (Transformers) for time series analysis and forecasting, as well as autoencoders (Autoencoders) for processing attribute-based data. An adaptive approach is proposed, integrating the advantages of different models depending on the type and characteristics of a business process. The research is based on a large dataset extracted from a BPMS system, including enriched process graphs with business parameters. Four main types of anomalies are considered: Missing Steps, Duplicate Steps, Wrong Route, and Abnormal Duration. The experimental part of the study evaluates model performance using key metrics, including Precision, Recall, F1-score, AUC-ROC, AUPRC, ADR (Anomaly Detection Rate), FAR (False Alarm Rate), and FNR (False Negative Rate), as well as training time and a confusion matrix for a detailed analysis of predicted class distributions. The results indicate that Transformers achieve the highest accuracy in detecting complex event sequences, GNNs excel in identifying structural anomalies, and Autoencoders perform effectively when working with attribute-based datasets. Recommendations are provided for selecting an appropriate architecture based on the specific characteristics of a business process and computational resource constraints. The findings can be applied to optimize process monitoring, automate anomaly detection, and enhance the efficiency of business process management in a rapidly evolving digital environment.
... Sensörlerden alınan veriler, birbirine bağlı cihazlar, akıllı ev uygulamaları, akıllı şehir uygulamaları, iletişim, sağlık gibi uygulamaların artması ile birlikte gerçek zamanlı olarak akan veri miktarı günden güne artmaktadır [1]. Hızla gelişen teknolojilere paralel olarak günümüzde günlük olarak üretilen veri miktarı üssel bir oranda artarken üretilen büyük miktarda verinin depolanması ve yönetilmesi gerekmekte, anlamlı bilgiler çıkarılması önem arz etmektedir. ...
Günümüzde sensör sayısının artması ile birlikte yüksek hızda, çeşitlilikte ve hacimde veri üretilmektedir. Üretilen yüksek hızda ve farklı kaynaklardan gelen verinin birlikte analiz edilmesi önem arz etmektedir. Bu noktada büyük veri sistemleri katmanlı mimarisi ile birlikte çözümler sunmaktadır. Her bir katmanda farklı uygulamalar çalışmakta ve birbirleri ile iletişim kurarak çalışmaktadırlar. Bu çalışma, kubernetes mimarisi kullanılarak sensör verilerinin birleştirilmesi, yapay zeka yöntemleri ile anomali tespiti ve anlık verilerin işlenmesine yönelik model sunmaktadır. Önerilen sistem modeli verilerin işlenmesi ve birleştirilmesi, yapay zekâ tabanlı model geliştirilmesi ve görselleştirme aşamalarından oluşmaktadır. Bu noktada kubernetes mimarisi ile birlikte orkestrasyon işlemi sağlanarak açık kaynak kodlu uygulamalar aracılığı ile veri birleştirme, işleme ve görselleştirme işlemleri dağıtık, hata toleranslı ve etkin kaynak yönetimi sağlayacak şekilde oluşturulmuştur. Anomali tespit işlemi için ise yapay zekâ algoritmalarından LSTM, GRU ve Conv1d algoritmaları kullanılarak karşılaştırmalı analiz yapılmıştır. Sistemin normal durumuna, eğitim ve anlık veri analizi aşamasındaki durumlarına ilişkin işlemci kullanım oranları karşılaştırılarak sonuçlar sunulmuştur. Sonuç olarak önerilen kubernetes tabanlı anomali tespit modeli ile baştan sona veri toplama aşamasından başlayarak veri işleme ve görselleştirme aşamalarına ilişkin bir sistem modeli gerçekleştirilmiştir. Bu bağlamda önerilen sistem modelinin akış verilerini birleştirerek analizinde dağıtık işlem gerçekleştirme ve hata toleranslı şekilde işlemleri gerçekleştirdiği görülmüştür.
Traditional hospital property management faces many challenges, such as data silos, real-time query lag, insufficient prediction intelligence, lack of interactivity in visualization, inefficient system integration, and lack of intelligent decision support. These problems affect the accuracy and real-time performance of hospital equipment maintenance, resource allocation, and financial management. To address these issues, this paper uses the Long Short-Term Memory (LSTM) algorithm to construct an intelligent hospital financial management system. The system collects hospital property data in real-time through Internet of Things (IoT) equipment, and uses the LSTM model for analysis and prediction to achieve real- time query and optimization of equipment status, resource consumption, and financial expenditure, thereby improving the visualization of property data. The study trains the LSTM model to analyze real-time data, and then constructs a management system through a modular structure. The comparative experimental findings demonstrate that the constructed system is superior to other similar systems in performance, economic benefits, and stability. The equipment prediction accuracy and recall rate are 92.5% and 88.4% respectively, and no failures occur during the experiment. Therefore, the intelligent financial management solution proposed in this paper has strong practicality and promotion value.
The rapidly changing nature of cyber threats is becoming difficult to deal with, and this requires developing innovative solutions. This study presents an innovative AI-based approach for anomaly detection in real-time network traffic. Wireshark, a widely used tool in network traffic analysis, was used in the data collection phase of the proposed model and as a testbed for live anomaly detection. A high-performance hybrid model was developed by combining Long Short-Term Memory (LSTM) and Graph Convolutional Networks (GCN), which perform well on time series data. A comprehensive dataset consisting of packets containing many attacks and harmless network packets was created and made available for open access. The proposed hybrid model achieved 97% accuracy on a large and comprehensive dataset. In addition, it was proven that live anomaly detection in network traffic could be achieved by using the developed test code and the Wireshark tool in an integrated manner. It is anticipated that in the future, artificial intelligence and deep learning-based methods will find more place in critical tasks such as anomaly detection and threat analysis in the field of cybersecurity and that these technologies will provide more effective and dynamic solutions against constantly evolving threats.
With the increasing liberalization of energy markets, the penetration of renewable clean energy sources, such as photovoltaics and wind power, has gradually increased, providing more sustainable energy solutions for energy-intensive industrial sectors or parks, such as iron and steel production. However, the issues of the intermittency and volatility of renewable energy have become increasingly evident in practical applications, and the economic performance and operational efficiency of localized microgrid systems also demand thorough consideration, posing significant challenges to the decision and management of power system operation. A smart microgrid can effectively enhance the flexibility, reliability, and resilience of the grid, through the frequent interaction of generation–grid–load. Therefore, this paper will provide a comprehensive summary of existing knowledge and a review of the research progress on the methodologies and strategies of modeling technologies for intelligent power systems integrating renewable energy in industrial production.
Over 25 million people suffered from cerebral strokes in a span of 23 years. Many systems are being developed to monitor and improve the life of patients that suffer from different diseases. However, solutions for cerebral strokes are hard to find. Moreover, due to their widespread utilization, smartphones have presented themselves as the most appropriate devices for many e-health systems. In this paper, we propose a cerebral stroke detection solution that employs the cloud to store and analyze data in order to provide statistics to public institutions. Moreover, the prototype of the application is presented. The three most important symptoms of cerebral strokes were considered to develop the tasks that are conducted. Thus, the first task detects smiles, the second task employs voice recognition to determine if a sentence is repeated correctly and, the third task determines if the arms can be raised. Several tests were performed in order to verify the application. Results show its ability to determine whether users have the symptoms of cerebral stroke or not.
With the development of 5G and Internet of Vehicles (IoV) technology, the remote wireless attack on the in-vehicle network have been proven by security researchers. The anomaly detection technology can effectively alleviate the security threat, as then first line of security defense. Based on this, this paper proposes a distributed anomaly detection system using hierarchical temporal memory (HTM) to enhance the security of vehicular Controller Area Network (CAN) bus. The HTM model can predict the flow data in real time, which depends on the state of the previous learning. In addition, we improved the abnormal score mechanism to evaluate the prediction.We manually synthesized field modification and replay attack in data field. Compared with RNN and HMM detection models, the results show that the distributed anomaly detection system based on HTM networks achieves better performance in AUC score, precision and recall.
As an emerging field in the healthcare industry, mobile health service has been paid more and more attention in recent years. To explore the key determinants of individual attitude and behavioral intention, this study, based on 35 related empirical researches, conducted a meta-analysis to develop a comprehensive framework regarding the adoption of individual mobile health services and analyzed the moderating effect of age. Through descriptive statistics, reliability statistics, and correlation analysis, the results of meta-analysis indicate that perceived usefulness, perceived ease of use, perceived vulnerability and perceived severity all have significant impacts on individual attitude, while perceived usefulness, perceived ease of use, subjective norm, trust, perceived risk and attitude significantly influence behavioral intention. The moderator analysis confirmed that different age groups have specific moderating effects on mobile health services adoption, and results suggest that perceived ease of use, perceived vulnerability and perceived severity are more important factors for middle-aged and older users to use mobile health services.
Intrusion Detection is a topic that is of interest both in the corporate world as well as academia. In the advent of Big Data Analytics, multiple analytics techniques can be used on the enormous amounts of data that is being generated every single day in order to discover knowledge. This inherently poses a threat to the security and privacy of all the parties involved. Therefore, it is a necessity in today’s world to reinforce the security systems with robust Intrusion Detection and Prevention Systems. A nominal Cybersecurity System can no longer suffice for detecting and minimizing the damage from cyber-attacks especially since many of the attacks do not fall under a pre-discovered category. In this paper we review the various works particularly concerning Big Heterogeneous Data as well as present opportunities for further research to be conducted in these areas. Keywords: Intrusion, Detection, Cybersecurity, Big Data, Machine Learning, KDDCup99
In the last five decades, maturity models have been introduced as reference frameworks for Information System (IS) management in organizations within different industries. In the healthcare domain, maturity models have also been used to address a wide variety of challenges and the high demand for hospital IS (HIS) implementations. The increasing volume of data, is exceeded the ability of health organizations to process it for improving clinical and financial efficiencies and quality of care. It is believed that careful and attentive use of Data Analytics in healthcare can transform data into knowledge that can improve patient outcomes and operational efficiency. A maturity model in this conjuncture, is a way of identifying strengths and weaknesses of the HIS maturity and thus, find a way for improvement and evolution. This paper presents a proposal to measure Hospitals Information Systems maturity with regard to Data Analytics. The outcome of this paper is a maturity model, which includes six stages of HIS growth and maturity progression.
Fog computing is an emerging paradigm that extends computation, communication, and storage facilities towards the edge of a network. Compared to traditional cloud computing, fog computing can support delay-sensitive service requests from End-Users (EUs) with reduced energy consumption and low traffic congestion. Basically, fog networks are viewed as offloading to core computation and storage. Fog nodes in fog computing decide to either process the services using its available resource or send to the cloud server. Thus, fog computing helps to achieve efficient resource utilization and higher performance regarding the delay, bandwidth, and energy consumption. This survey starts by providing an overview and fundamental of fog computing architecture. Furthermore, service and resource allocation approaches are summarized to address several critical issues such as latency, and bandwidth, and energy consumption in fog computing. Afterward, compared to other surveys, this paper provides an extensive overview of state-of-the-art network applications and major research aspects to design these networks. In addition, this study highlights ongoing research effort, open challenges, and research trends in fog computing.
Participatory sensing has become an emerging technology of this era owing to its low cost in big sensor data collection. Prior to participatory sensing, large-scale deployment complexities were found in wireless sensor networks when collecting data from widespread resources. Participatory sensing systems employ handheld devices as sensors to collect data from communities and transmit to the cloud, where data are further analyzed by expert systems. The processes involved in participatory sensing, such as data collection, transmission, analysis, and visualization, exhibit certain management issues. This study aims to identify big data management issues that must be addressed at the cloud side during data processing and storing and at the participant side during data collection and visualization. It then proposes a framework for big data management in participatory sensing to resolve the contemporary big data management issues on the basis of suggested principles. Moreover, this work presents case studies to elaborate the existence of the highlighted issues. Finally, the limitations, recommendations, and future research directions for academia and industry in the domain of participatory sensing are discussed.
There is a strong interest among academics and practitioners in studying branding issues in the big data era. In
this article, we examine the sentiments toward a brand, via brand authenticity, to identify the reasons for
positive or negative sentiments on social media. Moreover, in order to increase precision, we investigate sentiment
polarity on a five-point scale. From a database containing 2,282,912 English tweets with the keyword
‘Starbucks’, we use a set of 2204 coded tweets both for analyzing brand authenticity and sentiment polarity.
First, we examine the tweets qualitatively to gain insights about brand authenticity sentiments. Then we analyze
the data quantitatively to establish a framework in which we predict both the brand authenticity dimensions and
their sentiment polarity. Through three qualitative studies, we discuss several tweets from the dataset that can be
classified under the quality commitment, heritage, uniqueness, and symbolism categories. Using latent semantic
analysis (LSA), we extract the common words in each category. We verify the robustness of previous findings
with an in-lab experiment. Results from the support vector machine (SVM), as the quantitative research method,
illustrate the effectiveness of the proposed procedure of brand authenticity sentiment analysis. It shows high
accuracy for both the brand authenticity dimensions’ predictions and their sentiment polarity. We then discuss
the theoretical and managerial implications of the studies.