Article

Protecting Water Infrastructure From Cyber and Physical Threats: Using Multimodal Data Fusion and Adaptive Deep Learning to Monitor Critical Systems

If you want to read the PDF, try requesting it from the authors.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Such data is useful to identify a possible deterioration in the person's health status or emergency situations, generate personalized interventions, but also alert caregivers in case of emergency situations [5]. Data can be fused, and machine learning techniques and technologies can be used to extract patterns and detect anomalies or situations that require intervention [6]. The daily living activities data of a person could be used to detect regular daily activities patterns (i.e., daily routine) as well as deviations from them [7]. ...
... We extract from all days, the distinct lengths of activities sequences and the frequency of the occurrence of those lengths: < period, lenght 1 , f requence lenght 1 , . . . , lenght n , f requence lenght n > (6) where: length is the length of some sequence of activities that are part of the filtered days and correspond to a period, f requence lenght is the frequency of occurrence of that length, n is the number of distinct lengths corresponding to the sequences of activities. Based on the frequency of occurrence of a length, we compute a minimum threshold for the activity sequences of that length to be accepted. ...
... For example, if we have 4 clusters, the sequence with the id 12 would remain distributed in its own cluster with a single element. The calculated Dunn and Davies-Bouldin indexes are shown in Table 4 for varying the number of clusters in the range [2,6]. We observe that the best values for the Dunn Index (i.e., when grouping vectors of time span corresponding to the sequences of activities performed during the night) are obtained when using 3 or 4 clusters. ...
Article
Full-text available
The monitoring of the daily life activities routine is beneficial, especially in old age. It can provide relevant information on the person’s health state and wellbeing and can help identify deviations that signal care deterioration or incidents that require intervention. Existing approaches consider the daily routine as a rather strict sequence of activities which is not usually the case. In this paper, we propose a solution to identify flexible daily routines of older adults considering variations related to the order of activities and activities timespan. It combines the Gap-BIDE algorithm with a collaborative clustering technique. The Gap-BIDE algorithm is used to identify the most common patterns of behavior considering the elements of variations in activities sequence and the period of the day (i.e., night, morning, afternoon, and evening) for increased pattern mining flexibility. K-means and Hierarchical Clustering Agglomerative algorithms are collaboratively used to address the time-related elements of variability in daily routine like activities timespan vectors. A prototype was developed to monitor and detect the daily living activities based on smartwatch data using a deep learning architecture and the InceptionTime model, for which the highest accuracy was obtained. The results obtained are showing that the proposed solution can successfully identify the routines considering the aspects of flexibility such as activity sequences, optional and compulsory activities, timespan, and start and end time. The best results were obtained for the collaborative clustering solution that considers flexibility aspects in routine identification, providing coverage of monitored data of 89.63%.
... Nevertheless, the large size of the training data set (i.e., hourly readings for 365 days) used in [18][19][20][21] can slow down the deep neural network (DNN) training procedure, which can also become computationally costly. In [22], multimodal data fusion and adaptive deep learning were used to develop a cyberattack detection model in WDS, where various weighted channels of information were considered, and deep learning approach was used to estimate the weight of each channel. However, the best accuracy and performance levels achieved using the proposed DNN-based detection model in [22] were 87.62% and 85.41%, respectively, meaning that more than 10% of cyberattacks could potentially bypass the proposed model and damage the WDS. ...
... In [22], multimodal data fusion and adaptive deep learning were used to develop a cyberattack detection model in WDS, where various weighted channels of information were considered, and deep learning approach was used to estimate the weight of each channel. However, the best accuracy and performance levels achieved using the proposed DNN-based detection model in [22] were 87.62% and 85.41%, respectively, meaning that more than 10% of cyberattacks could potentially bypass the proposed model and damage the WDS. In [23], multilayer perception (MLP) and support vector machine (SVM) were used to predict the measurement parameters and to identify and classify the outliers in WDS. ...
... Training Size Performance Sensitivity [18] 365 days F1 = 0.897 High to θ [19] 365 days AUC = 0.953 High to noise [20] 365 days F1 = 0.806 NA [22] 365 days F1 = 0.852 NA Proposed 21 days F1 = 1.00 Low ...
Article
Full-text available
A cyberattack detection model based on supervised deep neural network is proposed to identify random false data injection (FDI) on the tank’s level measurements of a water distribution system. The architecture of the neural network, as well as various hyper-parameters, is modified and tuned to acquire the highest detection performance using the smallest size of training data set. The efficacy of the proposed detection model against various activation functions including sigmoid, rectified linear unit, and softmax is examined. Regularization and momentum techniques are applied to update the weights and prohibit overfitting. Moreover, statistical metrics are presented to evaluate the performance and effectiveness of the proposed model in the presence of a range of measurement noise levels. The proposed model is tested for three attack scenarios composed for the battle of the attack detection algorithms. Results confirm that the size of the data sets required to train the neural network (NN) to accomplish the highest levels of accuracy and precision is significantly decreased as the number of hidden layers is increased. The trained 4- and 5-layer deep neural networks are able to detect the readings’ FDIs with 100% precision and accuracy in the presence of 30% background noise in the sensory data.
... Bakalos et al. [94] developed a cyber-attack detection approach for water systems using multimodal data fusion and adaptive deep learning. Multimodal data fusion involves combining different channels of information, including visual data from thermal camera streams, Wi-Fi reflection, and ICS data. ...
... The weight attached to each of these streams of data is determined through a deep learning model process. The proposed adaptive deep learning approach uses a tapped delay line (TDL) convolutional neural network (CNN) with autoregressive moving average [94]. The data used to evaluate the approach is from STOP-IT project. ...
Article
Full-text available
Critical infrastructure systems are evolving from isolated bespoke systems to those that use general-purpose computing hosts, IoT sensors, edge computing, wireless networks and artificial intelligence. Although this move improves sensing and control capacity and gives better integration with business requirements, it also increases the scope for attack from malicious entities that intend to conduct industrial espionage and sabotage against these systems. In this paper, we review the state of the cyber-security research that is focused on improving the security of the water supply and wastewater collection and treatment systems that form part of the critical national infrastructure. We cover the publication statistics of the research in this area, the aspects of security being addressed, and future work required to achieve better cyber-security for water systems.
... Audebert [14] studied the use of a deep full convolutional neural network (DFCNN) in pixel-level scene markers of Earth observation images in the image field and achieved good experimental results. Bakalos et al. [15] used multi-modal data fusion and adaptive deep learning to monitor critical water infrastructure and also gained valuable application results. ...
... , w n }, we extract the word vector of the keyword t i as the attention matrix. The attention matrix s and the word vector matrix are subjected to the arithmetic operation shown in Equation (15), and the attention feature matrix A c can be obtained, wherein A c is a diagonal matrix. The operation process is shown in Figure 5: ...
Article
Full-text available
Convolutional neural networks (CNN) and long short-term memory (LSTM) have gained wide recognition in the field of natural language processing. However, due to the pre- and post-dependence of natural language structure, relying solely on CNN to implement text categorization will ignore the contextual meaning of words and bidirectional long short-term memory (BiLSTM). The feature fusion model is divided into a multiple attention (MATT) CNN model and a bi-directional gated recurrent unit (BiGRU) model. The CNN model inputs the word vector (word vector attention, part of speech attention, position attention) that has been labeled by the attention mechanism into our multi-attention mechanism CNN model. Obtaining the influence intensity of the target keyword on the sentiment polarity of the sentence, and forming the first dimension of the sentiment classification, the BiGRU model replaces the original BiLSTM and extracts the global semantic features of the sentence level to form the second dimension of sentiment classification. Then, using PCA to reduce the dimension of the two-dimensional fusion vector, we finally obtain a classification result combining two dimensions of keywords and sentences. The experimental results show that the proposed MATT-CNN+BiGRU fusion model has 5.94% and 11.01% higher classification accuracy on the MRD and SemEval2016 datasets, respectively, than the mainstream CNN+BiLSTM method.
... In fact, feedforward neural networks are capable of approximating nonlinear ARMA relationships, therefore improving the performance in TEC modeling. Nonlinear ARMA filters with recursive capabilities have been also proposed in [22] and [23]. Other works in this field apply radial basic function (RBF) models [24], with advance nonlinear interpolation capabilities, or support vector machines (SVMs) [25]. ...
... In (22), σ is the sigmoid function, and b 1 u and b 1 r are the respective biases of each component for the GRU. Variables W and U are the transition matrices of the lth GRU. ...
Article
Full-text available
Modeling ionospheric variability throughout a proper total electron content (TEC) parameter estimation is a demanding, however, crucial, process for achieving better accuracy and rapid convergence in precise point positioning (PPP). In particular, the single-frequency PPP (SF-PPP) method lacks accuracy due to the difficulty of dealing adequately with the ionospheric error sources. In order to apply ionosphere corrections in techniques, such as SF-PPP, external information of global ionosphere maps (GIMs) is crucial. In this article, we propose a deep learning model to efficiently predict TEC values and to replace the GIM-derived data that inherently have a global character, with equal or better in accuracy regional ones. The proposed model is suitable for predicting the ionosphere delay at different locations of receiver stations. The model is tested during different periods of time, under different solar and geomagnetic conditions and for stations in various latitudes, providing robust estimations of the ionospheric activity at the regional level. Our proposed model is a hybrid model comprising of a 1-D convolutional layer used for the optimal feature extraction and stacked recurrent layers used for temporal time series modeling. Thus, the model achieves good performance in TEC modeling compared to other state-of-the-art methods.
... The proposed Bayesian optimised bi-directional LSTM outperforms the other classifiers, providing reliably accurate space positioning which must be further exploited by the presented mobile application, thus expanding the potentials of indoor 3D cadastral mapping. For future work, more complex deep machine learning architectures such as multichannel recurrent neural networks (Kaselimi et al., 2019b) or deep NARMA filters (Bakalos, 2019) should be investigated. These structures, are capable of (i) processing simultaneously signals from heterogeneous sensors (e.g., Bluetooth and WiFi signals) in order to increase indoor localisation accuracy (multichannel RNNs) and (ii) of introducing an autoregressive behaviour to the LSTM neural network structure for further improvements in precision accuracy. ...
... In addition, as future work we can reduce the granularity of the targets positions labels in order to further improve the proposed architecture. Another extension is to incorporate in the sensing infrastructure, apart from Bluetooth sensors, Channel State Information of WiFi signals, which has been studied as an additional localisation modality (Bakalos, 2019). The next step of this research will be the integration between the developed indoor positioning system and the cadastral mobile application. ...
Article
Full-text available
With the rapid global urbanization, several multi-dimensional complex infrastructures have emerged, introducing new challenges in the management of the vertically stratified buildings spaces. 3D indoor cadastral spaces consist a zestful research topic as their complexity and geometry alterations during time, prevents the assignment of the corresponding Rights, Restrictions and Responsibilities (RRR). In the absence of the necessary horizontal spatial data infrastructure/floor plans their determination is weak. In this paper a fit-for-purpose technical framework and a crowdsourced methodology for the implementation of 3D cadastral surveys focused on indoor cadastral spaces, is proposed and presented. As indoor data capturing tool, an open-sourced cadastral mobile application for Android devices, is selected and presented. An Indoor Positioning System based on Bluetooth technology is established while an innovative machine learning architecture is developed, in order to explore its potentials to automatically provide the position of the mobile device within an indoor environment, aiming to add more intelligence to the proposed 3D crowdsourced cadastral framework. A practical experiment for testing the examined technical solution is conducted. The produced results are assessed to be quite promising.
... the CNN-XGBoost [34] with a XGBoost as classifier surpass our UDSOM model, so a UDSOM-XGBoost model integrating a trainable Deep UDSOM extractor to automatically obtain input functionality and XGBoost as a recognition module at the upper level of the network to produce results is an important future improvement of this work. Moreover, using multiple data modalities as proposed by Bakalos et al. [3] is good idea for a deep SOM feature extraction based a multiple data modality. ...
... Build N SOMs with each map focusing on modelling a local sub-region.3. Do Sub-sampling (Convolution and pooling). ...
Article
Full-text available
In this work we propose a new Unsupervised Deep Self-Organizing Map (UDSOM) algorithm for feature extraction, quite similar to the existing multi-layer SOM architectures. The principal underlying idea of using SOMs is that if a neuron is wins n times, these n inputs that activated this neuron are similar. The basic principle consists of an alternation of phases of splitting and abstraction of regions, based on a non-linear projection of high-dimensional data over a small space using Kohonen maps following a deep architecture. The proposed architecture consists of a splitting process, layers of alternating self-organizing, a rectification function RELU and an abstraction layer (convolution-pooling). The self-organizing layer is composed of a few SOMs with each map focusing on modelling a local sub-region. The most winning neurons of each SOM are then organized in a second sampling layer to generate a new 2D map. In parallel to this transmission of the winning neurons, an abstraction of the data space is obtained after the convolution-pooling module. The ReLU is then applied. This treatment is applied more than once, changing the size of the splitting window and the displacement step on the reconstructed input image each time. In this way, local information is gathered to form more global information in the upper layers by applying each time a convolution filter of the level. The architecture of the Unsupervised Deep Self-Organizing Map is unique and retains the same principle of deep learning algorithms. This architecture can be very interesting in a Big Data environment for machine learning tasks. Experiments have been conducted to discuss how the proposed architecture shows this performance.
... Actually, feedforward neural networks are capable of approximating non-linear ARMA relationships and therefore, improving performance in timeseries modeling. Non-linear ARMA filters with recursive capabilities have been also proposed in [32], [33]. Other works in this field apply Radial Basic Function (RBF) models [34], with advance non-linear interpolation capabilities, or Support Vector Machines (SVMs) [35]. ...
... One way to model the unknown function g(·) of Eq. (2) is by means of a Feedforward Neural Network [32]. Assuming L hidden neurons and one linear output layer, the estimate p j (n) is given byp ,1 · p(n)) . . . is omitted for simplicity. ...
Preprint
Full-text available
In this dissertation is provided a comparative analysis that evaluates the performance of several deep learning (DL) architectures on a large number of time series datasets of different nature and for different applications. Two main fruitful research fields are discussed here which were strategically chosen in order to address current cross disciplinary research priorities attracting the interest of geodetic community. The first problem is related to ionospheric Total Electron Content (TEC) modeling which is an important issue in many real time Global Navigation System Satellites (GNSS) applications. Reliable and fast knowledge about ionospheric variations becomes increasingly important. GNSS users of single frequency receivers and satellite navigation systems need accurate corrections to remove signal degradation effects caused by the ionosphere. Ionospheric modeling using signal processing techniques is the subject of discussion in the present contribution. The next problem under discussion is energy disaggregation which is an important issue for energy efficiency and energy consumption awareness. Reliable and fast knowledge about residential energy consumption at appliance level becomes increasingly important nowadays and it is an important mitigation measure to prevent energy wastage. Energy disaggregation or Nonintrusive load monitoring (NILM) is a single channel blind source separation problem where the task is to estimate the consumption of each electrical appliance given the total energy consumption. For both problems various deep learning models (DL) are proposed that cover various aspects of the problem under study, whereas experimental results indicate the proposed methods superiority compared to the current state of the art.
... In essence though the incident can be modelled as an abnormal behavior detection problem, where the normal situation consists of a normal capturing a seafaring vessel, while the abnormality would be the capturing of a fall. To this end, the main approaches for abnormal event recognition involve either the use of supervised deep learning techniques to learn a dictionary of abnormal sub-events or unsupervised outlier detection techniques. in many applications [7]- [9]. Examples include surveillance in industrial environments [7] or critical infrastructures [9] for safety/security and quality assurance, traffic flow management [10] and intelligent monitoring of public places [11] Regarding outlier detection, the works of [12], [[13], [14] learn dictionary of subevents, through a training process, and then those events that do not lie in the partitioned sub-space are marked as abnormal ones. ...
... To this end, the main approaches for abnormal event recognition involve either the use of supervised deep learning techniques to learn a dictionary of abnormal sub-events or unsupervised outlier detection techniques. in many applications [7]- [9]. Examples include surveillance in industrial environments [7] or critical infrastructures [9] for safety/security and quality assurance, traffic flow management [10] and intelligent monitoring of public places [11] Regarding outlier detection, the works of [12], [[13], [14] learn dictionary of subevents, through a training process, and then those events that do not lie in the partitioned sub-space are marked as abnormal ones. ...
Chapter
Man overboard incidents in a maritime vessel are serious accidents where the rapid detection of the even is crucial for the safe retrieval of the person. To this end, the use of deep learning models as automatic detectors of these scenarios has been tested and proven efficient, however, the use of correct capturing methods is imperative in order for the learning framework to operate well. Thermal data can be a suitable method of monitoring, as they are not affected by illumination changes and are able to operate in rough conditions, such as open sea travel. We investigate the use of a convolutional autoencoder trained over thermal data, as a mechanism for the automatic detection of man overboard scenarios. Morever, we present a dataset that was created to emulate such events and was used for training and testing the algorithm.
... In addition, non-causality issues are not addressed since the current models only "look" at previous and not at future choreographic patterns. These limitations are addressed by the paper by proposing a Bayesian optimized bi-directional long-range dependent deep learning [17], [18] model, called BOBi-LSTM, for pose identification in dance sequences. The model receives as inputs 3D kinematic descriptors of the skeleton human joints and provides an estimate of the main choreographic primitive at every image frame t. ...
... Therefore, a recurrent neural network can be used to characterize the system behavior in a non-parametric manner, especially when the complexity of the system obstructs the establishment of a physics-based model. To take into consideration network traffic data and the sensor data, in [18], the authors proposed a multi-model data fusion and adaptive deep learning method based on a convolutional neural network to characterize the normal system behavior. The framework then detects cyberattacks as well as physical intrusions in a single ICS pertaining to water infrastructure. ...
Preprint
Cybersecurity of Industrial Control Systems (ICS) is drawing significant concerns as data communication increasingly leverages wireless networks. A lot of data-driven methods were developed for detecting cyberattacks, but few are focused on distinguishing them from equipment faults. In this paper, we develop a data-driven framework that can be used to detect, diagnose, and localize a type of cyberattack called covert attacks on smart grids. The framework has a hybrid design that combines an autoencoder, a recurrent neural network (RNN) with a Long-Short-Term-Memory (LSTM) layer, and a Deep Neural Network (DNN). This data-driven framework considers the temporal behavior of a generic physical system that extracts features from the time series of the sensor measurements that can be used for detecting covert attacks, distinguishing them from equipment faults, as well as localize the attack/fault. We evaluate the performance of the proposed method through a realistic simulation study on the IEEE 14-bus model as a typical example of ICS. We compare the performance of the proposed method with the traditional model-based method to show its applicability and efficacy.
... Many of the aforementioned limitations can be addressed through the use of thermal cameras since human is a warmblooded organism, a property that distinguishes him from the environment around him in thermal images. Thus, their use is an important aspect of computer vision systems related to human detection [5] and furthermore, if the observation is made at a close distance, we are able to extract information related to skin temperature distribution, which may be critical in processes such as face recognition [6], [7]. ...
... A CNN can be used to estimate crowd density at railway stations [173],to detect intrusions in track areas, such as pedestrians or large livestock via images captured in railway areas [174], to monitor railway construction [152] and for intrusion detection at railway crossings [175]. From the security side, the method been used for detecting violent crowd flows [176], protect the critical infrastructure [177], and identifying tools wielding by attackers such as knives, guns and Explosives [178]. ...
Article
Full-text available
Railway stations are essential aspects of railway systems, and they play a vital role in public daily life. Various types of AI technology have been utilised in many fields to ensure the safety of people and their assets. In this paper, we propose a novel framework that uses computer vision and pattern recognition to perform risk management in railway systems in which a convolutional neural network (CNN) is applied as a supervised machine learning model to identify risks. However, risk management in railway stations is challenging because stations feature dynamic and complex conditions. Despite extensive efforts by industry associations and researchers to reduce the number of accidents and injuries in this field, such incidents still occur. The proposed model offers a beneficial method for obtaining more accurate motion data, and it detects adverse conditions as soon as possible by capturing fall, slip and trip (FST) events in the stations that represent high-risk outcomes. The framework of the presented method is generalisable to a wide range of locations and to additional types of risks.
... Recently, data-driven action recognition has become a popular research topic due to the flourish of deep learning [4][5][6][7][8]. Based on the types of input data, existing literature of action recognition can be divided into two categories: skeleton-based and image-based methods. ...
Article
Full-text available
Action recognition has gained great attention in automatic video analysis, greatly reducing the cost of human resources for smart surveillance. Most methods, however, focus on the detection of only one action event for a single person in a well-segmented video, rather than the recognition of multiple actions performed by more than one person at the same time for an untrimmed video. In this paper, we propose a deep learning-based multiple-person action recognition system for use in various real-time smart surveillance applications. By capturing a video stream of the scene, the proposed system can detect and track multiple people appearing in the scene and subsequently recognize their actions. Thanks to high resolution of the video frames, we establish a zoom-in function to obtain more satisfactory action recognition results when people in the scene become too far from the camera. To further improve the accuracy, recognition results from inflated 3D ConvNet (I3D) with multiple sliding windows are processed by a nonmaximum suppression (NMS) approach to obtain a more robust decision. Experimental results show that the proposed method can perform multiple-person action recognition in real time suitable for applications such as long-term care environments.
... The application layer is the interface of CPS for human service. The main function of this layer is to provide various CPS services for users, so that users can interact with the system without needing to know the following two layers in detail [10][11]. ...
Article
Full-text available
In the era of big data, the global data is growing explosively. The huge growth rate makes data processing and storage difficult, especially in the field of transportation. Based on the above background, this paper aims to study the autonomous coordinated control strategy for the complex process of traffic information physical fusion system based on big data. In this paper, the information physical fusion system is applied to the modern transportation system, and it is used to realize the high integration of computation, communication and control. Realize the independent and coordinated control of the transportation system. This paper proposes an autonomous traffic management mechanism based on multi-agent CPS system. In view of the instability and untimely of the original control strategy, a new traffic optimization control strategy conflict reduction control strategy is proposed. In order to solve the complexity of traffic system, the generation method of CPS autonomous control strategy based on multi-agent is studied and analyzed. Through the evaluation and verification of the conflict reduction control strategy and the online simulation of the incremental data synchronization strategy, it can be seen that the inconsistency ratio curves of message quantity and byte transmission quantity are always kept at a relatively low level, 1% and 2%, respectively. During the whole experiment, the average number of inconsistent messages and byte transmission of the agent are ideally controlled at 1.2 messages / train and 0.5kb/train.
... In the experiments, the proposed MARWIIoT (Genetic-KNN) model is compared with five machine learning techniques such as for the classification purpose. 10 As shown in Table 1, this matrix is known as a confusion matrix to assess the performance of a classification model for which true or false values have been applied on a series of test data. 25 Four essential parameters are True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). ...
Article
Full-text available
The Industrial Control System (ICS) has an important role to reduce the interaction between humans and the industrial system. Cyber‐Physical Systems (CPSs) operate in critical infrastructures such as transport networks, gas, and water distribution networks, Unmanned Aerial Vehicle Systems (UASs), and nuclear power generation. In this paper, we provide a Machine Learning Model for Malicious Activities Recognition in Water‐based Industrial Internet of Things (MARWIIoT). This model can help industrial operators and administrators to recognize any malicious activities against industrial infrastructure. To classify the malicious activity, various machine learning methods, such as Logistic Regression (LR), Linear Discriminant Analysis (LDA), k‐nearest neighbors (KNN), Naïve Bayes (NB), and Support Vector Machine (SVM) can be used. The proposed MARWIIoT model is based on Genetic‐KNN with normalization to classify malicious activities efficiently. This model has been evaluated using a dataset covering 15 anomaly circumstances, including normal system behavior. The circumstances existing addressed various incidents from hardware failure to critical water infrastructure damage. The comprehensive results have indicated that the MARWIIoT model is superlative in compared existing machine learning models because it has the highest performance based on different evaluation metrics such as accuracy, precision, recall, and F1‐Score.
... To achieve this correlation, there is a need to employ techniques that can extract features from multiple modalities, a task to which deep machine learning techniques have been employed successfully in the past (Voulodimos et al. 2018). Convolutional Neural Networks (CNN) have exhibited excellent feature extraction capabilities (LeCun et al. 2015), and some adaptations of CNN architectures have shown to be able to classify situations in highly dynamic environments Bakalos, Voulodimos, Doulamis, Doulamis, Ostfeld et al. 2019), using multiple data modalities. Moreover, in the simulation scenarios, low-level feature extraction over non-overlapping frame patches and density-based clustering will be used, which are techniques that have been used for real time analysis and classification of video data in previous research (see Papadakis et al. 2019). ...
Conference Paper
Full-text available
Increasing automation is ongoing in all areas of transport. This raises new challenges for the design and training of Human-Machine Interfaces (HMI) for different user groups. The EU-project Drive2theFuture investigates the needs and wants of transportation users, operators, passengers and passersby to gain their acceptance and to set the ground for a sustainable market introduction of automated transport. This paper describes how HMI concepts for the transport modes road, rail, maritime and aviation in Drive2theFuture are developed and comparatively assessed in order to be able to support an educated use of automated transport. By relying on a stepwise process, adaptable HMI strategies for different user clusters and levels of automation are defined. As a universal method, a comprehensive HMI development toolkit is developed, which can be adopted as training tool to create realistic expectations and enhance acceptance among users, operators and drivers in light of the deployment of automated vehicles.
... 3) η εφαρμογή ημι-επιβλεπόμενων ή/και μη-επιβλεπόμενων τεχνικών βαθιάς μηχανικής μάθησης, και 4) ο συνδυασμός δεδομένων από ποικίλες πηγές και δέκτες (multimodal data fusion) [459]. ...
... The separated channel-spatial convolution can be closely combined with separated channel-spatial attention blocks. Finally, the two kinds of information is fused to generate a reasonable combination of features [6]. As a result, the encoder can extract features with stronger spatial expression ability to reconstruct higher-quality 3D shapes by using the decoder. ...
Article
3D object reconstruction is a challenging problem in computer vision, especially, the single-view reconstruction. In this paper, we proposed a new 3D reconstruction network, termed as separated channel-spatial convolution net with attention (SCSCN), which can reconstruct the 3D shape of objects by given a 2D image from any viewpoint. Our method is a simple encoder-decoder structure, where the encoder uses separated channel-spatial convolution and separated channel-spatial attention to extract features from the input image, and the decoder recovers 3D shapes from the features. The separated channel-spatial convolution can obtain channel information and spatial information through the channel path and spatial path separately. At the same time, in order to select a more reasonable combination of features according to the degree of contribution to the reconstruction task, channel attention and spatial attention are relevantly inserted into these two paths. As a result, the encoder can extract a strong representation of object. Quantitative experiments show that our SCSCN has a weak dependence on 3D supervision and achieves high-quality reconstruction just under 2D supervision, which proves the effectiveness of the encoder. In addition, we conducted the qualitative visualization experiment to confirm the rationality of the attention blocks in the feature extraction process.
... These models, receive as inputs, instead of features, the raw sensory data, thus obviating the need for feature extraction and avoiding the drawbacks involved in this process [5][6][7][8]. However, a typical deep learning architecture [9] contains a huge number of trainable parameters implying that a large number of samples is also needed to accurately train the classifier. ...
... In this work, Hidden Markov models are used to support the filters for understanding the behaviors at an industrial plant. Bakalos et al. [8], used adaptive deep learning and multimodal fused data to monitor critical systems for the protection of water infrastructure from different types of attacks. It used visual surveillance, ICS sensor data, and channel state information (CSI) to detect different attacks including cyberattacks and human presence assaults. ...
Article
Full-text available
Recent advances in visionary technologies impacted multi-object recognition and scene understanding. Such scene-understanding tasks are a demanding part of several technologies such as augmented reality based scene integration, robotic navigation, autonomous driving and tourist guide applications. Incorporating visual information in contextually unified segments, super-pixel-based approaches significantly mitigate the clutter, which is normal in pixel wise frameworks during scene understanding. Super-pixels allow customized shapes and variable size patches of connected components to be obtained. Furthermore, the computational time for these segmentation approaches can significantly decreased due to the reduced number of super-pixel target clusters. Hence, the super pixel-based approaches are more commonly used in robotics, computer vision and other intelligent systems. In this paper, we propose a Maximum Entropy scaled Super-Pixels (MEsSP) Segmentation method that encapsulates super-pixel segmentation based on an Entropy Model and utilizes local energy terms to label the pixels. Initially, after acquisition and pre-processing, image is segmented by two different methods: Fuzzy C-Means (FCM) and MEsSP. Then, to extract the features from these segmented objects, the dynamic geometrical features, fast Fourier transform (FFT), blob extraction, Maximally Stable Extremal Regions (MSER) and KAZE features are extracted using the bag of features approach. Then, to categorize the objects, multiple kernel learning is applied. Finally, a deep belief network (DBN) assigns the relevant labels to the scenes based on the categorized objects, intersection over union scores and dice similarity coefficient. The experimental results regarding multiple objects recognition accuracy, precision, recall and F1 scores over PASCAL VOC, Caltech 101 and UIUC Sports datasets show a remarkable performance. In addition, the evaluation of proposed scene recognition method over these benchmark datasets outperforms the state of the art (SOTA) methods.
... which an observation belongs, based on the set of features (explanatory variables). A training dataset of observations with known class membership is typically available, so classification is a special case of supervised learning [3,9]. A classification problem is categorized into (a) Binary classification (the class label takes only two values), (b) Multi-class classification (the class label takes more than two values) and (c) Multi-label classification (each observation is associated with multiple classes). ...
... Muzammal et al. [13] proposed a mathematical model based on a multisensor data fusion algorithm. Bakalos et al. [14] used multimodal data fusion and adaptive deep learning to monitor critical systems. Zhang et al. [15] proposed a method based on multisensor data fusion for UAV safety distance diagnosis. ...
Article
Full-text available
Optical sensor data fusion technology is a research hotspot in the field of information science in recent years, which is widely used in military and civilian fields because of its advantages of high accuracy and low cost, and target recognition is one of the important research directions. Based on the characteristics of small target optical imaging, this paper fully utilizes the frontier theoretical methods in the field of image processing and proposes a small target recognition algorithm process framework based on visible and infrared image data fusion and improves the accuracy as well as stability of target recognition by improving the multisensor information fusion algorithm in the photoelectric meridian tracking system. A practical guide is provided for the solution of the small target recognition problem. To facilitate and quickly verify the multisensor fusion algorithm, a simulation platform for the intelligent vehicle and the experimental environment is built based on Gazebo software, which can realize the sensor data acquisition and the control decision function of the intelligent vehicle. The kinematic model of the intelligent vehicle is firstly described according to the design requirements, and the camera coordinate system, LiDAR coordinate system, and vehicle body coordinate system of the sensors are established. Then, the imaging models of the depth camera and LiDAR, the data acquisition principles of GPS and IMU, and the time synchronization relationship of each sensor are analyzed, and the error calibration and data acquisition experiments of each sensor are completed.
... Based on multimode and high-dimensional sensor data influx, Nweke et al. f conducted in-depth summary of human activity recognition methods using mobile and wearable sensors [7]. Based on multimode data fusion, Bakalos et al. studied the channel status information of visual monitoring, Wi-Fi signals, and utilities from ICS sensor data [8]. Du and Zare proposed a novel multi-instance multiresolution fusion framework. ...
Article
Full-text available
With the progress of society and the improvement of economic level, the traditional manufacturing industry is gradually transformed and upgraded in the changing times. In addition to the need for innovation in production technology, product design also plays a vital role in the sustainable development of the entire industry. However, the existing design methods have been unable to meet the high-quality requirements of consumers for material life. Therefore, this paper integrates multimodal sensor data fusion and transmission into product design. The multimodal sensor data fusion and transmission technology is developed on the basis of the human organs’ ability to comprehensively judge the environmental information. This paper gives a brief overview of product design methods on the basis of this theory. Among them, the generation, development, classification, attribute level, and emotional level of products of product design methods are studied. And through the research on multimodal data fusion algorithm, the design method based on multimodal sensor data fusion and transmission technology is designed and practiced, and compared with the traditional QFD technology design method. The experimental results show that the design method proposed in this paper has the highest market demand adaptability of 86.37%, which proves its effectiveness. Popularizing this technology and method in the current stage of manufacturing industry product design can effectively improve the quality and efficiency of the design process. This has practical significance for enhancing the core competitiveness of products in the market.
... The paper also considers that adverse events suddenly occur and evolve rapidly, giving little time to react. In [30] visual surveillance data, channel state information from Wi-Fi signals for human-presence detection, and ICS sensor data from the utility are analyzed for protecting critical water infrastructures. In [31] wired signal distinct native attribute finger-printing is investigated as a non-intrusive physical-based security augmentation. ...
Article
Full-text available
Critical infrastructures are assets of invaluable importance, essential for the whole world. Since they serve core functions of our societies, they often become targets of terrorists. Many critical infrastructures are vulnerable, due to their short distance from public roads and in the past years, several vehicle-bomb incidents have been recorded. This paper focuses on the case of truck-bombs, which can either be created from scratch, or terrorists can easily hijack truck cargos carrying dangerous goods and turn them into bombs. The latter are typically called ADR truck cargos, according to the respective agreement of the 30th of September 1957, concerning the international carriage of dangerous goods by road. The proposed scheme performs threat assessment of neighboring critical infrastructures, aiming at preventing explosions of truck-bombs. To do so, each crucial point of a critical infrastructure is initially associated with a level of importance. Next, three scenarios are analyzed: (a) single-attack single-infrastructure, (b) multiple-attack single-infrastructure, and (c) multiple-attack multiple-infrastructure. To reduce computational complexity, the third scenario is simplified to one of the two other scenarios, by introducing a novel fusion technique for the non-overlapping segments of the Voronoi tessellation. By this way, an area of threat assessment is estimated for each critical infrastructure. Then, the threat level is assessed in real time by an innovative algorithm, which: (a) estimates the impact of multiple consecutive explosions, (b) uses five adapted threat levels and (c) introduces multiple criteria and minimum classification conditions based on the number of crucial points and their levels of importance. Extensive real world experimental results and comparisons to other works, exhibit the pros and cons of the proposed scheme. In particular, compared to related work, the proposed scheme improves: (a) computational time by 74.5%, (b) threat notification time by 86.9% and (c) estimated surveillance cost by 98.6%.
... Future work will concentrate on maximizing the performance of our anomaly detection scheme by utilizing additional information modalities, such as thermal imaging data and radar signals, as well as multimodal information fusion techniques for the efficient automated recognition of the man overboard event [73]. Recent studies have underlined that the use of thermal sensors is a crucial factor in various computer vision surveillance systems, since humans are warm-blooded organisms, a property that distinguishes them from their environment in thermal imagery [74]. ...
Article
Full-text available
Man overboard is an emergency in which fast and efficient detection of the critical event is the key factor for the recovery of the victim. Its severity urges the utilization of intelligent video surveillance systems that monitor the ship’s perimeter in real time and trigger the relative alarms that initiate the rescue mission. In terms of deep learning analysis, since man overboard incidents occur rarely, they present a severe class imbalance problem, and thus, supervised classification methods are not suitable. To tackle this obstacle, we follow an alternative philosophy and present a novel deep learning framework that formulates man overboard identification as an anomaly detection task. The proposed system, in the absence of training data, utilizes a multi-property spatiotemporal convolutional autoencoder that is trained only on the normal situation. We explore the use of RGB video sequences to extract specific properties of the scene, such as gradient and saliency, and utilize the autoencoders to detect anomalies. To the best of our knowledge, this is the first time that man overboard detection is made in a fully unsupervised manner while jointly learning the spatiotemporal features from RGB video streams. The algorithm achieved 97.30% accuracy and a 96.01% F1-score, surpassing the other state-of-the-art approaches significantly.
... which an observation belongs, based on the set of features (explanatory variables). A training dataset of observations with known class membership is typically available, so classification is a special case of supervised learning [3,9]. A classification problem is categorized into (a) Binary classification (the class label takes only two values), (b) Multi-class classification (the class label takes more than two values) and (c) Multi-label classification (each observation is associated with multiple classes). ...
... where r refers to coordinate transformation distance; θ refers to coordinate transformation angle; ω refers to rectangular area's scale; r ′ refers to the normalized value of coordinate transformation distance; and θ ′ refers to the normalized value of coordinate transformation angle [17,18]. rough R(x, y), filter waves of R(r, θ). e local subblock is qualitatively described as 1 and 0. e single-modal biometrics are described through the description of the local sub-block, and the corresponding multibit feature codes of various single-modal are created to realize the extraction of multimodal biometrics [19,20]. ...
Article
Full-text available
Multimodal biometrics fusion plays an important role in the field of biometrics. Therefore, this paper presents a multimodal biometrics fusion algorithm using deep reinforcement learning. In order to reduce the influence of user behavior, user’s personal characteristics, and environmental light on image data quality, data preprocessing is realized through data transformation and single-mode biometric image region segmentation. A two-dimensional Gobar filter was used to analyze the texture of local sub-blocks, qualitatively describe the similarity between the filter and the sub-blocks and extract the phase information and local amplitude information of multimodal biometrics features. Deep reinforcement learning was used to construct the classifier of different modal biometrics, and the weighted sum fusion of different modal biometrics was implemented by fractional information. The multimodal biometrics fusion algorithm was designed. The Casia-iris-interval-v4 and NFBS datasets were used to test the performance of the proposed algorithm. The results show that the fused image quality is better, the feature extraction accuracy is between 84% and 93%, the average accuracy of feature classification is 97%, the multimodal biometric classification time is only 110 ms, the multimodal biometric fusion time is only 550 ms, the effect is good, and the practicability is strong.
... Defenses for attacks on ICS hardware have also been studied extensively, among others these include protecting the PLC using control invariants (correlation between sensor readings and PLC commands) [44], extracting control logic rules [16] and detecting safety violations [26]. Similarly, Machine Learning-based (ML) solutions [20,4,27,30,43] have been employed for detecting FDI attacks. ...
Preprint
Full-text available
Industrial Control Systems (ICS) have played a catalytic role in enabling the 4th Industrial Revolution. ICS devices like Programmable Logic Controllers (PLCs), automate, monitor and control critical processes in industrial, energy and commercial environments. The convergence of traditional Operational Technology (OT) with Information Technology (IT) has opened a new and unique threat landscape. This has inspired defense research that focuses heavily on Machine Learning (ML) based anomaly detection methods that run on external IT hardware which means an increase in costs and the further expansion of the threat landscape. To remove this requirement, we introduce the ICS Machine Learning inference framework (ICSML) which enables the execution of ML models natively on the PLC. ICSML is implemented in IEC 61131-3 code and works around the limitations imposed by the domain-specific languages, providing a complete set of components for the creation of fully fledged ML models in a way similar to established ML frameworks. We then demonstrate a complete end-to-end methodology for creating ICS ML models using an external framework for training and ICSML for the PLC implementation. To evaluate our contributions we run a series of benchmarks studying memory and performance and compare our solution to the TFLite inference framework. Finally, to demonstrate the abilities of ICSML and to verify its non-intrusive nature, we develop and evaluate a case study of a real defense for process aware attacks against a Multi Stage Flash (MSF) desalination plant.
... Literature [17] obtained candidate samples by scattering particles and obtained the scores of these particle samples through the twin network, and the final tracking result was the highest among the particle scores. Literature [18] uses the convolution 4-3 layer and convolution 5-3 layer of two VGG networks to calculate the final response graph. Literature [19] uses the last layer of the network to generate a heat map for target tracking. ...
Article
Full-text available
With the advent of the artificial intelligence era, target adaptive tracking technology has been rapidly developed in the fields of human-computer interaction, intelligent monitoring, and autonomous driving. Aiming at the problem of low tracking accuracy and poor robustness of the current Generic Object Tracking Using Regression Network (GOTURN) tracking algorithm, this paper takes the most popular convolutional neural network in the current target-tracking field as the basic network structure and proposes an improved GOTURN target-tracking algorithm based on residual attention mechanism and fusion of spatiotemporal context information for data fusion. The algorithm transmits the target template, prediction area, and search area to the network at the same time to extract the general feature map and predicts the location of the tracking target in the current frame through the fully connected layer. At the same time, the residual attention mechanism network is added to the target template network structure to enhance the feature expression ability of the network and improve the overall performance of the algorithm. A large number of experiments conducted on the current mainstream target-tracking test data set show that the tracking algorithm we proposed has significantly improved the overall performance of the original tracking algorithm.
... Moreover, it is critical in the search for lost people in huge crowds. The mentioned vital applications are indispensable for enhanced public safety and security [3,4]. As a result, the person re-ID problem in computer vision and machine learning has been receiving growing importance and attention in the recent past [5][6][7][8][9]. ...
Article
Full-text available
Person Re-Identification is an essential task in computer vision, particularly in surveillance applications. The aim is to identify a person based on an input image from surveillance photographs in various scenarios. Most Person re-ID techniques utilize Convolutional Neural Networks (CNNs); however, Vision Transformers are replacing pure CNNs for various computer vision tasks such as object recognition, classification, etc. The vision transformers contain information about local regions of the image. The current techniques take this advantage to improve the accuracy of the tasks underhand. We propose to use the vision transformers in conjunction with vanilla CNN models to investigate the true strength of transformers in person re-identification. We employ three backbones with different combinations of vision transformers on two benchmark datasets. The overall performance of the backbones increased, showing the importance of vision transformers. We provide ablation studies and show the importance of various components of the vision transformers in re-identification tasks. .
... It addresses the construction of a system to detect unknown anomalies (not based on heuristic tools, lists, or threats already detected) using different sources of information, with automatic learning abilities, and with the supervision of a specialist to validate complex threats to be included in the knowledge base of the system [37]. Context analysis will include interdependencies with other infrastructures (ICT networks, power supply, etc.), social networks, or information that may directly affect its security and resilience. ...
Article
Full-text available
Water CIs are exposed to a wide number of IT challenges that go from the cooperation and alignment between physical and cyber security teams to the proliferation of new vulnerabilities and complex cyber-attacks with potential disastrous consequences. Although novel and powerful solutions are proposed in the literature, most of them lack appropriate mechanisms to detect cyber and physical attacks in real time. We propose a Cross-Layer Analytic Platform (denoted as CLAP) developed for the correlation of Cyber and Physical security events affecting water CIs. CLAP aims to improve the detection of complex attack scenarios in real time based on the correlation of cyber and physical security events. The platform assigns appropriate severity values to each correlated alarm that will guide security analysts in the decision-making process of prioritizing mitigation actions. A series of passive and active attack scenarios against the target infrastructure are presented at the end of the paper to show the mechanisms used for the detection and correlation of cyber–physical security events. Results show promising benefits in the improvement of response accuracy, false rates reduction and real-time detection of complex attacks based on cross-correlation rules.
... In addition, there are other wider contexts of anomaly detection. For example, Bakalos et al. [40] proposed an approach to detect abnormalities involved in various forms of attacks on water infrastructure. They proposed a framework based on multimodal data fusion and adaptive deep learning for the purpose. ...
Article
Full-text available
The continuous development of intelligent video surveillance systems has increased the demand for enhanced vision-based methods of automated detection of anomalies within various behaviors found in video scenes. Several methods have appeared in the literature that detect different anomalies by using the details of motion features associated with different actions. To enable the efficient detection of anomalies, alongside characterizing the specificities involved in features related to each behavior, the model complexity leading to computational expense must be reduced. This paper provides a lightweight framework (LightAnomalyNet) comprising a convolutional neural network (CNN) that is trained using input frames obtained by a computationally cost-effective method. The proposed framework effectively represents and differentiates between normal and abnormal events. In particular, this work defines human falls, some kinds of suspicious behavior, and violent acts as abnormal activities, and discriminates them from other (normal) activities in surveillance videos. Experiments on public datasets show that LightAnomalyNet yields better performance comparative to the existing methods in terms of classification accuracy and input frames generation.
Article
Energy disaggregation, or Non-Intrusive Load Mon-itoring (NILM), describes various processes aiming to identify the individual contribution of appliances, given the aggregate power signal. In this paper, a non-causal adaptive context-aware bidirec-tional deep learning model for energy disaggregation is intro-duced. The proposed model, CoBiLSTM, harnesses the represen-tational power of deep recurrent Long Short-Term Memory (LSTM) neural networks, while fitting two basic properties of NILM problem which state of the art methods do not appropri-ately account for: non-causality and adaptivity to contextual fac-tors (e.g. seasonality). A Bayesian-optimized framework is intro-duced to select the best configuration of the proposed regression model, driven by a self-training adaptive mechanism. Further-more, the proposed model is structured in a modular way to ad-dress multi-dimensionality issues that arise when the number of appliances increases. Experimental results indicate the proposed method’s superiority compared to the current state of the art.
Article
Full-text available
This article presents the evolving challenges and roles of our water resources in this contemporary world. First, water quality issues surrounding water supplies are discussed. Potential pathways to address the water quality challenges are presented, which include technological approaches for minimizing waste and enhancing resource recovery. Focused discussions on emerging global pollutants such as microplastics and PFAS (per- and poly-fluoro alkyl substances) and treatment alternatives are included. Next, the roles of used water (wastewater) in the wake of circular economy and recent outbreaks are discussed. The potential for energy and resource recovery possibilities and the critical role of wastewater treatment plants in controlling the spread of outbreaks are discussed in detail. Finally, perspectives on some of the key developments essential for transforming our water infrastructure, addressing water-centered socio-economic issues and the critical needs of digitalization in water sector operations are presented. HIGHLIGHTS Roles of water and wastewater in circular economy and community resiliency are discussed.; Microplastics and PFAS and emerging issues in water and wastewater industry are discussed.; Critical need for digitalization, data control and infrastructure investment are emphasized.;
Article
Full-text available
An increase in the number and strength of natural catastrophes experienced over the past few decades has accelerated the damage sustained by infrastructures. Drinking water and wastewater infrastructure systems are critical aspects of a healthy environment, and their ability to withstand disasters is vital for effective disaster response and recovery. Although numerous studies have been conducted to determine the challenges that natural disasters render to water infrastructures, few extensive examinations of these challenges have been conducted. The goal of this study, therefore, was to identify and categorize the challenges related to the resilience of drinking water and wastewater infrastructures, and to determine the strategies that most effectively minimize their unintended consequences. A comprehensive evaluation of the existing literature was conducted, and 537 publications were collected. After extensive screening, 222 publications were selected for rigorous evaluation and analysis based on the data collection methods and other criteria. A total of fifty-one (51) challenges were determined and classified, within the following five categories: environmental, technical and infrastructure, social, organizational, and financial and economic. The challenges were then ranked within each category according to their frequency of occurrence in previous research. The results reveal that climate change, aging infrastructure, lack of infrastructure capital, population growth, improper maintenance of water infrastructure, and rapid urbanization are the most frequently cited challenges. Next, 30 strategies and approaches were identified and categorized into either preventive or corrective actions, according to their implementation time. The findings of this study will help decision- and policymakers properly allocate their limited funding to enhance the robustness of their water infrastructures before, during, and after natural hazards.
Article
Full-text available
Hyperspectral image classification has drawn significant attention in the recent years driven by the increasing abundance of sensor-generated hyper- and multi-spectral data, combined with the rapid advancements in the field of machine learning. A vast range of techniques, especially involving deep learning models, have been proposed attaining high levels of classification accuracy. However, many of these approaches significantly deteriorate in performance in the presence of noise in the hyperspectral data. In this paper, we propose a new model that effectively addresses the challenge of noise residing in hyperspectral images. The proposed model, which will be called DCT-CNN, combines the representational power of Convolutional Neural Networks with the noise elimination capabilities introduced by frequency-domain filtering enabled through the Discrete Cosine Transform. In particular, the proposed method entails the transformation of pixel macroblocks to the frequency domain and the discarding of information that corresponds to the higher frequencies in every patch, in which pixel information of abrupt changes and noise often resides. Experiment results in Indian Pines, Salinas and Pavia University datasets indicate that the proposed DCT-CNN constitutes a promising new model for accurate hyperspectral image classification offering robustness to different types of noise, such as Gaussian and salt and pepper noise.
Article
On the one hand, smart cities have brought about various changes, aiming to revolutionize people's lives. On the other hand, while smart cities bring better life experiences and great convenience to people's lives, there are more hidden dangers of cyber security, including information leakage and malicious cyber attacks. The current cyber security development cannot keep up with the eager adoption of global smart city technologies so correct design based on deep learning methods is essential to protect smart city cyber. This paper summarizes the knowledge and interpretation of Smart Cities (SC), Cyber Security (CS), and Deep Learning (DL) concepts as well as discussed existing related work on IoT security in smart cities. Specifically, we briefly reviewed several deep learning models, including Boltzmann machines, restricted Boltzmann machines, deep belief networks, recurrent neural networks, convolutional neural networks, and generative adversarial networks. Then we introduced cyber security applications and use cases based on deep learning technology in smart cities. Finally, we describe the future development trend of smart city cyber security.
Article
Full-text available
The article is devoted to an urgent problem related to the security of critical information infrastructure. The introduction of digital technologies in all spheres of society is in line with the priority policy for the development of the digital industry. At the same time, the number and quality of cyberattacks on significant objects of critical information infrastructure is constantly increasing in the world. But not all subjects of information relations, even understanding the presence of threats, are able to adequately assess and organize an effective security system for these objects. In this regard, ensuring the security of significant objects of critical information infrastructure is currently the primary task of the state - both the Russian Federation and other countries. The article provides a comparative analysis of approaches to ensuring the security of critical information infrastructure in Russia and in foreign countries. The problems of legal regulation of critical information infrastructure in Russia are identified and solutions are proposed to overcome them. There are traced the shortcomings associated with the implementation of legislation on the security of critical information infrastructure in Russia: in the digital industry, it becomes difficult to differentiate information infrastructure objects and classify some of them as critical; not all relevant legal entities have provided information on critical information infrastructure facilities, and therefore the register of facilities was not compiled in full, cyberattacks on which would create dangerous consequences for the country; some subjects of the critical information infrastructure deliberately underestimate the importance of their objects.
Article
Demand for security, efficiency, and environmental regulations have led to rapid deployment of Cyber-Physical Systems (CPSs), which are now an integral part of modern industries. Wireless Sensor Networks (WSNs), which provide distributed intelligent devices for monitoring physical or environmental conditions, are a hallmark of CPSs. CPSs often consist of distributed devices such as sensors and actuators to connect with and influence the physical layer system. Commonly, CPSs are monitored by Supervisory Control And Data Acquisition (SCADA) systems. All the benefits gained by integrating the cyber and physical worlds assist in operating CPSs more effectively and efficiently. However, the gains obtained through deployment of CPS come with the price that CPSs are vulnerable to cyberattacks. This article starts with the definitions and introduction to fundamental concepts in CPSs. Following that, various types of cyber-attacks in a CPS are illustrated, and a detailed literature review of control-oriented attack detectors is presented. Afterward, prominent examples of CPSs and their attack detectors are demonstrated. The article concludes with a discussion of certain future research directions in this critical and challenging area.
Article
Full-text available
Water distribution systems are one of the critical infrastructures and major assets of the water utility in a nation. The infrastructure of the distribution systems consists of resources, treatment plants, reservoirs, distribution lines, and consumers. A sustainable water distribution network management has to take care of accessibility, quality, quantity, and reliability of water. As water is becoming a depleting resource for the coming decades, the regulation and accounting of the water in terms of the above four parameters is a critical task. There have been many efforts towards the establishment of a monitoring and controlling framework, capable of automating various stages of the water distribution processes. The current trending technologies such as Information and Communication Technologies (ICT), Internet of Things (IoT), and Artificial Intelligence (AI) have the potential to track this spatially varying network to collect, process, and analyze the water distribution network attributes and events. In this work, we investigate the role and scope of the IoT technologies in different stages of the water distribution systems. Our survey covers the state-of-the-art monitoring and control systems for the water distribution networks, and the status of IoT architectures for water distribution networks. We explore the existing water distribution systems, providing the necessary background information on the current status. This work also presents an IoT Architecture for Intelligent Water Networks - IoTA4IWNet, for real-time monitoring and control of water distribution networks. We believe that to build a robust water distribution network, these components need to be designed and implemented effectively.
Article
Full-text available
Over the last years deep learning methods have been shown to outperform previous state-of-the-art machine learning techniques in several fields, with computer vision being one of the most prominent cases. This review paper provides a brief overview of some of the most significant deep learning schemes used in computer vision problems, that is, Convolutional Neural Networks, Deep Boltzmann Machines and Deep Belief Networks, and Stacked Denoising Autoencoders. A brief account of their history, structure, advantages, and limitations is given, followed by a description of their applications in various computer vision tasks, such as object detection, face recognition, action and activity recognition, and human pose estimation. Finally, a brief overview is given of future directions in designing deep learning schemes for computer vision problems and the challenges involved therein.
Article
Full-text available
Detection of moving objects in videos is a crucial step towards successful surveillance and monitoring applications. A key component for such tasks is called background subtraction and tries to extract regions of interest from the image background for further processing or action. For this reason, its accuracy and real-time performance is of great significance. Although, effective background subtraction methods have been proposed, only a few of them take into consideration the special characteristics of thermal imagery. In this work, we propose a background subtraction scheme, which models the thermal responses of each pixel as a mixture of Gaussians with unknown number of components. Following a Bayesian approach, our method automatically estimates the mixture structure, while simultaneously it avoids over/under fitting. The pixel density estimate is followed by an efficient and highly accurate updating mechanism, which permits our system to be automatically adapted to dynamically changing operation conditions. We propose a reference implementation of our method in reconfigurable hardware achieving both adequate performance and low power consumption. Adopting a High Level Synthesis design, demanding floating point arithmetic operations are mapped in reconfigurable hardware; demonstrating fast-prototyping and on-field customization at the same time.
Conference Paper
Full-text available
To diminish the limitations of the visual and thermal face images, a fusion of them is carried out in order to upgrade the performance. This paper presents fusion based on maximum selection of DB4 wavelet coefficients of visual and thermal images. The dimensions of these fused images are reduced using ICA and these low dimension fused images are classified with the help of SVM. The experimental result of fusion gives approximately 2% and 6% more accuracy in case of expression and illumination IRIS datasets than the visual and thermal images respectively. Further this work is extended with a comparative study of fusion methods based some other wavelet decompositions such as Haar/DB2, Coiflets, and Symlets. In addition, a comparison between two ICA architectures and PCA is also carried out over wavelet fusion methods. It has been noticed that maximum information, classification accuracy, and less error acheieved for haar/db2 fused images.
Conference Paper
Full-text available
Channel State Information (CSI) obtained from commercial Wi-Fi chipsets has proven to be efficient in detecting human interaction with radio waves. However, there is a lack of analytical modelling to define the impact of human presence on multidimensional CSI vectors. Existing approaches include linear, parameter-less techniques to reduce signal space dimensions and filter noise by assuming linear correlations among sub-carriers. In this work, we first model the human presence and then analyse occurrences of non-linear correlations among sub-carriers. We then exploit these correlations by introducing non-linear techniques to reduce CSI dimensions and filter noise. These techniques offer adjustable parameters that enhance signal quality depending on the environment and the amount of human interaction. We analyse the performance of human presence detection using the introduced techniques with just two transceivers. Our results show that when human motion is insignificant or occurs far from the link, non-linear techniques improve the detection accuracy up to 5% compared to the linear approach.
Conference Paper
Full-text available
This paper presents a novel video descriptor, referred to as Histogram of Oriented Tracklets, for recognizing abnormal situation in crowded scenes. Unlike standard approaches that use optical flow, which estimates motion vectors only from two successive frames, we built our de-scriptor over long-range motion trajectories which is called tracklets in the literature. Following the standard procedure , we divided video sequences in spatio-temporal cuboids within which we collected statistics on the tracklets passing through them. In particular, we quantized orientation and magnitude in a 2-dimensional histogram which encodes the motion patterns expected in each cuboid. We classify frames as normal and abnormal by using Latent Dirichlet Allocation and Support Vector Machines. We evaluated the effectiveness of the proposed descriptors on three datasets: UCSD, Violence in Crowds and UMN. The experiments demonstrated (i) very promising results in abnormality detection, (ii) setting new state-of-the-art on two of them, and (iii) outperforming former descriptors based on the optical flow, dense trajectories and the social force model.
Article
Full-text available
This paper investigates the problem of detection and isolation of attacks on a water distribution network comprised of cascaded canal pools. The proposed approach employs a bank of delay-differential observer systems. The observers are based on an analytically approximate model of canal hydrodynamics. Each observer is insensitive to one fault/attack mode and sensitive to other modes. The design of the observers is achieved by using a delay-dependent linear matrix inequality method. The performance of our model-based diagnostic scheme is tested on a class of adversarial scenarios based on a generalized fault/attack model. This model represents both classical sensor-actuator faults and communication network-induced deception attacks. Our particular focus is on stealthy deception attacks in which the attacker's goal is to pilfer water through canal offtakes. Our analysis reveals the benefits of accurate hydrodynamic models in detecting physical faults and cyber attacks to automated canal systems. We also comment on the criticality of sensor measurements for the purpose of detection. Finally, we discuss the knowledge and effort required for a successful deception attack.
Article
Full-text available
Indoor positioning systems have received increasing attention for supporting location-based services in indoor environments. WiFi-based indoor localization has been attractive due to its open access and low cost properties. However, the distance estimation based on received signal strength indicator (RSSI) is easily affected by the temporal and spatial variance due to the multipath effect, which contributes to most of the estimation errors in current systems. How to eliminate such effect so as to enhance the indoor localization performance is a big challenge. In this work, we analyze this effect across the physical layer and account for the undesirable RSSI readings being reported. We explore the frequency diversity of the subcarriers in OFDM systems and propose a novel approach called FILA, which leverages the channel state information (CSI) to alleviate multipath effect at the receiver. We implement the FILA system on commercial 802.11 NICs, and then evaluate its performance in different typical indoor scenarios. The experimental results show that the accuracy and latency of distance calculation can be significantly enhanced by using CSI. Moreover, FILA can significantly improve the localization accuracy compared with the corresponding RSSI approach.
Article
Full-text available
In this paper, we propose a novel system for visual recognition and summarization of pick and place tasks, that may be executed in settings such as an industrial assembly line. Our novel approach is based on the utilization of hidden Markov models for online task recognition as well as on the use of prior knowledge via a Hopfield-based optimization scheme. To facilitate offline analysis we extract summaries of the captured content based on these features. We extract the motion energy using the norms of the Zernike moments, looking for local minima and maxima that indicate distinctive visual events and thus key-frames. The proposed scheme is not threshold-dependent and therefore the number of extracted key-frames varies according to the complexity of motion energy variation. We validate our system by experimenting on two datasets.
Conference Paper
Full-text available
Typical location determination systems require the presence of a physical device that is attached to the person that is being tracked. In addition, they usually require the tracked device to participate actively in the localization process. In this paper, we introduce the concept of Device-free Passive (DfP) localization. A DfP system is envisioned to be able to detect, track, and identify entities that do not carry any device, nor participate actively in the localization process. The system works by monitoring and processing changes in the received physical signals at one or more monitoring points to detect changes in the environment. Applications for DfP systems include intrusion detection and tracking, protecting outdoor assets, such as pipelines, railroad tracks, and perimeters. We describe the DfP system's architecture and the challenges that need to be addressed to materialize a DfP system. We show the feasibility of the system by describing algorithms for implementing different functionalities of a DfP system that works with nominal WiFi equipment. We present two techniques for intrusion detection and a technique for tracking a single intruder. Our results show that the system can achieve very high probability of detection and tracking with very few false positives. We also identify different research directions for addressing the challenges of realizing a DfP system.
Article
Full-text available
We are pleased to announce the release of a tool that records detailed measurements of the wireless channel along with received 802.11 packet traces. It runs on a commodity 802.11n NIC, and records Channel State Information (CSI) based on the 802.11 standard. Unlike Receive Signal Strength Indicator (RSSI) values, which merely capture the total power received at the listener, the CSI contains information about the channel between sender and receiver at the level of individual data subcarriers, for each pair of transmit and receive antennas. Our toolkit uses the Intel WiFi Link 5300 wireless NIC with 3 antennas. It works on up-to-date Linux operating systems: in our testbed we use Ubuntu 10.04 LTS with the 2.6.36 kernel. The measurement setup comprises our customized versions of Intel's close-source firmware and open-source iwlwifi wireless driver, userspace tools to enable these measurements, access point functionality for controlling both ends of the link, and Matlab (or Octave) scripts for data analysis. We are releasing the binary of the modified firmware, and the source code to all the other components.
Article
Full-text available
Multimedia services and especially digital video is expected to be the major traffic component transmitted over communication networks [such as internet protocol (IP)-based networks]. For this reason, traffic characterization and modeling of such services are required for an efficient network operation. The generated models can be used as traffic rate predictors, during the network operation phase (online traffic modeling), or as video generators for estimating the network resources, during the network design phase (offline traffic modeling). In this paper, an adaptable neural-network architecture is proposed covering both cases. The scheme is based on an efficient recursive weight estimation algorithm, which adapts the network response to current conditions. In particular, the algorithm updates the network weights so that 1) the network output, after the adaptation, is approximately equal to current bit rates (current traffic statistics) and 2) a minimal degradation over the obtained network knowledge is provided. It can be shown that the proposed adaptable neural-network architecture simulates a recursive nonlinear autoregressive model (RNAR) similar to the notation used in the linear case. The algorithm presents low computational complexity and high efficiency in tracking traffic rates in contrast to conventional retraining schemes. Furthermore, for the problem of offline traffic modeling, a novel correlation mechanism is proposed for capturing the burstness of the actual MPEG video traffic. The performance of the model is evaluated using several real-life MPEG coded video sources of long duration and compared with other linear/nonlinear techniques used for both cases. The results indicate that the proposed adaptable neural-network architecture presents better performance than other examined techniques.
Article
Full-text available
Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day
Article
Due to the advancement of camera hardware and machine learning techniques, video object tracking for surveillance has got noticeable attention of the computer vision research community. Object tracking and trajectory modeling have important applications in surveillance video analysis. For example, trajectory clustering, summarization or synopsis generation, and detection of anomalous or abnormal events in videos are mainly being exploited by the research community. However, barring one research work (which is almost a decade old), there is no recent review that emphasizes the use of video object trajectories, particularly in the perspective of visual surveillance. This paper presents a survey of trajectory-based surveillance applications with focus on clustering, anomaly detection, summarization, and synopsis generation. The methods reviewed in this paper broadly summarize the above applications. The main purpose of this survey is to summarize the state-of-the-art video object trajectory analysis techniques used in indoor and outdoor surveillance.
Article
Modern Water Distribution Systems (WDSs) are often controlled by Supervisory Control and Data Acquisition (SCADA) systems and Programmable Logic Controllers (PLCs) which manage their operation and maintain a reliable water supply. As such, and with the cyber layer becoming a central component of WDS operations, these systems are at a greater risk of being subjected to cyberattacks. This paper offers a model-based methodology based on a detailed hydraulic understanding of WDSs combined with an anomaly detection algorithm for the identification of complex cyberattacks that cannot be fully identified by hydraulically based rules alone. The results show that the proposed algorithm is capable of achieving the best-known performance when tested on the data published in the BATtle of the Attack Detection ALgorithms (BATADAL) competition (http://www.batadal.net).
Article
The BATtle of the Attack Detection ALgorithms (BATADAL) is the most recent competition on planning and management of water networks undertaken within the Water Distribution Systems Analysis Symposium. The goal of the battle was to compare the performance of algorithms for the detection of cyber-physical attacks, whose frequency increased in the past few years along with the adoption of smart water technologies. The design challenge was set for C-Town network, a real-world, medium-sized water distribution system operated through Programmable Logic Controllers and a Supervisory Control And Data Acquisition (SCADA) system. Participants were provided with datasets containing (simulated) SCADA observations, and challenged with the design of an attack detection algorithm. The effectiveness of all submitted algorithms was evaluated in terms of time-to-detection and classification accuracy. Seven teams participated in the battle and proposed a variety of successful approaches leveraging data analysis, model-based detection mechanisms, and rule checking. Results were presented at the Water Distribution Systems Analysis Symposium (World Environmental & Water Resources Congress), in Sacramento, on May 21-25, 2017. This paper summarizes the BATADAL problem, proposed algorithms, results, and future research directions.
Article
Systems based on wireless sensor have been presented as an alternative to location and navigation for visually impaired in environments where GPS is not effective. However, dealing with the information obtained from these sensors may present some challenges. The difficulties described by several authors are related mainly to the oscillation of the signals, which are susceptible to numerous interfering agents such as the scene itself, the presence of people, etc. It is necessary to increase the accuracy of the location in all the techniques observed. Therefore, this paper proposes to apply data fusion of wi-fi sensors with inertial sensors to allow visually impaired to location and navigation in indoor places using aspects of healthcare like sensor based systems embedded in wearable devices. The methodology adopted was associate redundant fusion and complementary fusion to construct a database obtained from several readings of the wi-fi sensors and an inertial sensor to construct a reference position to compare the data of the test readings to this reference and indicate the reliability in an indoor location. The results show that RSSI with SNR and magnetometer with gyroscope provide high error rates with raw and isolated data, but when fused, they show a more efficient location, decreasing the error rates observed in isolated mode. The error margins between the real position and the position indicated by the system were 0,386 m, 0,336 m and 0,148 m to RSSI with SNR, inertial and fusion application, respectively.
Article
Due to rapid developments of smart devices and mobile applications, there is an urgent need for a new human-in- the-loop architecture with better system efficiency and user experience. Compared with conventional device-based human-computer interactive (HCI) methods, device-free technology with WiFi provides a new HCI method and is promising for providing better user-perceived quality-of-experience. Being essential for device-free applications, device-free human detection has gained increasing interest, of which through-the-wall (TTW) human detection is of great challenge. Existing TTW detection systems either rely on massive deployment of transceivers or require specialized WiFi monitors, making them inapplicable for realworld applications. Recently, more and more researchers have tapped into the physical layer for more robust and reliable human detection, ever since Channel State Information (CSI) can be exported with commodity devices. Despite of great progress achieved, there have been few works studying TTW detection. In this paper, we propose a novel scheme for Robust device-free Through-The-Wall Detection of moving human (R-TTWD) with commodity devices. Different from the time dimensionbased features exploited in previous works, R-TTWD takes advantage of the correlated changes over different subcarriers and extracts first-order difference of eigenvector of CSI across different subcarriers for TTW human detection. Instead of direct feature extraction, we first perform a PCA-based filtering on the preprocessed data since a simple low-pass filtering is insufficient for noise removal. Furthermore, the detection results across different transmit-receive antenna pairs are fused with a majority-vote based scheme for more robust and accurate detection. We prototype R-TTWD on commodity WiFi devices and evaluate its performance both in different environments and over long test period, validating the robustness of R-TTWD with both detection rates for moving human and human absence over 99% regardless of different wall materials, dynamic moving speeds, etc.
Article
A novel sensor fusion methodology is presented, which provides intelligent vehicles with augmented environment information and knowledge, enabled by vision-based system, laser sensor and global positioning system. The presented approach achieves safer roads by data fusion techniques, especially in single-lane carriageways where casualties are higher than in other road classes, and focuses on the interplay between vehicle drivers and intelligent vehicles. The system is based on the reliability of laser scanner for obstacle detection, the use of camera based identification techniques and advanced tracking and data association algorithms i.e. Unscented Kalman Filter and Joint Probabilistic Data Association. The achieved results foster the implementation of the sensor fusion methodology in forthcoming Intelligent Transportation Systems.
Article
This work contributes a modelling framework to characterize the effect of cyber-physical attacks (CPAs) on the hydraulic behavior of water distribution systems. The framework consists of an attack model and a MatLab toolbox named epanetCPA. The former identifies the components of the cyber infrastructure (e.g., sensors or Programmable Logic Controllers) that are potentially vulnerable to attacks, whereas the latter allows determining the exact specifications of an attack (e.g., timing or duration) and simulating it with EPANET. The framework is applied to C-Town network for a broad range of illustrative attack scenarios. Results show that the hydraulic response of the network to a cyber-physical attack depends not only on the attack specifications, but also on the system initial conditions and demand at the junctions. It is also found that the same hydraulic response can be obtained by implementing completely different attacks. This has some important implications on the design of attack detection mechanisms, which should identify anomalous behaviors in a water network as well as the cyber components being hacked. Finally, the manuscript presents some ideas regarding the next steps needed to thoroughly assess the risk of cyber-attacks on water distribution systems.
Article
We present YOLO, a unified pipeline for object detection. Prior work on object detection repurposes classifiers to perform detection. Instead, we frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. Our unified architecture is also extremely fast; YOLO processes images in real-time at 45 frames per second, hundreds to thousands of times faster than existing detection systems. Our system uses global image context to detect and localize objects, making it less prone to background errors than top detection systems like R-CNN. By itself, YOLO detects objects at unprecedented speeds with moderate accuracy. When combined with state-of-the-art detectors, YOLO boosts performance by 2-3% points mAP.
Article
One approach to identifying outliers is to assume that the outliers have a different distribution from the remaining observations. In this article we define outliers in terms of their position relative to the model for the good observations. The outlier identification problem is then the problem of identifying those observations that lie in a so-called outlier region. Methods based on robust statistics and outward testing are shown to have the highest possible breakdown points in a sense derived from Donoho and Huber. But a more detailed analysis shows that methods based on robust statistics perform better with respect to worst-case behavior. A concrete outlier identifier based on a suggestion of Hampel is given.
Article
One approach to identifying outliers is to assume that the outliers have a different distribution from the remaining observations. In this article we define outliers in terms of their position relative to the model for the good observations. The outlier identification problem is then the problem of identifying those observations that lie in a so-called outlier region. Methods based on robust statistics and outward testing are shown to have the highest possible breakdown points in a sense derived from Donoho and Huber. But a more detailed analysis shows that methods based on robust statistics perform better with respect to worst-case behavior. A concrete outlier identifier based on a suggestion of Hampel is given.
Article
The increasing interconnectivity of SCADA (Supervisory Control and Data Acquisition) networks has exposed them to a wide range of network security problems. This paper provides an overview of all the crucial research issues that are involved in strengthening the cyber security of SCADA networks. The paper describes the general architecture of SCADA networks and the properties of some of the commonly used SCADA communication protocols. The general security threats and vulnerabilities in these networks are discussed followed by a survey of the research challenges facing SCADA networks. The paper discusses the ongoing work in several SCADA security areas such as improving access control, firewalls and intrusion detection systems, SCADA protocol analyses, cryptography and key management, device and operating system security. Many trade and research organizations are involved in trying to standardize SCADA security technologies. The paper concludes with an overview of these standardization efforts.
Conference Paper
Moving object detection is a critical task for many computer vision applications: the objective is the classification of the pixels in the video sequence into either foreground or background. A commonly used technique to achieve it in scenes captured by a static camera is Background Subtraction (BGS). Several BGS techniques have been proposed in the literature but a rigorous comparison that analyzes the different parameter configuration for each technique in different scenarios with precise ground-truth data is still lacking. In this sense, we have implemented and evaluated the most relevant BGS techniques, and performed a quantitative and qualitative comparison between them.
Background subtraction techniques: Systematic evaluation and comparative analysis
  • S Herrero
  • J Bescs
S. Herrero and J. Bescs, "Background subtraction techniques: Systematic evaluation and comparative analysis," in Advanced Concepts for Intelligent Vision Systems (Lecture Notes in Computer Science, vol. 5807), J. Blanc-Talon, W. Philips, D. Popescu, and P. Scheunders, Eds. Berlin: Springer-Verlag, 2009.
Vulnerabilities in SCADA and critical infrastructure systems
  • R J Robles
  • M K Choi
  • E S Cho
  • S S Kim
  • G C Park
  • S Yeo
R. J. Robles, M. K. Choi, E. S. Cho, S. S. Kim, G. C. Park, and S. Yeo, "Vulnerabilities in SCADA and critical infrastructure systems," Int. J. Future Generation Commun. Netw., vol. 1, no. 1, pp. 99-104, 2008.
Recurrent neural networks and robust time series prediction
  • J Connor
  • D Martin
  • L Altas
J. Connor, D. Martin, and L. Altas, "Recurrent neural networks and robust time series prediction," IEEE Trans. Neural Netw., vol. 5, no. 2, pp. 240-254, 1994.
DNN or k-NN: That is the Generalize vs. Memorize Question
  • G Cohen
  • G Sapiro
  • R Giryes
G. Cohen, G. Sapiro, and R. Giryes, DNN or k-NN: That is the Generalize vs. Memorize Question. 2018. [Online]. Available: arXiv:1805.06822