Anomaly Detection - Science topic

Dear Doctor

[A bold agenda to advance health equity and build resilience in a turbulent world This new strategy for global health, WHO’s Fourteenth General Programme of Work, 2025–2028 (GPW 14), sets a bold agenda to get the world back on track to achieve the health-related Sustainable Development Goals (SDGs) while advancing health equity and building health systems resilience in our increasingly turbulent world. Anchored in WHO’s mission to promote, provide and protect health and well-being for all, and WHO’s constitutional commitment to gender equality, universality and human rights, the new strategy has six strategic objectives that respond to the major health challenges and crises of our time: tackling health risks associated with our rapidly changing climate; preventing disease through joint action on the determinants of health; advancing primary health care (PHC) and essential health system capacities in order to accelerate efforts to achieve common goals1 and progress towards universal health coverage (UHC); improving health service coverage and financial protection; and strengthening prevention of, preparedness for and response to health emergencies.]

For anomaly detection in real-time environmental data using Federated Learning (FL), the most beneficial methods are those that support efficient, privacy-preserving, and adaptive learning across distributed nodes (e.g., edge devices with gas sensors, anemometers, etc.).

Here’s a breakdown of the most suitable FL methods and why they work well for anomaly detection in this context:

✅ Best Federated Learning Methods for Real-Time Anomaly Detection

💡 Use with LSTM-Autoencoders or One-Class SVM for time series anomaly detection.

💡 Combine with streaming-friendly models like online k-means, online PCA, or incremental autoencoders.

🧠 Choosing the Right Approach Depends On:

FactorSuggested ApproachMany heterogeneous sensorsFAD with model personalizationHigh privacy requirementDP-FedAvg or Secure AggregationNeed for real-time updatesFederated Online LearningRegional data patternsHierarchical FLSimpler, general patternsFedAvg with Autoencoder

🛠️ Real-World Toolkits & Frameworks

Today is A. Einstein's birthday. The sooner the fallacy of the theory of relativity is recognized, the better it will be for this remarkable scientist. Let's not forget that he received the Nobel Prize NOT FOR the theory of relativity (in this case, the Nobel Committee showed amazing foresight).

Kubilay Ayturan thank u so much Kubilay Ayturan

Quite interesting. I can always be your research partner in areas of fraud detection and MFA

Try to represent your data with a contouring software like "surfer" (by goldensoftware)

Good luck

Rainer

well written!

The choice of approach totally depends on the specific dataset, available resources, research objectives, and most importantly the desired level of accuracy. You can experiment with different methods to find the one that best suits your needs.

There are several approaches you can explore to detect anomalies in image datasets. You can explore different machine and deep learning algorithms, such as Autoencoders, Isolation Forests, Support Vector Machine, Convolutional Neural Networks, and Contrastive Learning. Additionally, you can explore hybrid approaches that combine machine learning, computer vision, and deep learning techniques to extract relevant features from images to detect anomalies.

I have a few references in the enclosed 2014 poster. See on p. 3, section "Outliers". I have never investigated the kind of data you mentioned.

Thanks to all

You are welcome, Sayyed Ahmad Naghavi Nozad .

I see. A potential direction could involve leveraging techniques from active learning or human-in-the-loop approaches. These methods allow for iterative improvement of models by selectively labeling data points that are most informative or uncertain. By strategically annotating a small subset of your data and iteratively refining your model, you may be able to achieve reliable outlier detection without relying solely on predefined labels.

I hope that helps.

Kind regards,

Dr. Samer Sarsam

Thank you

Machine learning techniques can greatly enhance anomaly detection in network traffic and improve cybersecurity defenses by leveraging their ability to analyze vast amounts of data and identify patterns that may indicate malicious activity. Here are some ways machine learning can be applied:

1. Feature extraction: Machine learning algorithms can analyze various network traffic features, such as packet size, protocols, header fields, and timings, to identify normal patterns and create a baseline for comparison. Any deviation from this baseline can be flagged as a potential anomaly.

2. Unsupervised anomaly detection: Unsupervised machine learning algorithms, like clustering or dimensionality reduction techniques, can identify outliers or unusual patterns in network traffic without relying on labeled training data.

3. Supervised anomaly detection: Machine learning models can be trained using labeled datasets to classify network traffic as normal or malicious. This requires a training phase where the model learns from known patterns of attacks and then applies that knowledge to detect similar attacks in real-time.

4. Behavior-based detection: Machine learning can develop models that learn the behavior of normal network traffic over time. Any deviation from this learned behavior can be flagged as an anomaly, even if the specific attack hasn't been encountered before.

5. Real-time threat intelligence: By incorporating machine learning with threat intelligence feeds, cybersecurity defenses can benefit from up-to-date information about known threats and attack patterns, enabling faster detection and response.

6. Adaptive defenses: Machine learning models can continuously learn from new data and adapt their detection capabilities to evolving attack techniques, making them more effective in combating emerging threats.

It's important to note that while machine learning can enhance anomaly detection, it's not a foolproof solution. Cybersecurity requires a multi-layered and comprehensive approach that combines machine learning techniques with expert analysis, human oversight, and other security measures.

Wasim Wahid

Elaine Lu Great, Thanks for Sharing.

Nz Kh could you please share the links such datasets?

Great to know

@Qamar Ul Islam, Hi Qamar, can you please provide links or means of access to the datasets you mentioned? Some of them can't be found by myself. Great thanks.

Manzoor Hussain Yes, you can calculate the deviation (or change) in steering as the difference between the anomalous values and the normal values. In your example, the deviation in steering is -0.22058, which means that the adversarial attack deviated the actual steering by -0.22058. This is a common way to express the change in a value due to an adversarial attack or any other factor. Depending on the context, you may also see this value referred to as the "perturbation" or "offset" in the steering angle.

Thankyou :)

_____

_____

https://davideliu.com/2019/10/26/log-analysis-for-anomaly-detection/

Dear António Brandão ,

SmartSignal can detect, diagnose, predict, and prevent critical failures. These analytics are built on unrivaled deep industry expertise and proven across the world’s largest energy organizations. Unlike generic AI/ML solutions, SmartSignal provides users access to powerful Digital Twin blueprints that accelerate time-to-value across your investments.

Regards,

Shafagat

Hello,

For the general case of time series anomaly detection, several benchmarks have been recently proposed. These two benchmarks contain labeled time series from different domains (for instance, room occupancy detection from temperature, CO2, light, and humidity [1] or accelerometer sensors of a wearable assistant for Parkinson's disease patients [2]).

You may find the links to the benchmarks below. Good luck with your thesis!

- TSB-UAD: https://github.com/johnpaparrizos/TSB-UAD (paper: )

- TimeEval: https://github.com/HPI-Information-Systems/timeeval-evaluation-paper (paper: )

[1] Luis M. Candanedo and Véronique Feldheim. 2016. Accurate occupancy detection of an office room from light, temperature, humidity, and CO2 measurements using statistical learning models. Energy and Buildings 112 (2016), 28–39. https://doi.org/10.1016/j.enbuild.2015.11.071

[2] Marc Bachlin, Meir Plotnik, Daniel Roggen, Inbal Maidan, Jerey M. Hausdor, Nir Giladi, and Gerhard Troster. 2010. Wearable Assistant for Parkinson's' Disease Patients With the Freezing of Gait Symptom. IEEE Transactions on Information Technology in Biomedicine 14, 2 (2010), 436–446. https://doi.org/10.1109/TITB. 2009.2036165

This dataset has different data sources, including system logs . Check it

(PDF) X-IIoTID: A Connectivity-and Device-agnostic Intrusion Dataset for Industrial Internet of Things (researchgate.net)

(PDF) X-IIoTID-intrusion-dataset-ready-for-machine-learning (researchgate.net)

Hi Amer.

Check this dataset which include cloud network traffic.

(PDF) X-IIoTID-intrusion-dataset-ready-for-machine-learning (researchgate.net)

Hello,

There are many solid libraries for shapelet discovery and transformation, depending on the specific method of interest e.g Learned Shapelets, Shapelet Forests, etc. My personal favorites are tslearn (which provides a great implementation of learned shapelets: https://tslearn.readthedocs.io/en/stable/index.html and wildboar (which provides an excellent implementation of shapelet forests): https://github.com/isaksamsten/wildboar .

In terms of using shapelets for anomaly detection, I believe that wildboar offers this or related functionality out of the box. Please visit this page for more information and a code example: https://isaksamsten.github.io/wildboar/master/examples/unsupervised.html#outlier-detection

Hope this helps.

Jingchen Wu Did you find an answer to your question? I am looking to do something similar.

Dear Adam Ww ,

Anomaly detection is one of the most common use cases of machine learning. Finding and identifying outliers helps to prevent fraud, adversary attacks, and network intrusions that can compromise your company’s future.

Regards,

Shafagat

Thank you. Can you please clarify what does 'all k classes' mean in your first sentence. Do I have to train my Anomalous data also in autoencoder? Also please let me know if there is any resource I can follow to figure out this process. Thank you.

Dear Utsav,

In Matlab, training an autoencoder is so easy. The below link shows how to define and train an autoencoder

Actually I am looking for some research based open source code that I can download and run on some remote machines containing netflow data files of an ISP.

I think Subsampling would be the better approach.

why not consider using S-transform as it combines the properties of FFT and wavelet transform.

Ever hear of quality control. An out of control point is an outlier. David Booth

I have work with anomaly detection using auto-encoders. Howeve, I have used semi -supervised learning. Here is the link for the work:

Dear Dr Hoffmann,

Thank you for sharing this interesting topic and I would be happy to participate in the future step of this project. the below paper might be interesting to read,

see

chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/viewer.html?pdfurl=https%3A%2F%2Farxiv.org%2Fpdf%2F2001.01293.pdf&clen=2380896&chunk=true

Dear Maryam Khaliji,

You may want to look over the stuff below:

In most case studies, the LSTM is applied.

I'm agree

Thank you for your responses..! i will look into it..!

Dear Mengjie Zheng ,

In addition to these helpful answers, I would not recommend adding or subtracting a multiple of the IQR to Q1 or Q3 as this might create non-physical values for non-normal data in the same way that SD differences from the mean can.

It would make more sense to divide your data into a greater number of percentage intervals, such as every 10% of the data, then use the values nearer the lower and upper limits.

Tariq Sm , you need to do Adaptive thresholding techniques . I would suggest it to do it in python using OpenCV. This would be easier than Matlab. Here is short article on the usage of OpenCV for Adaptive thresholding.

Thanks for your collaboration. This paper is about pedestrian tracking through JPDAF, but I required paper in which JPDA is used only for association not for tracking

I think these sites may help you

Best Regards

Fehmi Özbayrak Nonstationarity also depends on the kernel density estimates, and may not reject stationarity when m is small. Log-spline bivariate density estimation and LLDM by Loader may also be considered to improve tail estimation, in the presence of moderate to low values of m.

That's what the videos are for

I suggest you follow

Hope this helps

You can use boxplot in SPSS to identify outliers

As a metric you can use the variance and see how it varies when eliminating one data-point from the batch while keeping all the others. Scan the batch and test eliminating each data-point, one at a time. When an eliminated data-point is an outlier the variance of the batch decreases a lot. This would work with the histogram method.

Dear Taimoor Muzaffar Gondal

You cam develop a new scheme of an anomaly detection by doing the following:

1- Reduce the power communication overhead.

2- Reduce the process complexity.

3- reduct the data which have been collected from the sensors.

4- Try to design non central communication ( Distributed communication ) .

5- Try to use a light weight algorithm to classify the gathering data.

Dear All, Thank you very much for sharing these information.

Do we have any paper with comparision of these datasets with CMU-CERT r4.2 dataset ?

Hi;

Please chek this link:

Regards

Hi,

I suggest you to see links and attached files un topic.

https://arxiv.org/abs/1811.08055

Best regards

In my opinion we can you find the autoencoder in RNN architecture, I suggest if you can you use CNN architecture, maybe that can you give the good results without the overfitting.

I hope that be Claire for you. @ Anh Tram Nguyen

Hi Joo,

This is an interesting research, is it possible for your to share a sample image to see what you are actually looking for, then may be share some ideas with you.

regards

Thomas W Kelsey Thank you very much for sharing valuable information

you can read those articles is very useful

https://towardsdatascience.com/anomaly-detection-for-dummies-15f148e559c1

https://towardsdatascience.com/best-clustering-algorithms-for-anomaly-detection-d5b7412537c8

there are many datasets available online especially for anomaly detection,

one of the best websites that can provide you different datasets is the Canadian Institute for Cybersecurity.

here te link :

If you dont have the history data of both motors, estimate the motor-to-motor variation by comparing two healthy and new motors. This way, you have an educated guess of how much individual motors differ when they are produced. Create an anomaly detection model with healthy data only, and try to determine the proper classification cutoff for anomalous result with the known variation of healthy motors.

Reza Fotohi that’s my question only stack exchange and it doesn’t have any answer. Not sure why you are sharing it with me.

As anomaly detection algorithms aim to classify whether the target is an anomaly or not, it falls under binary classification. So, mostly the evaluation metrics used are accuracy, precision and recall and ROC-AUC curve. Usually, the number of anomaly samples will be less when compared to normal samples. Hence just accuracy measure is not a correct evaluation metric. The best and suitable evaluation metric for anomaly detection algorithms is the ROC-AUC curve.

The anomaly detection with ultrasonic waves depends on different parameters like crystallisation, geometry, thermic history, internal friction and others for the inspected materials and, on the other hand, on the ultrasonic equipment like voltage excitation of the probes, beam forming, transducer properties and others. Furthermore, some post-processing techniques extract the interesting information from the data matrices.

Which anomalies you want to detect?

May be, this helps: http://www.signal.uu.se/Publications/pdf/c1011.pdf

you can do patient matching (like propensity score based methods) or kNN, and look at patients who do not match.

Thank you Duc P. Truong David Morse Cristian Ramos-Vera for the response.

My experiment is based on user activities inside the network. I am trying to group users into similar groups (clusters) based on their activities. Also, identify outlier data points as anomalies.

The shared papers look useful, I am going through these.

Furthermore, there could be something on these lines for me.

These attributes are being used to raise alerts and incidents in some of the commercial cyber security solutions such as Microsoft CloudApp Security (MCAS), Microsoft Azure AD Identity Protection (MAADIP), and Dark Web Security Solution from Vology. At least MS Solutions build up reputation for users and devices based on multiple attributes and then rate risk level of them or generate an alert or an incident using Machine Learning and AI. I suggest you take a look at the existing solutions and possible white papers of them.

When you perform a binary classification, you are actually trying to learn a separation function that can separate two distributions. However, having samples of only one class means that your classifier really needs to learn the distribution of data samples on that class, which is much more difficult (same with GANs).

Interesting question!

Following

You can go through this tutorial.

https://www.datascience.com/blog/fraud-detection-with-tensorflow

Any effective supervised learning method is my choice