For questions related to anomaly detection (or outlier detection) algorithms, which is the identification of rare items, events or observations which raise suspicions by differing significantly from the majority of the data. There are unsupervised, supervised and semi-supervised anomaly detection algorithms.
Questions tagged [anomaly-detection]
34 questions
8
votes
1 answer
Which unsupervised learning technique can be used for anomaly detection in a time series?
I've started working on anomaly detection in Python. My dataset is a time series one. The data is being collected by some sensors which record and collect data on semiconductor-making machines.
My dataset looks like this:
ContextID Time_ms…
some_programmer
- 225
- 1
- 4
6
votes
1 answer
Find anomalies from records of categorical data
I have a data-set with $m$ observations and $p$ categorical variables (nominal), each variable $X_1, X_2,\dots, X_p$ has several different possible values.
Ultimately, I am looking for a way to find anomalies i.e. to identify rows for which the…
bat
- 61
- 1
5
votes
1 answer
What is the difference between out of distribution detection and anomaly detection?
I'm currently reading the paper Likelihood Ratios for Out-of-Distribution Detection, and it seems that their problem is very similar to the problem of anomaly detection. More precisely, given a neural network trained on a dataset consisting of…
Mahmoud
- 243
- 2
- 7
4
votes
1 answer
Which unsupervised learning algorithm can be used for peaks detection?
So, I have a dataset that has around 1388 unique products and I have to do unsupervised learning on them in order to find anomalies (high/low peaks).
The data below just represents one product. The ContextID is the product number, and the StepID…
some_programmer
- 225
- 1
- 4
3
votes
1 answer
Is overfitting not always bad for anomaly detection?
If I train an autoencoder until it's overfit from the training set, doesn't it mean it's good? I mean, since it learns the entire training set (which is considered non-anomalous), then when it meets the new data, it's supposed to be anomalous due to…
Muhammad Ikhwan Perwira
- 800
- 3
- 10
3
votes
0 answers
Defect Detection System using Deep Learning
What is the general approach to defect detection in deep learning?
Would the approach be better if we try to learn the positive images (defects in images) as much as possible or we try to learn the negative images (images without blemishes) and try…
user1538798
- 131
- 1
3
votes
1 answer
How can auto-encoders compute the reconstruction error for the new data?
Autoencoders are used for unsupervised anomaly detection by first learning the features of the data set with mainly "normal" data points. Then new data can be considered anomalous if the new data has a large reconstruction error, i.e. it was hard to…
Brian
- 131
- 2
3
votes
0 answers
How to perform unsupervised anomaly detection from log file with mostly textual data?
I have a log file of the format,
Index, Date, Timestamp, Module, App, Context, Session, Verbosity level, Description
The log file can be considered as a master log, which consists of individual logs from several modules constituting a distributed…
Kraken10
- 31
- 3
3
votes
2 answers
Which unsupervised anomaly detection algorithms are there?
I need to create model which will find suspicious entries or anomalies in a network, whose characteristics or features are the asset_id, user_id, IP accessed from and time_stamp.
Which unsupervised anomaly detection algorithms or models should I use…
Abishek
- 43
- 6
2
votes
2 answers
How to detect outlier images?
Before I describe my challenge, I want to point out that I have searched extensively online for "outlier image detection", "anomaly images detection", etc., but all returned results are about finding anomalies or outliers within an image (e.g., a…
pookie
- 131
- 3
2
votes
1 answer
How to calculate a meaningful distance between multidimensional tensors
TLDR: given two tensors $t_1$ and $t_2$, both with shape $(c,h,w),$ how shall the distance between them be measured?
More Info: I'm working on a project in which I'm trying to distinguish between an anomalous sample (specifically from MNIST) and a…
Hadar Sharvit
- 381
- 1
- 13
2
votes
0 answers
Application of Blockchain in Fraud detection in stock market
I want to develop a fraud detection application in the stock market Using Blockchain technology, we have some pattern that defines the anomaly for use of supervised machine learning but there is one question remain:
What is role of machine learning…
R1-
- 137
- 1
- 9
2
votes
1 answer
Are there any advantages of using rules-based approaches versus models for detecting spam?
Suppose that we have unlabeled data. That is, all we have are a collection of emails and want to determine whether any of them is spam or not. Let's say we have $1,000$ rules to determine whether a particular email is spam or not. For example, one…
rulesguy
- 21
- 1
1
vote
0 answers
Replicating conv autoencoder for anomaly detection, very blurry reconstructions
I’m trying to train an autoencoder on the hazelnut dataset of MVTec AD for reconstruction to detect anomalies.
I’m am trying to replicate the results of this study:
https://arxiv.org/pdf/2008.12977.pdf
In particular, the simpler setup used in the…
JeanMi
- 165
- 4
1
vote
0 answers
Pseudo Label Generation for Generative Cooperative Learning
I am trying to implement this paper for unsupervised video anomaly detection.
The gist of the paper seems to be:
Create a dataset for an unsupervised setting, by mixing up the train and anomalous videos (section 4)
Divide each video into $p(=16)$…
satan 29
- 141
- 3