88 - Applications of Autoencoders - Anomaly Detection

[@HadiSepehrvand]

The best explanation of anomaly detection using autoencoders. It would be great if you talk a bit about the calculation of the threshold.

[@sandeepmandrawadkar9133]

Excellent explanation for anomaly and its detection using Autoencoders 👍. Super simplified explanation. Enjoyed knowing how to deal with skewed dataset. Thanks for everything.

[@DigitalSreeni]

Glad you enjoyed it!

[@Erosis]

For those of you wondering what to do if you can't possibly label the anomalies yourself, you can instead train your autoencoder on your entire dataset and then use a clustering algorithm on the center dense layer outputs (called embeddings). Hopefully you will have decent separation from the "good" cluster and a "bad" cluster.

[@DigitalSreeni]

Thanks for the tip Austin.

[@cmfrtblynmb02]

Hello. Can you provide a guide for what you described? Thank you

[@hansalas]

@@cmfrtblynmb02 Actually, you can can run the clustering on the raw data iself but due to the curse of the dimensionality, it is better to do it in the compressed data (centre dense layer)

[@hussamalh8554]

Nice and useful video!

[@adithiajovandy8572]

very nice material for beginner :), im waiting anastropic difussion

[@learning_with_irving4266]

Thank you for your contribution

[@binaysharma397]

Easy to understand video...Can you please make one for detecting anomaly in images or provide link if any?

[@Anis-f2t]

Hi, thank you for the great video. Could I please know where can I find the excel data file?

[@iftikhar58]

Hi I have one question for you. The example you did is very good but in the real world problem these solution are not applicable. For example i have 3 years of sensors data and bilions of points there. i know there are anamolies in the dataset which you referred as bad dataset. So in the real world most of the time have bad dataset and its very diffuclt to clean the data and converting data from bad data to good data .if we find the way to convert bad to good data why we neeed to autoencoder we already fine the machanicsm to find the anamolies and find good data from the bad data.

[@harshitsingh7556]

i cant get over your videos they are so informative lovedd them

[@DigitalSreeni]

Glad you like them!

[@j0shm0o1]

Thanks!

[@DigitalSreeni]

Thank you very much for your contribution Joshi ji. Please keep watching.

[@jamespaz4333]

From my understanding, I read that Autoencoders work with unsupervised learning. I see you are labeling your data, so these can be used with Supervised Learning?

[@DigitalSreeni]

Autoencoder technique itself where you provide an input and it reconstructs the same input as an output is unsupervised. But what good is a system that gives an output same as input? So for any real application using autoencoders you need to provide 'labeled' data to fool the autoencoder to reconstruct something different than the original input. For example, provide color image as label for B&W image inputs so you can train the autoencoder to reconstruct a color image from a B&W image.

[@jamespaz4333]

@@DigitalSreeni thank you so much!!!!

[@DhamuR-c1b]

It is not clear what is the input and why we use autoencoders to predict the result

[@yacine074]

Thnak's a lot for those videos , can you please post a video about MRI segmentation based on Deep Learning ?

[@serhatgvn]

Thank you very much for the video. I have a question: If we have the labeled data and only 2 variables, wouldn't we simply create a classification model using both the "good" and the "bad" data, by splitting the train/test sets? I understand that this is just an example but in practice we'd apply autoencoders to high-dimentional datasets, correct?

[@adityagupta-hm2vs]

we are not using 'bad' data for training, :)

[@bonadio60]

Hi @DigitalSreeni, in this example once we detect an anomaly feeding new data, would be possible to identify what data caused the anomaly, like if was the "power" or the "detector" value? Thanks for the great content

[@DigitalSreeni]

You will know the data points tagged as anomalies so you can always extract those and investigate the real reason. You may want to look into predictive maintenance applications if interested in tracking multiple devices to predict failure (anomaly).

[@SANJAY2ROKA]

Nice video I like it..............................

[@DigitalSreeni]

Thank you so much 😀

[@suvabbaral5880]

@DigitalSreeni my root mean squared error changes on everytime I train this model. Now, I think that is because of the random train/test split. The rmse is sometimes as high as 6/7 or so for the same dataset. Is that normal?

[@umairsabir6686]

Hi @Sreeni Can you please make a video on an optimized version of ANOGAN for unsupervised anomaly detection ? Thanks. Yours lovingly !

[@ietezazhassan2094]

Nice 👍

[@taranjeetsingh1712]

But we are judging our data on the basis of the reconstruction error. How to know what are the anomalies in the dataset. Here you have mentioned on ur own that this data is good and this is bad what if we don't know it and we have to find the anomalies. please reply

[@DigitalSreeni]

Anomaly detector highlights the areas that do not look normal. A 'normal' is something you define during the training process. Anomaly detector by itself will not tell you why it is an anomaly, other than providing you with metrics (e.g., reconstruction error). Once the anomaly is detected, you can go to that specific data point and investigate further, manually or using another network or algorithms.

[@umairsabir6686]

Thanks for these videos. Can you please make a video on Variational Autoencoders as well. Thanks

[@DigitalSreeni]

Added to my list.

[@umairsabir6686]

@@DigitalSreeni Thanks Sreeni. Your explanation is great and uncomplicated. Anomaly detection is the biggest issue that every industry is having. Can you please think about a solution in which we can extract convolutional features from a bunch of images and perform Novelty detection on those features using one class classification techniques like One class SVM, LOF or Isolation Forest ?

[@dibyakantaacharya4104]

how to get the csv file ...please tell me

[@DigitalSreeni]

You can find the code and files on my github page: github.com/bnsreenu/python_for_microscopists

[@dibyakantaacharya4104]

@@DigitalSreeni how didi u create this csv file ...can u explain

[@dibyakantaacharya4104]

how did u find this power and detector no??

[@bmwme4689]

Hey Sreeni, i have an important question for my master thesis. I also want to use autoencoder for anomaly detection on multivariate sensor data. (not forecasting) I built the sequences already. But i don’t know how to code the models. You just used the sequential but i thought on time series data it is a nogo. I wanted to use instead of this 1D CNN or LSTM. I am so confused of this. Can you help me out there? 🥲