Visualizing Convolutional Filters from a CNN

[@deeplizard]

Check out the blog for this video here: deeplizard.com/learn/video/cNBBNAxC8l4

[@debgandharghosh3981]

The github repository in the blog isn't available anymore

[@Arcaerus]

I've been struggling with this subject because my prof can't explain things but you explain it so clearly!!!!!!!!! Thank you so much!

[@inbb510]

I was looking for this kind of video for ages. Everytime when I see tutorials on CNNs and building it through code, they never ever explain what sort of filters are being used in the architecture. This video cleared by confusions on this matter. Thank you very much.

[@MuhammadArnaldo]

so far this channel is the best to learn machine learning. I hope you keep uploading more videos... about RNN, LSTM, segmentation, spiking NN... and more maybe

[@tymothylim6550]

Thank you very much for the video! I learnt quite a lot more from seeing the different complexities of the different conv layers!

[@liammellor3270]

Just wanted to say, I love your videos. They are very informative and explained extremely well, please continue doing work in this area of work!!

[@deeplizard]

Thank you, Liam!

[@ayush612]

Awesome!!! In just 2 videos I feel I have got a deeper intuition of this whole thing! Thank you!

[@HAL9OOOTUBE]

Just finished watching all 22 of these videos, they were super helpful! Just getting started with TF and was completely lost without understanding the vocabulary and concepts. Still lost but a little less so now :). Hopefully you decide to keep making these types of videos and I will recommend them to anyone looking to get into ML.

[@deeplizard]

Hey Javed, thanks for letting me know! I'm glad these videos were helpful for you. If you're interested, I also have a Keras playlist that goes through some basics of building the network, training, predicting, etc. Keras is built on top of Tensorflow and is a higher level neural network API. Either way, good luck on your ML journey! kzbin.info/aero/PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL

[@HAL9OOOTUBE]

Yeah I'll probably be taking a look at those too this weekend even though I have to focus on TF for my current project, thanks!

[@ygpsk3860]

thank you for these videos! amazing series... been binge-watching your channel for the past few hours, and feel like i learned more than i did in previous few months

[@familywu3869]

Same here

[@edobr3384]

Thanks for the video! I was wondering how to explain what is a filter in a CNN to my students in an easy way and I found your video :3 ty!

[@qusayhamad7243]

thank you very much for this clear and helpful explanation.

[@alfadhelboudaia1935]

I am so hyped to discover your channel, would hope you upload videos on GANs, VAEs, LSTM, NFs.

[@sinamohamadi9580]

absolutely awesome.

[@justchill99902]

You are right! It is very interesting. One of the best videos in this series!

[@deepcodes]

Great channel!!!, such an ease to learn these topics.

[@davidmitchell9934]

Great job on these videos! You're great at explaining the intuition of these analyses. Do you work in a data science field?

[@deeplizard]

Thank you, David! I'm glad you're liking the videos. My experience in data science comes from personal projects and research. 🤓

[@nithinkvijayan2708]

Your videos are so informative. Glad I found this channel. Thank you, you should have more subs.

[@noeltam75]

Sorry I am still not able to understand the last part, when you illustrate the dog face. Why are we not seeing the edges of the object in your demonstration, instead we see only random patterns. I know you have explained in the video, but can you simplify what do you mean again?

[@deeplizard]

Hey Noel - So, in this video, we were passing the network a plain gray image with some random noise with the objective of being able to visualize what sort of input would maximize the activation for any given filter. The images that we're visualizing from these filters are the transformed grey images that would maximize the corresponding activations the most. In the previous video of this playlist on CNNs, however, we did something a bit different. There, we looked at the patterns that a given filter was able to detect from _specific images_ (dog faces, etc.) that highly activated the filter. Let me know if this helps clear things up.

[@sgrimm7346]

Nice video....you should have more views. My question is, HOW do the filters learn which features to look for? Example, how does one filter learn vertical lines and another filter learn horizontal lines? And eventually, the higher order filters learning angles and textures? Thank you.

[@Biggzlar]

It's weird, the video is so well done but you neglect to mention the entire idea of the processing code. That we perform gradient ascent and instead of updating our network with the gradients, we update the input image. Thus the image gets morphed into a matrix, that maximizes the filter activation. Came here to learn this but had to read the blog post instead.

[@saigopalpotturi2926]

could you please explain code line by line for better understanding

[@pseudooduesp2805]

thanks for video

[@DanielBurrueco]

Great video! There's something I don't get yet (I haven't gone through the code): a filter is usually small (3x3, 5x5, 7x7...) but those filters showing patters seemed huge (compared to the ones I imagined). They must be no less than... 32x32. Are they really that big, or am I missing something?

[@deeplizard]

Thanks, Daniel! You're right, filters are usually small. In fact, the filters for each convolutional layer within the VGG16 network (used in this video) are all 3x3. The visuals we saw of the gray squares with patterns on them are not the filters themselves. Rather, we're passing the network a 128x128 plain gray image with some random noise with the objective of being able to visualize what sort of input would maximize the activation for any given filter. The outputs from this process are the transformed grey 128x128 images that would maximize the corresponding activations the most. Let me know if this helps clear things up.

[@DanielBurrueco]

Hi, I didn't answer before because I didn't completely understand it. But I came across some nice code from the Keras team that does exactly what you said: it creates a loss function that maximizes the activation for the filter whose activation map we want to visualize. This is the code: github.com/keras-team/keras/blob/master/examples/conv_filter_visualization.py I'm not quite sure about how the K.gradients function works, but assuming it works, it's not difficult to visualize any filter of any layer. Amazing. Only after having played with it I can say I understand it. Thanks

[@deeplizard]

Yes, definitely took me some time to play with the code in this video myself before I fully got what exactly it was doing. Also, yes, that link is to the same code we used here, so you're on the right track! I'm glad you were able to develop an understanding for it. :)

[@Paul-lt7ij]

Sweet voice :)

[@rishabjain9275]

hey, what is the difference between block3_conv2 and block2_conv2 ?

[@mohamedmahdy969]

Hi, How are you ? really you did a great job. I watched your previous videos in this list and I can say i got a good understanding because of your well preparation of the video and your simple presentation of the information. yet, in this video, I watched it twice and still i have the feeling that there are a lot of missing parts to me. 1- I understand that, in this video, you are passing a gray input image with random noise and you are displaying to us or (visualizing) the input images that will give the most activation for the filters. Am i correct ? "Please correct me if i am wrong". (So, your input input is gray images with random noise and what we are seeing in the end of the videos are the instances of the input images which most activated certain filters) 2- if I am right in the first part, I don't understand what is conv1-block1, conv2_block3, and so on. can u explain it in the comment? and why we are seeing many of the images not only just the one the most activated the filter ? sorry if I asked the wrong question maybe I misunderstood the whole video; in this case, can you simply correct me ?

[@deeplizard]

Hey Mohamed - You are absolutely correct for number (1), and your questions are completely valid! For (2), "conv1-block1" means the first convolutional layer in the first group of convolutional layers. So, for example, a network may have five convolutional layers that are then followed by some dense layers, and then another group of five convolutional layers followed again by some more fully connected layers. We're calling these groups of convolutional layers "convolutional blocks." So, the first group would be block1, the second group would be block2. When we want to refer to the first conv layer in block1, we say "conv1-block1." Additionally, the 25 squares for one convolutional layer represent the 25 different filters contained within the layer. Each filter detects a different pattern. Does this all make sense?

[@mohamedmahdy969]

@@deeplizard Thanks a lot for your response and declaration. You still dedicated and giving a quick response even after 9 months after posting the video. Now, I can say that there are sets of convolution layers spread within the hidden layers. The set is called block. So, the first set of the convolution layers is block-1, and the conv-3 is the third convolution layer in the set, of course according to which block we are talking about. Moreover, each convolution layer is composed of number of filters. Therefore, if we are talking about block-1 conv-2, what we are seeing is the instances of the input gray image with random noise that activated the filters in the 2nd convolution layer in the 1st set, block, of convolution layers. Thanks again. I hope I got it right.

[@deeplizard]

Yes, this is a completely accurate explanation! Great job!

[@gaureesha9840]

In this video the layers i.e. conv1-block1, conv2-block3 are just the filters(weights) that we get after each layer. We did not convolute these filters with anything. They are just filters that we have learned. In the last part she says that now when we apply those filters i.e. convolute them with dog pics, then we get a convolutions that actually look like dogs.

[@messapatingy]

What were the images used to train this CNN. Wild guesses - Cells, Fabrics, Snakes.

[@deeplizard]

Hey Andre - The network was trained on images from the Imagenet library: image-net.org/explore

[@CosmiaNebula]

0:30 overview and link to full post 1:50 the jupyter notebook 3:20 results 5:00 recall previous lesson

[@johanneszwilling]

😳 Where do the very first filters come from? Are they only always those four from the beginning, filtering for straight edges up, down, left, right?

[@deeplizard]

Hey Joe - Are you referring to the filters that are shown at 0:26 in the video? If so, those filters were pulled from the previous video in the playlist on CNNs. In that video, I go into more detail about filters. kzbin.info/www/bejne/j4PLqZeMoMSmf9U I just created these filters for illustration purposes to show what an "edge detector" filter would look like. In general though, all the filters throughout a network are randomly initialized, and the values will change during training. With this being said, networks have far more complex filters than the ones I showed at 0:26.

[@jeevithavk5084]

How to detect what kind of filter is used in CNN? Also how to visualize this filter? Kindly help me

[@ismailelabbassi7150]

i love you

[@Titu-z7u]

Hi, I just don't understand how filters can be visualised. And also why would you say that some filters are highly activated for some images? How can can filters be activated? I mean filters can only extract features from an input image or matrix. So what does does it mean when you say they can be visualised or highly activated?

[@salmanzcy]

Is there a way for me to extract the values of the filters and inspect them?

[@uaeustream2562]

First, this is great work. I find error in using: grads = K.gradients(loss, input_img)[0] and I am also not sure if I have to insert an image at (I am using it as it is): input_img = model.input Can you help in running this?

[@tunkyi7162]

Need help please answer, I would like to know if you pass a grey image suppose it is a one channel, then after convolving with the filters, why the image appears colorful like green, pink in your video. It has something to do with RGB channels. Please explain thanks.

[@sgt.mcgragon359]

Halo, does number of filters depends on number of nodes ?....like one filter per node in a conv layer?

[@deeplizard]

Yes. Only, the input and output are channels. This is why diagrams of CNNs look like the one at the top of the page here: deeplizard.com/learn/video/k6ZF1TSniYk

[@ujjwalkumar8173]

what are blocks in a convolutional layer??

[@deeplizard]

conv block == group of conv layers

[@ujjwalkumar8173]

@@deeplizard Well I had'nt expected that u will reply .. bcz this video was posted almost 3 years ago..Still u are maintaining it ..that's something awesome.. Loving ur series :)

[@thespam8385]

{ "question": "Gradient ascent differs from gradient descent in trying to _______________ loss in order to _______________", "choices": [ "maximize / emphasize pattern detection of the filter", "minimize / increase accuracy", "maximize / identify overfitting", "minimize / isolate the activation function" ], "answer": "maximize / emphasize pattern detection of the filter", "creator": "Chris", "creationDate": "2019-12-14T01:42:45.651Z" }

[@deeplizard]

Thanks, Chris! Just added your question to deeplizard.com

[@tallwaters9708]

Thanks for the video. Could you please clarify a bit more what the visualisation of the dog faces at the end was? Was that a deep layer's filter applied to the raw input image? Or something else?

[@deeplizard]

Hey TallWaters - So, in this video, we were passing the network a plain gray image with some random noise with the objective of being able to visualize what sort of input would maximize the activation for any given filter. The images that we're visualizing from these filters are the (transformed grey) images that would maximize the corresponding activations the most. In the previous video of this playlist on CNNs (kzbin.info/www/bejne/j4PLqZeMoMSmf9U), however, we did something a bit different. There, we looked at the patterns that a given filter was able to detect from specific images that highly activated the filter. That's what we were looking at with the dog faces example. Let me know if this helps clear things up.

[@tallwaters9708]

Oh so you're just taking the raw filter in deeper layers and running it over certain training/test images? But those filters would usually be for blocks with more dimensions right? I mean initially the filters would be like (5, 5, 3) where 3 represents RGB colours. But the later layers would have filters like (8, 8, 64) perhaps? Am I totally misunderstanding? :(

[@Sikuq]

Great video. Thanks. I understand the general model of conv16,maxpool,conv32, maxpool ... flatten, dense , dense, non-linearity. But do all those filter values respectively get added into one image, or is it simply a single filter value given a success rating during training and then those total yes rated filter images use in essence human understanding "gestalt" to a pixel figure we call say 8. So the more layers and the more neurons in each layer give us more fine tuning control to a point? So a cat's two ears is a function of perhaps 100 filtered success rated images?

[@deeplizard]

Hey Christian - Yes, your latter explanation is correct :) The section called Output Channels And Feature Maps in the blog below may be helpful in this area as well. deeplizard.com/learn/video/k6ZF1TSniYk

[@Sikuq]

​@@deeplizard Thank you so much for your answer and exciting reference. I need to learn more Pytorch obviously. Every time I think I know something, you have another video I need to learn. And you have a long list of videos making Deeplizard the best learning source online bar none. On your vlog you quit your jobs but it looks like you have more than two full-time jobs now, lol.

[@BruceSchwartz007]

What do you mean by a the "first convolutional block of the first convolutional layer"?

[@ashwinv8305]

conv1-block1" means the first convolutional layer in the first group of convolutional layers. So, for example, a network may have five convolutional layers that are then followed by some dense layers, and then another group of five convolutional layers followed again by some more fully connected layers. We're calling these groups of convolutional layers "convolutional blocks." So, the first group would be block1, the second group would be block2. When we want to refer to the first conv layer in block1, we say "conv1-block1." Additionally, the 25 squares for one convolutional layer represent the 25 different filters contained within the layer. Each filter detects a different pattern.

[@messapatingy]

I may have spoken too soon - having watched passed that point - but even then, I'm not sure what I'm seeing, which can't be good, right?

[@deeplizard]

Hey Andre - So, in this video, we were passing the network a plain gray image with some random noise with the objective of being able to visualize what sort of input would maximize the activation for any given filter. The images that we're visualizing from these filters are the (transformed grey) images that would maximize the corresponding activations the most. In the previous video of this playlist on CNNs, however, we did something a bit different. There, we looked at the patterns that a given filter was able to detect from specific images that highly activated the filter. At 4:57 in this video, I attempt to make that point. Let me know if this helps clear things up.

[@riop7600]

Could u please explain the code with more details ..Thank you

[@deeplizard]

Hey Rio - Are you asking about the code specific to this video?

[@sharkk2979]

i watched all the series. I am impressed by mandy !! wish I could get girlfriend like her.

[@matharbarghi]

No access to real code and no discussion about the code in the video....

[@deeplizard]

We summarize the code from 1:15 to 3:12. The full code is available at the Keras link in the description.

[@Paul-lt7ij]

Am i the only person who thought her voice is so sweet?

[@deeplizard]

Machine Learning / Deep Learning Tutorials for Programmers playlist: kzbin.info/aero/PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU Keras Machine Learning / Deep Learning Tutorial playlist: kzbin.info/aero/PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL

[@SvSzYT]

first, what is the difference between keras.layers.Conv2D(32, (3, 3)) and keras.layers.Conv2D(256, (3, 3)) ?

[@comalab2387]

Surprising how the inner mechanics of a neural network can sometimes be visualized in a humanly comprehensive way. Most of the time i only encounter chaos ^^ Cool demo!

[@deeplizard]

Totally agree, Coma Lab! And thank you!

[@Kenspectacle]

How does the convolutional network layers and block work exactly in the example in the video? like, what does block5_conv2 refers to exactly?

[@paragjp]

Why we want to maximize our loss ? Not understood very clearly. Secondly once we have maximum loss then how we reduced to minimum loss further ? Can you pl explain Thanks

[@MVTN]

Thanks for the video, it was really helpful

[@ling6701]

Link to previous video is here: kzbin.info/www/bejne/j4PLqZeMoMSmf9U

[@devsutong]

wish i could contribute to your patreon page😒

[@mariaarbenina6551]

Hi. I can't find the notebook you're using in this video on your website. I found deep-learning-fundamentals-deeplizard.ipynb but it doesn't have the Visualizing Convolutional Filters from a CNN part. Where can I found it? Thank you.

[@deeplizard]

Hey Maria - The code used in this episode is from this original Keras blog: blog.keras.io/how-convolutional-neural-networks-see-the-world.html As stated there, the author has since updated the blog, now at this link: keras.io/examples/vision/visualizing_what_convnets_learn/ The corresponding github link and Jupyter Notebook for the updated code from the blog are below: colab.research.google.com/github/keras-team/keras-io/blob/master/examples/vision/ipynb/visualizing_what_convnets_learn.ipynb github.com/keras-team/keras-io/blob/master/examples/vision/visualizing_what_convnets_learn.py

[@mariaarbenina6551]

@@deeplizard Thanks! Great series, by the way, thank you for your work! I wish you were my uni professor.

[@akshatgarg6635]

img_width img_hight not defined?

[@tomwu163]

Could you please clarify how the loss function for maximizing the activation works? I don't understand what each gradient ascent step actually updates, since here our trained model already has fixed weights (which is what the loss function for the output of the model updates) and also a fixed input picture. So what can this loss function possibly be maximized on, in order to visualize what the filter is looking for?

[@tobiask5131]

What annoys me is this: everyone does the vgg16 model but what if I actually trained my own keras model? As far as I can tell this code only works for this specific example, throwing weird assertion errors without explanation and really is no help if you have another model.

[@tobiask5131]

Now I get errors like: "Could not find resource: localhost/conv2d_9/bias" so it really seems to be model specific. That's just no help at all..

[@gamma_v1]

The previous 21 videos were very clear. But this one had a lot of gaps. For example the code explanation was very short. Great work though. Keep up the good work.

[@deeplizard]

Hey Gamma - Appreciate the feedback. My intention was to give a high-level overview of the code and focus more on how we can interpret the visualizations. Maybe a I'll add a video to the Keras playlist (below) where we go over the specifics of this program. In that series, the focus is on all the code-level details :) kzbin.info/aero/PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL

[@Blank027-r5p]

How can i know filter value?

[@loaialamro9699]

Great tutorial, where can I find the filters images after run the code because I can't find them the in code folder. Thank You

[@deeplizard]

At 2:45 in the episode, you can see the path to where the images should be saved. You should create a directory called conv_images inside of the directory for which your code resides. The images should be saved in conv_images if you follow the same code shown at 2:45.

[@guardrepresenter5099]

These picture which shown in video are filter or future map?Sorry i confuse

[@shirleyhe4941]

I guess the pictures are feature maps , not filters , so I am confused also .

[@Mia-vz6yt]

Thanks for the video. But may we have the code that you used in the video?

[@deeplizard]

Hey Mia - The code is based on the blog referenced at the start of the video: blog.keras.io/how-convolutional-neural-networks-see-the-world.html

[@giovannisinclair9785]

What are the requirements of the convolutional neural net?

[@deeplizard]

Hey Giovanni - Check out the previous video and blog where this is explained: deeplizard.com/learn/video/YRhxdVk_sIs Additionally, the video and blog below explain even more technical details regarding CNNs as well: deeplizard.com/learn/video/k6ZF1TSniYk

[@mavee_shah]

Deeplizard: thanks for watching the video Me: No thanks for even existing and bringing this content to my life, you're a blessing to be found by anyone so thankyou!

[@freakphysics]

I love you girl, we should go out.

[@spamspamer3679]

has anyone a good German or English video or website for learning the dot product of matrices?

[@deeplizard]

I elaborate more on this in the corresponding blog in the section "a note about the usage of the dot product": deeplizard.com/learn/video/YRhxdVk_sIs