Loading in your own data - Deep Learning basics with Python, TensorFlow and Keras p.2

[@AnimilesYT]

5:04 I don't know why, but the way you casually said "ha, bluedog", and then continue on, was hilarious to me xD

[@mirkx7382]

You are one of the most chill and laid-back smart teachers i ve ever seen. such an informative tutorial. Thank You :)

[@mockingbird3809]

These were videos that I requested. Please make more Project videos in Machine learning and deep learning videos and real-world machine learning projects in PYTHON because You Are The Best to learn from

[@MSFTSTRIO]

This is so true. I've been so frustrated trying to learn this topic and a lot of videos are just people explaining what neural networks are, like the begging of the previous video in this series, but nobody actually gets into the code and explains how to set up functions. Like there is A HUGE DIFFERENCE BETWEEN DRAWING A NEURAL NETWORK AND ACTUALLY CODING ONE!!! ANYONE CAN DRAW ONE

[@kaushilkundalia2197]

So true ! Make more such project videos. They prove to be of great help.

[@a.n.7338]

Can someone tell me what X=[] and Y=[] is used for?

[@anwayeerc]

@@a.n.7338 Yes I do want to know

[@bennri]

@@a.n.7338 That's to initialize it to an empty array. But at 15:30, he learned that it did not initialize the type. and that's something I especially like about his videos: he doesn't just edit out the mistakes, any talks about things like the need to reshape being kind of stupid.

[@uditsingh9576]

Just wow , didn't wanted to watch the whole video , but your are a magnet !! Excellent style of teaching

[@Taha-jj1kr]

A nice alternative to pickle is np.save('features.npy',X) #saving X=np.load('features.npy')#loading

[@sentdex]

Thanks for sharing!

[@flosset9640]

thanks

[@dineshgandhi2255]

Thanks

[@vanluutran726]

very helpful

[@rohitborra2507]

thanks

[@seza1231]

For those who want to use RGB/color images, modify these lines! Change these: img_array = cv2.imread(os.path.join(path,img) ,cv2.IMREAD_GRAYSCALE) plt.imshow(img_array, cmap='gray') X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 1) To: img_array = cv2.imread(os.path.join(path,img)) plt.imshow(img_array) X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 3) And this should work, good luck!

[@mahmoud-khaled-abo-elmagd]

you are the man thanks

[@PubgMaan007]

Thanks alot for the videos! Great way to start my Data Science Journey. Really grateful to you for posting such content for free.

[@sentdex]

Thank you for the super thanks, best wishes to you on your programming journey!

[@minjaewoo5243]

This is currently the best TensorFlow tutorial on youtube. Can't express how thankful I am, after wasting much time on crappy videos named TensorFlow in 10 mins, 5 mins, 1 min and etc...

[@GilLianni]

Epic moment: "haa, blue dog" @ 5:05

[@AaqibMehrban]

hahahahaa i laugh so hard

[@janbiel900]

The value of these videos is fucking incredible. After some setup with anaconda to get tensorflow and python 3.6 to work in pycharm, i was able to reproduce all of this with my own data. Your explanations are absolutely on point and i have no questions left after this part.

[@sentdex]

Thanks, that's awesome to hear!

[@ssagonline]

Works like a charm. For those(beginners like me) who had issue in layers like "kernel size not defined". Just replace "(3,3 ) by kernel_size=3 " in layer 1 and 2 and it will be good to go.

[@quanghuyngo7556]

Hi Harrison. You've been doing an absolutely amazing list of implementating Deep learning videos with Python, Tensorflow, Keras, etc. This is the most useful job you've ever done. I've learned the Machine learning, Deep learning theory easily but implementation and application is something difficult to me. Keep doing this please.

[@KHM95]

Here's a course you'll need. Face Mask Detection Using Deep Learning . It's paid but it's worth it. khadymschool.thinkific.com/courses/data-science-hands-on-covid-19-face-mask-detection-cnn-open-cv

[@Xaminn]

Another amazing set of tutorials. You truly are helping me understand Python and Deep Learning at a whole different level. Thank you for your time and expertise, Sentdex.

[@bartoszturkowyd3608]

I love the way how you personified the NN. The part with shuffling makes me laugh!

[@PepeTostado]

I keep Getting: error: (-215:Assertion failed) !ssize.empty() in function 'cv::resize' What does it mean, could not find anything on the web, help!

[@KHM95]

Here's a course you'll need. Face Mask Detection Using Deep Learning . It's paid but it's worth it. khadymschool.thinkific.com/courses/data-science-hands-on-covid-19-face-mask-detection-cnn-open-cv

[@akinyemisodiq7081]

i started my python journey with you back in the university days, thanks for being there boss.

[@niclaswustenbecker8902]

Great tutorial, but the way you load the data is not very memory efficient and this will cause problems with large datasets. First the training_data list is written into RAM and afterwards the same amount of memory is reserved when converting into a numpy array. So this approach is only good for datasets < RAM size/2. Another option would be to create the numpy array at the beginning using np.empty and then write the data as entries into the array. This way the dataset can be as large as your RAM. If the dataset is larger than the RAM size it is suggested to use a generator that loads and yields the data during training. This way your dataset can be as large as your SSD, but training speed is most likely limited by the read speed of the drive. Just something I had to deal with during my thesis in the last couple of months. Maybe you could make a tutorial on the generator one, not a lot of people know about this. Anyways, keep up the good work!

[@will1337]

This looks very interesting and I'm experiencing some errors with this as well on my thesis. Can I contact you via email about this?

[@niclaswustenbecker8902]

Sure, can you contact me via youtube? Or post you email and I will contact you

[@liveleaky7571]

@@will1337 delete the comment with your email

[@stewie055]

​@@will1337 Did you fixed your problem? my python returns "MemoryError" when doing np.array(X).reshape(-1 , IMG_SIZE, IMG_SIZE, 3) step I'm doing the colored version (so I have about 3 channels of colors which is causing me trouble)

[@will1337]

@@stewie055 I did fix it with changing my sampling rate of my data. Maybe resize your images? I am not sure how to fix it with image data, sorry.

[@stephenderosa2412]

"the hand of a dog", so wise sentdex. forever indebted

[@r00t_sh3ll]

amazing videos, great AI tutorials, honestly one of the best programming channels on KZbin. thank you for making these videos

[@PandoraMakesGames]

Yeah, I also really enjoy his videos. He inspires me to make my own AI videos.

[@Huguillon]

1:52 I was taking it serious till the mug appears

[@gokulanand685]

THANK YOU SO MUCH!!! I just started with Machine Learning and Neural Networks and this video helped me a lot!!!

[@afailable]

Sentdex: understands neural nets Also Sentdex: doesn't know what to call a dogs paw

[@franklydoodle350]

More like neural pets lol

[@hurolinci5986]

Thank you so much for this video. As a programmer who just wants to start prototyping a simple model without a great DL background this video gave me the tools to get on with my work.

[@robertaradi9994]

This series is so thorough and easy to understand (for me at least :D)! I can't wait for the next part!

[@sentdex]

Great to hear!

[@liamfmackle]

6:50 "the hand of a dog" - it is called a paw! hahaha

[@benjamin-dubreu-data]

11:54 that's a great imitation of a model trying to learn ^^

[@KHM95]

Here's a course you'll need. Face Mask Detection Using Deep Learning . It's paid but it's worth it. khadymschool.thinkific.com/courses/data-science-hands-on-covid-19-face-mask-detection-cnn-open-cv

[@soeaung1984]

You are very good in teaching and the world need you, Sir.

[@jamesriri1810]

Sentdex these are the best videos I have ever seen in Deep Learning. Amazing tutorials. You are the best at what you do. Why did it take me long to find this channel.

[@sentdex]

Welcome here :D

[@jamesriri1810]

@@sentdex can I have your email if you don't mind?

[@jamesriri1810]

@@sentdex Hello. I got stuck when instructing my directory on the file. Kindly advise. Thank you.

[@malikadabare3719]

one of the best programming channels on KZbin. Subscribed and hit the bell ; )

[@sentdex]

Thanks!

[@RickertBrandsen]

These vids always cheer me up :) You are by far my most favourite instructor. :) When I feel depressed i just watch your videos.

[@sentdex]

Nice to hear :)

[@FurkanGulsen]

I've been looking for ways to upload 40k images to my Drive for 3 days. You in one word: you are perfect

[@dhruvagrawal3856]

you can download google drive on ur machine and sync it with your drive

[@trivukhac2965]

I like the teaching style, it's simple to understand

[@suleimanmustafa1473]

Thanks have been looking forward to this tutorial will help with my thesis. For windows, if you have anaconda installed and cannot find module cv2, you may simply have to do: pip install opencv-python if you are on linux you can do : pip install opencv-python

[@randytucker3083]

If you are having trouble here use anaconda prompt. This is in the Anaconda Manager where you start jupyter. Then simply type in pip install opencv-python and mine at least worked great

[@MrSpaceboyy]

opencv-python is installed, but it cannot find the module cv2

[@FatihKarakuzu]

Heay sentdex pls keep up with your videos. They are really helpful in so many ways. Im just starting to get into ML and started studying Computer-Science just because of ML and your videos are so helpful. Thumbs up to you

[@aakashdusane]

At 14:00 is there any reason why we dont just do: training_data = np.array(training_data) X = training_data.T[0] y = training_data.T[1]

[@rubenuribe]

this video is amazing, I am so glad I found your channel. I have tried learning this stuff for quite a while now through other KZbin videos but nobody could explain it that well.

[@aravindswamy7631]

@Ruben can u share the code

[@aravindswamy7631]

@RubenUribe through via email if possible

[@georgeseese]

Thanks. On Windows, I had problems with your DATADIR="X:/Datasets/PetImages" @2:10. I had my own version of course, but the code said (in effect) my path was not valid, even tho it was. I discovered Colab runs Linux-style OS. There are several methods of doing it; I used the Google Drive mount and zipfile to extract (instead of the Windows File Explorer Extract), ending with DATADIR=''/content/drive/MyDrive/datasets/training_set'. I finally got to see the gray dogs!

[@GelsYT]

This video was 2 years ago! Hey sentdex! THANKS :D

[@fun-ih5sc]

seriously you teaches better than my professors Thanks for teaching us. :)

[@PraYogiz]

i enjoyed this so much, i was from CS degree. But not have quite good moment with programming. So i decided to get job that not programming. But, since i was try to learn about pyautogui and selenium from your video, i was so exited to learn ML, and now here am i ... following your keras tutorial :D

[@michaelberna987]

Thanks Snowden, nice tutorial.

[@joewannimozzarella]

Can you show us how to make a image generator with own pictures?

[@neemaradhalalneema5418]

Hi how to use this method if we have 4 different categories?

[@veer1472]

Guys, the kaggle dataset that he is referring to no longer has folders named as cats and dogs... there are 2 folders, one is for training and another one is for test. You've gotta loop through images in the training folder and assign the labels using the image name

[@lemyul]

finally, I got through this video without any error

[@marcosbeliera1]

Hey sentdex Thanks for the video ! I can't see what you did to reshape the "y" list at the end of the video @ 16:20 ... Could you please clarify this ? Thanks again !

[@engr.inigo.silva2000]

Brother, you're amazing. This video has been a huge help. Thanks.

[@Nusiq]

14:42 I had to change both X and y into numpy array to make it work. y as a list didn't work.

[@kamalmoussa2239]

Did your model still work in the end? Im having this issue now

[@omarpasha2968]

So far, so good! The first dog grayscale image was successfully displayed. I was getting nervous there for a minute! I got confused when you added all that space at the bottom. It threw off my Jupyter notebook. I followed you thereafter, but my output did not print. We'll on to the other videos from other channels. I got to keep moving on. It was good while it lasted.

[@PlaneToTheBrainES]

plt.imshow(img_array, cmap="gray") plt.show()' shouldn't retrieve all images, not just the first one?

[@venkuburagaddaacc]

he had "break" in the for loops. So looped once and then "broke" out the for loop. That is why "img_array" has only 1 image data.

[@PlaneToTheBrainES]

@@venkuburagaddaacc Okay Thanks :)

[@lemyul]

@@venkuburagaddaacc ty

[@thefastreviewer]

Hey man. What should I write instead of the "training_data.append" line if I want a multiclass dataset? Yours has two classes, imagine I have a 5-class dataset.

[@coolstudios8273]

Great video! How could I modify this to use multiple categories for classification instead of just single category label?

[@robertue1]

Great video, looking forward to see the next one.

[@bibanez135]

Sentdex, I didn't understand when you did the reshape what the -1 exactly meant... you glossed over it a little bit. What does it exactly mean? Thanks

[@ricoturpisch357]

i also want to know, could some1 explain please?

[@nektoxyz1013]

Same problem! I ve stuck on that! I HAvent made it before, tried to make reshape after. But it didnt worked

[@joeywilmots1318]

It basically tells numpy the following: given all the other parameters IM_SIZE,IM_SIZE,1 figure out the other dimension (in this case the amount of images). its an automatic way of writting np.reshape(amount_of_samples, im_size, im_size, 1).

[@jamesdials13]

This was the most helpful video I've found. thank you!

[@neatpolygons8500]

10:22 what kinda cpu you have that could iterate over all of those images in few seconds :D

[@siddhisalvi1628]

Hello is it necessary to print all the images like it is printing only one dog image what about the others ? I am doing orb detection does it required to loop through all images?

[@manishanikam9325]

I like your way of teaching.

[@deepakathirvel8023]

It was really a very useful video...Thaaank u very much for ur timely help

[@kaushik5014]

Thanks @sentdex, just what I was looking for :)

[@garrettsolarsystem2585]

Great tutorial! However, I got a question. What if you have an image in multiple categories. So you could be sorting images based on size and colour and you stumble on an image that is red and big.

[@shubhamkedar508]

Can you make video on how to create raster(.geotiff) dataset in python

[@hpsalway6538]

Please make a video on how to load in the iam dataset.

[@pawegoebiowski1641]

16:25 There isnt problem in the kernel, I restart it and still have same issue. Dont know how it repair

[@내능지어디감]

Same here...

[@pawegoebiowski1641]

@@내능지어디감 I think i found solution, just enter anaconda cmd and type "install cv2" (or any lacking library) it helped me

[@내능지어디감]

@@pawegoebiowski1641 Well, that didn't work for me. I think I should post it on stackoverflow.... but thanks anyway :D

[@seth8141]

What is your opinion on setting an aspect ratio and adding padding during resizing? I just feel like forcing an n x n dimension distorts images too much when we have the varied original resolutions.

[@abbasmurtaza6085]

why dont u use cv2.imshow()???? 4:40

[@joedoe4484]

Hi, great work! I have a question, though, upon the "homework challenge" ! reshape(-1, IMG_SIZE, IMG_SIZE, 3) pops a ValueError: cannot reshape array of size 239640576 into shape (224,244,3). What's your opinion and solution ?? Thank you

[@GJ-bq9hd]

Really interesting and objective explanation of the topic! lol'd hard bc of the sudden blue dog

[@prophotographytml8122]

What are the things to do and consider to prepare my own data images. I'm a beginner.

[@cemlynwaters5457]

Sir your videos are epic! You are an excellent teacher

[@Kojifra]

How to load the image data set from my computer and split into two data set for training and testing?

[@shanerooney7288]

*sentdex* and *DeepLizard* have both been _VERY_ helpful with teaching me how to program. Thanks.

[@atharvapagare7188]

Amazing video man... Looking forward to the next one

[@user-yr1uq1qe6y]

It seems like all the videos and tutorials on this topic only deal with binary situations. Outside of the Keras docs on flowers there is a lack of variety on multiple classification approaches (> 2 classes). I have a feeling that might be where complexity and accuracy dive off a cliff.

[@eyosiyastibebu6102]

sir how i appy data Augmentation when i use this loading system

[@thefastreviewer]

Exactly the video I needed! Thx bro!

[@monkeysaregreat]

I suggest using context managers for file opening. Cleaner and is better for beginners as you don't have to remember to close the file

[@PepeTostado]

I keep Getting: error: (-215:Assertion failed) !ssize.empty() in function 'cv::resize' What does it mean, could not find anything on the web, help!

[@nikitavozniuk3087]

check path to image

[@nikitavozniuk3087]

it is because resize cant run on none image

[@PepeTostado]

@@nikitavozniuk3087 thx

[@prasmitpansare]

One of the most helpful video I came across as beginner. I still have not found anyone discussing how to create our own dataset and label them. I have 5000 PDF which I have converted to text and I am lost now, I dont know what to do here on, can someone give me a direction ?

[@ShakirKhan-th7se]

What is the best way to get input from different folders with different numbers of images each?

[@danielcox8306]

The first video was great, looking forward to watching this one through as well. Can make a video about using CPU vs GPU for some of these training processes? I would like to learn more about forcing the script to use the GPU for running instead of the CPU. For instance some of your older videos (like the Monte Carlo Simulation series) could benefit from this. Thanks!

[@sentdex]

To use the GPU, you just install the GPU version of TensorFlow. Depending on your OS this is slightly different, but: Windows: kzbin.info/www/bejne/qGiQiIOYrZ2Jq7s Ubuntu: kzbin.info/www/bejne/n6CZcp2car2el68 Obviously now you do the later version of TF and the correct matching CuDNN and cuda toolkit. Currently Cuda Toolkit 9.0 and CuDNN 7.0.

[@KashifShaheed]

X.append(features) AttributeError: 'numpy.ndarray' object has no attribute 'append' I got this error while preparing the dataset @sentdex how can i resolve

[@muratpq]

Hey, because your X variable is numpy array, you need list. X = [ ] # these are list y = [ ] # list and list has attribute "append".

[@MartinZatko]

you just need to write the :X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 1) out of the for loop.

[@DemonSlayer627]

If your using keras you should use the flow_from_directory function ,it's really the same thing without the hassle of running out of memory trying to load the entire dataset.

[@yanliu8152]

Hi Sir, I found you posted X, y empty lists to store the features and labels. However, I have no idea how you distinguish the corrected features and labels from training datasets. Please kindly advise me. Thanks in advance!

[@joshuachan2097]

Sorry I am new to this topic. I'm kind of confusing for the last part, the last row the X[1] can I say its the image and y[1] is the label for that image? Till the last row, we actually already done the data training and by reading the X and y the machine start to do prediction? Is that all for the cats and dogs machine learning? Look forward to some answers, thanks in advance!

[@Hysteresis11]

At 4:46 getting message that 'img_array' not defined, same exact code...

[@andihaki]

any tutorial for custom face keypoint training models?

[@giggetto71]

Great video! Got my own 2 category example working in minutes.which is fantastic! I am just getting crazy extending it to 3 category..what I did (apart from adding the folder with the 3rd category images): 1- add the 3rd category to CATEGORIES vector, 2- change "model.add(Dense(1))" to "model.add(Dense(3))" 3- change "model.add(Activation('sigmoid'))" to "model.add(Activation('softmax'))" 4- change from "loss='binary_crossentropy'" to "loss='sparse_categorical_crossentropy'" when I run the model fit, i get a good accuracy more that 0.9 but each prediction gets [0. 0. 1.]]. any clue? thanks!!!

[@Quinten-sn6it]

you figured it out yet?

[@Darkzzzz99]

i try to make my own model with 8 labels but my y.pickle is empty when i try to run it it says run out of output. Anyone know how to solve this?

[@georgeseese]

Re: Changing the X list to X array @16:35 "X = np.array(X).reshape(-1, IMG_SIZE, IMG_SIZE, 1)". This combines (conversion of list to np array) and (reshaping). (a) I can not find explanation of this syntax with 4 parameters. Any help? (b) The '-1' has various meanings in array reshape. What does it mean here? (c) edit: removed (d) The first array in X list starts as: [102 104 62 ... 69 75 83] (50 elements in dim). But X array starts as: [[[[102] [104] [ 62] ... [ 69] [ 75] [ 83]] One element in dim. Is that correct? (e) The last parameter is 1. Where does that show in the X array? (f) To simplify these questions for debugging, I used img size of (3, 2) (width, height) giving an array shape of 2r x 3c. And I process only two images, skipping random.

[@georgeseese]

After changing code to handle color, the "light" appears. (b) The '-1' says to flatten the array. With color, there are 3 elements in each dim. (e) The last parameter '1', as you mentioned is for gray images (one value per item). When using color images, change this to 3. First part of X array starts as [[[ 43 55 78]. Hope this helps.

[@IslanderIntelligence]

Thank you so much this video helped out so much with an up coming video of mine

[@heshamibrahim3414]

the plylist is amazing, however i came across this issues after running part 2 and part 3 back to back.. the y also needs to be an array, so the model.fit in part 3 can run... thank you once more :)

[@honzaadamek5186]

Great tutorial. Looking forwart to next part!

[@sentdex]

Expect it tomorrow!

[@vizotechz6104]

hi can you tell me if we have more than two classes then what to do. As I have a data with 10 classes.

[@NickxHaruka]

Hi, have you found an answer to this? I'm wondering too.

[@marcosvolpato8135]

Cant wait to the next video!! congratulations!!

[@angelikaprzeliorz8325]

Where are we using here the code from the first film? What about it?

[@davidg4893]

how would i convert this to using layers then giving me a list of the top 5 predictions if i add 10 categories and for instance defining a kitchen and a bathroom and whats in them like plastic bottle , glass, cup, person, food kind of thing ?

[@kevinmikez37]

hey bro i was wondering if you can do a 3rd part working with this dataset in an ANN, would be nice.

[@rezahosseini7851]

I did convert my x to a no.array without reshaping it as you did and I got the exact same shape as you did! So I guess you no longer need to reshape it to (-1,50,50,1). plz tell me if I've done something wrong

[@Olejika_CheatEngine]

4:18 For me it says "No such file or directory". Help

[@tanmaysharma2742]

hey can anyone help me at 3:23 , what if i don't want to convert it into gray scale, should i just skip that step?