Build a Deep Audio Classifier with Python and Tensorflow

[@sheikhshafayat6984]

This is exactly what I was looking for the past one month, and suddenly popped up on my recommendation! Can't thank you enough for this. You saved my semester!!

[@captainlennyjapan27]

1 minute into the video, absolutely amazed by the high high quality of this video. You are my favorite programming youtuber along with FireShip and NomadCoders! Thanks so much Nicholas!

[@captainlennyjapan27]

41 minutes into the video. Not even a second I was bored. Amazing

[@IronChad_]

You’re the absolute best with these tutorials

[@guillaumegalante]

Thanks so much for all these great tutorials! I’ve discovered your channel a few days ago, your way of teaching makes it really easy to understand and learn. I was wondering if you’d be able to do a series or video around recommender systems: building a recommendation engine (content-based, collaborative filtering), rather Netflix (movie) recommendations, Spotify’s music recommendation (could include audio modeling) or Amazon (purchases) predictions. Many thanks! Keep up the amazing tutorials :)

[@NicholasRenotte]

Definitely! I’m doing my own little deep learning challenge atm, will add it to the list!

[@prajiltp8852]

Can we use same if I wanted to seperate my bpos call recording from a conversation files, like if I train it based on my bpos recording and after that if I give a audio will it seperate my bpos sound?? Please help

[@dwiechannel3196]

@@NicholasRenotte please answer my question, I really need some direction.🙏🙏🙏

[@adarshd249]

Another great content from Nick. Thrilled to do a project on this

[@NicholasRenotte]

Yess! Let me know how you go with it!!

[@Maxwell-fm8jf]

I worked on similar project on Audio classification hooked up on raspberry with some sensors three months ago but using rcnn and librosa. A different approach from yours basically the same steps. Thumb up mate!!

[@NicholasRenotte]

Woahhh, nice! What was the latency like on the rpi? Noticed when I started throwing more hardcore stuff at it, it kinda struggled a little.

[@farhankhan5951]

What you have developed in your project?

[@ellenoorcastricum]

What where you using the pi for and have any tips on how to make a system that recognizes certain sound in real time?

[@gaspardbos]

Mc Shubap is spinning the decks in your memory palace 😆 Great tutorial so far.

[@rachitjasoria9041]

A much needed tutorial !! btw can you make a tutorial on tts synthesis? not with pyttsx3... train a model to speak from provided voice data of a human

[@NicholasRenotte]

You got it!

[@rachitjasoria9041]

@@NicholasRenotte 😃

[@lakshman587]

This video is Awesome!!! I got to know from this video that we convert Audio data to image data, to approach audio related tasks in ML!!!

[@abrh2793]

Nice one! Looking forward to a multi label text classification if you can! Thanks

[@NicholasRenotte]

Yup, code is ready, should be out this week or next!

[@abrh2793]

@@NicholasRenotte yo thanks a lot! The way you get inputs from the community and interact is nice to see

[@ChrisKeller]

Super, super helpful getting my project off the ground!

[@enzy7497]

Just discovered this channel on my recommended. Really awesome stuff man! Thanks for the great content.

[@thewatersavior]

58:00 - Another great one, thank you, already looking forward to applying. Quick question - why mix the audio signals on the MP3. I get that it gets us to one channel - is there a way to just process one channel at a time. Im imagining that would allow for some spatial awareness in the model? Or perhaps too many variables because we are just looking for the one sound? Thinking that it would be useful to associate density with directionality... but not sure that's accurate if the original recordings were not setup to actually be directional...

[@cadsonmikael9119]

I think this might also introduce distortion in the result, since we have to deal with stereo microphone separation, ideally about 100-150mm for human perception. I think the best idea is to just look at one channel in case of stereo, at least if the microphone separation is high or unknown.

[@davidcastellotejera442]

Man these tutorials are amazing. Congrats for creating such great content. And thank!!

[@luisalmazan4183]

Thank you so much for these tutorials, Nicolas. Will be great a tutorial about few shot learning. Grettings from México!

[@sederarandi1507]

bro you are absolute gold, thank you so much for all the effort you put on your videos and teachings +1 subscriber

[@pedrobotsaris2036]

good tutorial. Note that sample rate has nothing to do with the amplitude of an audio file but rather the number of times the audio file is sampled per seconds.

[@gregoryshklover3088]

Nice tutorial. A few inaccuracies there though about stft() usage: "abs()" there is not for getting rid of negatives, but for complex values amplitude. frame_length would probably better be power of 2...

[@orange-dd5rw]

how can i implement detection for example when the initial capuchin call started and ended how can i get this information in the end result( like in result it should show capuchin call - 2.3s)

[@gregoryshklover3088]

The classification works on sliding windows of fixed size (3sec in this tutorial). One can slide the window with overlap to try to approximate the start of the matching sequence, or use other signal processing methods to find start of the sequence.

[@12imr]

if anyone get error at 22:14 for pos = tf.data.Dataset.list_files(POS+'\*.wav') neg = tf.data.Dataset.list_files(NEG+'\*.wav') just use the / instead of \. pos = tf.data.Dataset.list_files(POS+'/*.wav') neg = tf.data.Dataset.list_files(NEG+'/*.wav')

[@TheHearts567]

thank you

[@henkhbit5748]

Awesome sound classification project👍I need a capuchino break after hearing the capuchind bird sound😎

[@ronakttawde]

Very Cool video Nick Bro!!

[@NicholasRenotte]

Thanks homie! Good to see you @Ronak!

[@DarceyLloyd]

Great video. Would love to see a version of this done using the GPU, with multiple classifications, not just binary.

[@0e0]

Tensorflow has GPU builds

[@mosharofhossain3504]

Thanks for such a great tutorial. I have a question: What happens when resampling is done to an audio file? Does its total time changes or its number of sample changes or both changes or it depends on specific algorithm?

[@andycastellon919]

Us humans can hear up to 22kHz approximately and due to Nyquist frequency, you need to sample it twice as its higher frequency, hence that 44100Hz you may have seen. However, on audio analysis, most useful data is found in up to 8000Hz, so we resample it up to 16000Hz, losing the rest of higher freq. The length of audio does not change. What changes is the amount of bits we need to save the audio.

[@stevew2418]

Amazing content and explanations. You have a new subscriber and fan!

[@NicholasRenotte]

Welcome to the team @Steve, glad you liked it!

[@primaryanthonychristian2419]

Bro, great video and very good detailed explanation. 👍👍👍

[@supphachaithaicharoen7929]

Thank you very much for your hard work. I really enjoy the video.

[@GuidoOliveira]

Incredible video, much appreciated, on the side note, I love your face cam, also audio is excellent!

[@vigneshm4916]

Thanks for a great video. Could please explain why we need tf.abs in preprocess function?

[@Uncle19]

What an amazing video. Definitely earned my sub.

[@oaydas]

Great content, keep it up man!

[@周淼-k3u]

Thank you so much for these nice tutorials! They are quite helpful! I have a small question. I saw your process of building up models and training and testing them. If I want to spend less time in classifying the model, do you think it's possible to introduce some existing datasets such as esc-10 or esc-50 in your method?

[@tatvamkrishnam6691]

23:30 What is the significance of that len(pos) or len(neg)? When len(pos) is replaced with 2 , I expect only first 2 sample data to have '1' label. However when I run -> positives.as_numpy_iterator().next(), I get '1' labelled not only for the first 2 samples but also for the rest.

[@benbelkacemdrifa-ft1xr]

It's a very interesting video. But can we do the test using sound sensor?

[@dimmybandeira7]

Very smart! can you identify a person speaking in the midst of others speaking more quietly?

[@ayamekajou291]

Hey nicholas, this project is great but how do i classify multiple animal calls using this model? I can classify the audio as capuchin or not capuchin this way but if i included more audio classes, how could i classify the audio file as the animal as well as the number of counts ?

[@guruprasadkulkarni635]

can I use this for classifying different guitar chords' audio?

[@mayankt28]

If you're encountering a shape issue when calling model.fit and getting the error "cannot take length of shape with unknown rank," the solution might be to explicitly set the shape of your tensors during preprocessing.

[@MahmoudSayed-hg8rb]

can you elaborate further ?

[@johndaniellet.castor7189]

​@@MahmoudSayed-hg8rb great! @mayankt28, thank you very much for your insight! basically, we can add the line: spectogram = tf.image.resize(spectogram, [1491, 257]) right before the "return spectogram, label" line in the preprocess() function

[@anthonylwalker]

Another great video. I love the setup and style of your videos now, have to love a good whiteboard! I'd be interested in seeing a drone video - I've recently got a Ryze tello. Enjoying the python package to control, and computer vision capabilities that come with it!

[@NicholasRenotte]

Definitely!! I've got one floating around somewhere, will get cracking on it! Thanks for checking it out @Anthony!

[@asfandiyar5829]

Had a lot of issues getting this to work. You need python 3.8.18 for this to work. I had that version of python on my conda env.

[@chamithdilshan3547]

What a great video is this. Thank you so much !!! 😍

[@urielcalderon1661]

It's him, he is back.

[@NicholasRenotte]

Ayyyyy Uriel!! What's happening!! Thanks a mill!

[@urielcalderon1661]

@@NicholasRenotte Always faithful man, while there deep learning tutorials we will be there

[@NoMercy8008]

LET'S GO, NICK :D This is actually pretty awesome and once again one of those things that i feel you can do TONS of different things with. Voice commands and animal calls are obvious examples, but maybe you could build a device that listen for a human's breath and heartrate and stuff like that and detects irregularities? This could be used for diagnostic purposes, but also as a warning device for for example elderly people. The moment it hears weird heart or breath sounds or something, it gives a warning and tells them to see a doctor. Same thing can be applied in a bunch of different fields, i think. Listening for weird engine sounds for example to help diagnose engine problems before internal parts suddenly and violently become external parts. Also, astronomy! Listening for gravitational wave events and things like that, though I'm pretty sure they're already using tons of AI for this anyway, so it's probably being done already. By the by, you posted about crowdsourcing labels/labeled data the other day, i think that's a great idea, especially if you're sharing that labeled data with the public! Doing it this way is a much more manageable way to get labeled data since hopefully the work is being done in parallel by distributed resources ("many humans") and sharing it online means that this project essentially helps everyone wanting to play around with ML and use the data for something cool, learn about things and maybe come up with awesome ideas to change our future. So, awesome idea and great way to leverage the outreach you have here on YT, I love it! ❤ What i really liked about this video in particular is the exploratory data analysis and the preprocessing alongside it. As we all know, it's very important to feed your data to your network in a way that makes it as easy to digest as possible, so learning more about this is absolutely essential and really fun aswell! Much appreciated! As always, Nick, thanks a ton for your videos and for doing all this for us, much much appreciated! Really love this video and looking forward to the next one! All the best, have a great week! :)

[@NicholasRenotte]

Heyyy, I had a good laugh at this comment "before internal parts suddenly and violently become external parts" 😂 but yes definitely agree! I'm hoping the crowdsourced labelling can become a thing! I know there's datasets out there but for a lot of the niche and practical stuff I have in mind, I can't really seem to find anything. I figure if people are willing to help label then I can give back to the community by showing everyone how to use and build with it! Have an awesome weekend!!

[@empedocle]

Amazing job Nicholas!! I have just a question, why didn't you calculate also the standard deviation of files' lenght so to have a more precise interval for your window?

[@riyazshaik4006]

Thanks so much sir, one request sir can you explain about how to classify audio as positive, negative and neutral

[@thetechmachine5446]

Why we need to calculate Mean , Min and Max in 30:20

[@MahmoudSayed-hg8rb]

to kind of pick a reasonable threshold to the length of our audio files as they vary in length and ML Models often expect equally sized inputs. so we chose 48000 as a threshold. Everything beyond that is either cropped out if it's longer or zero padded if it's shorter

[@ellenoorcastricum]

Is it possible to run this while i have my mic always listening and to do live proccesing on that? Btw this will be my first project and i know its a lot.

[@Kishi1969]

Always given inspiration of new knowledge...You are Great Thank Advice please Please my question is that can I buy Graphic card of 2G(NVidia for starting Computer Vision because my PC is too slow when I'm using my CPU..

[@NicholasRenotte]

I would be looking at something with more RAM is you can swing it @Benya, 2gb is a little too small for most image computer vision tasks.

[@unteejo3678]

Use Google Colab’s TPU (Tensor Processing Unit). It’s very fast if your model use only CNN.

[@Kishi1969]

@@NicholasRenotte Thanks 🙏

[@insidecode]

Amazing job

[@Uebermensch03]

Thanks for uploading a great video and one question! You sliced an audio file from the very start of the file. I think the position where we start slicing can affect the model accuracy. For instance, if you skip 1s and start slicing, it may yield different wav data. Do you think slicing audio from the very start of the file is golden rule?

[@marioskadriu441]

Amazing tutorial. Really enjoyed that Nick 🙏🏼 I guess in case we wanted to detect multiple sounds from the same animal the procedure would be the same but we would need an equal number of samples to train the neural network? Furthermore in case we wanted to detect sounds from multiple animals and categorize them would we follow the same procedure just at the end we would put softmax instead of sigmoid ?

[@kavinyudhitia]

Great tutorial! Thanks!!

[@TheOfficalPointBlankE]

Hey Nicholas, I was wondering if there was a way to change the code to print the timestamps in the audio clip that each sound is recognized?

[@zainhassan8421]

Awesome, kindly make a video on Speech Recognition model using Deep Learning.

[@harsh9558]

This was awesome 🔥

[@ahmedgon1845]

Great video thanks so much, I have a small question, In the line Spectogram = tf.signal.sftf Why you choose Fram_step =320 Fram_length=32 Can some one explain the method of choosing this please?

[@matthewcastillo8775]

I need help getting compatible version of tensorflow and tensorflow-io. The latest release of tensorflow io is 0.35.0, however my os is saying that only up to 0.31.0 is available. My tensorflow is updated to the latest version and I have Python 3.11.6.

[@Varadi6207]

Awesome explanation. Please help me to create audio augmentation for health records without losing information. I worked with time_shift(-0.5 to 0.5 variation in the wav). But, model ACC is not up to the mark.

[@rajkumaraj6848]

@NicholasRenotte The kernel appears to have died, It will restart automatically. Got this error while running model.fit. How can I solve this?

[@akashmishrahaha]

Why are we reducing the sample rate from 44khz to 16khz, it was not clear to me?

[@gangs0846]

Thank you. Can you make a realtime Audio Classification?

[@vishalm2338]

How to decide the values of frame_length and frame_step in tf.signal.stft(wav, frame_length=320, frame_step=32) ? Appreciate any help !

[@Computer.Music.And.I]

Hello Nicholas, I have been using this great video in my beginners courses and last year everything was fine. Unfortunately in today's lecture the code did not run on any of my machines or configurations ... The load_16k_ wav function is not able to resample the audio files, and much worse, the model.fit function complains about an input that could not be -1, 0 or 1. Are you willing to check and update your code ? (Spend 6 hours now to find the error😊) Thx jtm

[@sanjay6013]

did u find the fix for this pls help

[@johndaniellet.castor7189]

@@sanjay6013 @MahmoudSayed-hg8rb made a great insight for this model.fit issue basically, we can add the line: spectogram = tf.image.resize(spectogram, [1491, 257]) right before the "return spectogram, label" line in the preprocess() function this is to explicitly set the shape

[@johndaniellet.castor7189]

i think i also had the issue of tensorflow-io not being able to be installed so what we did is we used alternative code instead, in my case: original_length = tf.shape(wav)[0] new_length = tf.cast(16000 / tf.cast(sample_rate, tf.float32) * tf.cast(original_length, tf.float32), tf.int32) new_indices = tf.linspace(0.0, tf.cast(original_length - 1, tf.float32), new_length) new_indices = tf.cast(new_indices, tf.int32) wav = tf.gather(wav, new_indices) return wav credits to @shafagh_projects for this!

[@malice112]

thanks for the great video, is it possible to use .mp3 files in python instead of .wav to save disk space?

[@jawadmansoor2456]

Thank you for the great comment. How do you classify multiple sounds in a file and get time information as well like a sound was made at time 5 seconds into the audio file and another was made at 8 seconds how do we get time and class?

[@mrsilver8151]

thanks for the great tutorial as always sir in case i want to make voice recognition to identify for which person is this voice is this steps will help me to reach that or do i need to look for something which is more specific to this task.

[@plazon8499]

Dear Mr. Renotte, I'm trying to use this tutorial as a basis to build a classifier over several music genres. The only thing that I don't know how to adapt is the last layer of the CNN. How should I modify it so that it can get me as output let's say 10 different labels ? Should the labeling be modified upstream ? (I want to have 10 outputs instead of 1 at the last Dense layer, but I can't just modify it like it, so I'm wondering how I should do it) Thanks a lot !

[@armelayimdji]

Since the time you asked the question, you have probably solved it. However, my guess is that for your multi class problem, you should first have data for the 10 classes (samples of each of 9 music genres, plus a non classified genre) and the last layer should be a Dense layer with 10 neurons activated by a softmax function (instead of sigmoid) that gives the predicted probability of each class. You also have to change the loss function to be one of the 'categorical crossentropy' available in tf keras.

[@plazon8499]

@@armelayimdji Hey Armel Tjanks a lot for the advice ! Obviously I'm done with my project and I went for something else : Instead of taking the spectrograms as input of my CNN, I extracted features from the sound wave and all the physical aspects of the music to have an input vector of features that I passed through an MLP and it worked well !

[@farhankhan5951]

I have similar kind of problem, can you help me?

[@faresbecheikh7052]

15:00 file_contents = tf.io.read_file(CAPUCHIN_FILE) is not working can you tell me why plz ?

[@HananAhmed0311]

@NicholasRenotte can we use same thing in human voice classification with this?

[@NuncNuncNuncNunc]

Maybe a basic question, but what does zero padding do when getting the frequency spectrum?

[@tatvamkrishnam6691]

Tried to recreate the same. Somehow the program abruptly stops at hist = model.fit(train, epochs=4, validation_data=test) It has to do with using lot of RAM. Anyway for me? Thanks!

[@karlwatkins7054]

did you solve this issue?

[@leventelcicek6445]

you are wanderfull sir

[@orange-dd5rw]

how can i implement detection for example when the initial capuchin call started and ended how can i get this information in the end result ( like in result it should show capuchin call - 2.3s)

[@GArvinthKrishna]

what approach is the best to find a number of blows in the recording of a Jackhammer?

[@SA-oj3bo]

If I want to count how many times 1 specific dog barks / day ? Then it is clear that samples of this dog barking are needed, but how many? And what other sounds must be sampled and how many if at the same place many other sounds can be heard? Thx!

[@NicholasRenotte]

Heya, I would suggest starting out with 100+ samples of the dog, you can then augment with different background sounds and white noise to build out the dataset. This is purely a starting point though, would need to do error analysis from there to determine where to put emphasis next!

[@SA-oj3bo]

@@NicholasRenotte It would be very interesting if an accurate counter can be made. This is a case of animal abuse ( dog in a small cache and neglected by the owner and barking for attention for over 2 years), so an accurate counter would be very helpful and usefull for other projects. What I not understand is why 1 sample/spectogram of the barking dog would not work good enough to detect it in a recording of for example 24h, because there must be very few or no sounds that have the same spectogram. I understand it will always be different but can 2 different sounds ( cat and dog for example) have 2 spectograms that are very similar? So my question is why it is not possible to identify a specific sound in a recording by comparing the spectogram of the sound to detect to all possible spectograms in the recording ms after ms? If you accept payed projects I would love your help, because this is all new for me. Regards!

[@NicholasRenotte]

@@SA-oj3bo I think you could. If there were multiple dogs in the sample you would probably need a ton of data though to be able to clearly identify which dog is barking. This would allow the net to pick up the nuances for that particular bark.

[@ChristianErwin01]

I've gotten through to the part where you start testing the predictions and my validation_data isn't showing up. The epochs run fine, but I have no val_precision or val_loss values. All I have are loss and precision_2. Any fixes?

[@SaiCharan-ev8hu]

hey nicholas,trying to execute this but facing issue as you havent done any preprocessing on the training data,looking for help from you

[@badcatprod]

1k! ) //THANK YOU! 🤗

[@rohithak9419]

I subscribed to this channel last October But today I see that is unsubscribed today I feel it's better to list all channels in a piece of paper and keep it in wallet

[@tims.4396]

Im not sure about the batch and prefetch part, for me i generates empty training sets afterwards and also it only takes 8 prefetched files for training?

[@konradriedel4853]

hey nick, i was rebuilding this project of yours now on a local machine - i was struggling with the spectogram size and memory allocation for my gpu rtx3070.. i downscaled to frame_length=160, frame_step=64 with input_shape=(748, 129,1) so far so good, now regarding the .fit of the model my training time is 50ms per epoch, final results tend to yours...did the colab train on cpu for the sake of memory with "highscale" spectograms maybe? could you run that locally and give some info? im very insecure regarding the train time of 50ms to the 3mins in the vid.. thanks anyways for the great tutorial man!!!

[@GGBetmen]

hello, can I ask you about this things?

[@kiss-bws]

Please make video on making very own speech recognition using tensor flow

[@carlitos5336]

Thank you so much!!

[@mendjevanelle9549]

Hello sir! I installed tensor flow as presented but I don't understand the reason of the error message,no module named tensor flow.

[@Lhkk28]

Hello Nicholas, thanks for you video :) I have a question I am aiming to build a model for sound detection using deep learning algorithms ( I am thinking about using LSTM). for now I am done with preprocessing step. I have the spectrograms of the sounds (generated using Short time Fourier transform) also I have the labels (binary labels as arrays, 0s where there are no events and 1s where the events are present). I am now confused about who to fed this data to the model. The shape of each spectrogram is (257, 626) and the shape of each label is (626,). How should I give this data to the LSTM. Can I build a model that takes the spectrograms with their current shape and give the labels as sequence of ones and zeros or I have to segment the spectrograms and give each segment a label?

[@dzulgonzalezmarcosadalbert6511]

good video, new subscriber, but is there a way to export that training model, to be able to use it from a python file that can do sound detection through the use of said model

[@md.abdullaalmamun3965]

Please make video on image segmentation (semantic and instance).

[@NicholasRenotte]

Coming soon!

[@eggwarrior5630]

Hi i am working with a new audio dataset which does not require audio slicing part? What should I modify to loop through the folder for the last part. Any help would be greatly appreciated

[@kundansaha2369]

i am getting a error and tried to debug so many way but not solve. Error is "The procedure entry point could not be located in the dynamic link library - to Positive and Negati a', 'Parsed_Capuchinbird_Clips') a', 'Parsed_Not_Capuchinbird_Clips') C:\ProgramData\anaconda3\lib\site-packages\tensorflow_io\ python\ops\libtensorflow_i0.50."

[@Sachinkenny]

What happens when there are multiple birds in the dataset. Now how good is a CNN model on this kinda dataset? Again the source training audio samples can vary in length, sometimes in minutes. How can we do the pre processing in such cases?

[@gauranshluthra7520]

How did you uploaded the file as colab does not support folder upload until it is in zip file format

[@smokinep]

Hi Nicholas, how much data would I need if I just wanted to train on a single type sound ?

[@asimbhaivlog108]

Hi I have to detect tree cutting voices after detection in a forest using iot can you make a detailed viedo on it and which hardware sensors and module can be used Illegal Tree-Cutting Detection

[@farhankhan5951]

Are you done with your project?

[@eranfeit]

Thank you Eran

[@HananAhmed0311]

Can we use same method to classify human voice?

[@UzairKhan-gs3nq]

How can we use Linear predictive coding in the preprocessing function of this code?

[@swanandkulkarni5162]

great video but can you tell if we have to build a non binary classifier, we'll have to label it with a vector instead of just one value. How to do it 😬

[@sarash5061]

This is amazing

[@paulj9833]

In the cell 'hist = model.fit(train, epochs=1, validation_data=test)' the Kernel crashes in my case. Seems to be a tensor flow problem. I tried to install different versions of tensor flow, it didnt work though. Does anyone have any advice?

[@mohammedyacinmeshi3398]

Hi Nick ! Thank you for your extremely useful contribution to the AI learners' community