Build a Deep Learning Model that can LIP READ using Python and Tensorflow | Full Tutorial

[@נעמהאסתרלוגסי]

This is amazing! Can it work in real-time? Not only on existing videos?

[@MiniProject-y5l]

This project is f***ing awesome.Can you make a video on finding accuracy for LipNet

[@amartyadebdas2881]

Brilliant step by step tutorial! You earned a subscriber and a fan. Looking forward to more.

[@HideyHoleOrg]

Do words need to appear in the training to ever be predicted, or can they be built up of letter predictions? The dataset would be more useful if the number was a numeral, rather than spelled out, since that part of the alphabet never gets used. I would love to see this built out, maybe even have a "Lip Reading Keyboard" to type with your camera rather than a mouse or microphone, as it would have many accessibility applications, especially in the workplace. Great channel, I have been watching many of your previous shows, and was so happy to see a new episode!

[@NicholasRenotte]

Heya Michael, nope it doesn't. In fact this IS a character based model so it could build up sentences of words it's never seen before. I need to get more data though to extend it out!!

[@whoonie4685]

I just discovered your channel and 1 video made me fall in love with your style of sharing! Simple and easy to understand! One project that I think will be interesting is automatic assessment with deep learning esp in computer vision field! (as i saw many people are doing this these days hehe) Keep up the good work & cant wait for a new video!

[@ArisFilms]

Your knowledge and delivery is unmatched, subscribed after one video!

[@JinxUS00]

Yes we want next part. It would be a perfect portfolio project and to learn alot of stuff. Thank you

[@rbanondo]

We want that tutorial with custom dataset. Please please please sir...... Also, great tutorial again. You are a great teacher.

[@NicholasRenotte]

You got it!!

[@giuseppedimaria6253]

I tried to generate a dataset by saving 'tot' number of video frames in which I say certain words 'hello' for example, isolating only the lips though I appropriately divided them into different directories, renaming them with the words associated with those frames and used a 'flow_from_directory' function which allows the folder name to be associated with this data as a label.

[@HarshilDangar-tc3ns]

@@giuseppedimaria6253 can you share the code

[@mramanidevi5646]

@@giuseppedimaria6253 Hey Can you share code

[@blancoarnau]

I guess lip reading is a very difficult task because by simply looking at a person lips is very difficult to 100% know what they're saying. Some words are easy to pick while some others can be easily confused with other words... I guess that the model may work well for vowels because the mouth position changes a lot, but when it comes to consonants, it may be more tricky. One good idea would be to predict the next word based on the previous one. For example, if the model predicts "the", the next word is very likely to be a noun. I guess this would help discern what word is next.

[@perceptronnn]

data.as_numpy_iterator error: For the people who are using colab, I used tensorflow version 2.10.1 In load_data(), use file_name = path.split('/')[-1].split('.')[0] For mappable_function, use this code: from typing import Tuple def mappable_function(path:str) ->Tuple[tf.Tensor, tf.Tensor]: result = tf.py_function(load_data, [path], (tf.float32, tf.int64)) return result (The tf.py_function call returns a tuple of two tensors (tf.float32 and tf.int64), not a list of strings.)

[@akshithchowdary6410]

And did you get and error in mimsave() ??

[@ghostplays5291]

My epoch execution takes hours even though I have rtx 3050 what to do

[@varakalachandrakanth6972]

@@akshithchowdary6410 hey bro i got the same error did you solve it ?

[@devez007]

Thanks! this worked.

[@suryasreemanth3387]

Thank you so much, It worked for me 😃

[@doomatyourservice2218]

Hey Nick, can you do a video on implementing the model with custom dataset?? Please

[@not_humorouss]

Yes please Nick

[@shivakumarbhatkere1807]

hey Nick, this was a very helpful video. Just a small suggestion, if you create a requirements.txt file listing all the version of the packages you used it will be very helpful, even years to come we can install the specific version that yu have used otherwise it keeps creating dependencies issues.😃

[@akshithchowdary6410]

did you face mimsave() issue? if yes how did you solved it?

[@kunalkumar2717]

by squeezing (reducing the dimensionality) and converting the data type of the original nested list passed into the function, it does solve the issue.@@akshithchowdary6410

[@datapro007]

Amazing! Nick's videos are phenomenal.

[@NicholasRenotte]

Dataprooooo!! Glad you liked it!!

[@giuseppedimaria6253]

Thanks for the help you are giving with the videos Nick anyway I'm studying to make Lip Reading work in real time, a bit like seen for sign recognition For now I will try to retrain the model with the other samples from the grid dataset and in real time i tried to build the prediction on the lips every 75 frames but it still doesn't work very well the predictions obtained by me are currently unusable I will continue to study waiting for more of your videos.. Let's see what will happen

[@harshiramani7274]

Did u manage to built the real time lip reading?

[@giuseppedimaria6253]

@@harshiramani7274 I created a model to make it work in real time, but it's limited since I used a custom dataset generated by me. It would take (in my opinion) more train cycles and a dataset with people with different characteristics (whites, blacks, Asians, children, elderly, etc.) saying every word in the dictionary of every language. Perhaps even a second model to concatenate the predicted words with the first model, and predict a complete sentence. I stopped working on it at the moment. I remember generating for each word in my dataset small videos where I always pronounce the same word from different angles etc., capturing 30frames from these videos.

[@harshiramani7274]

@@giuseppedimaria6253 Yeah I think that is fair I am thinking of trying to make this model for my local language any idea how I should proceed for the same?

[@chakerayachi8468]

duuuuuuuude i've been looking for such a channel like this since forever your amazing thanks for sharing all your awesome project for free

[@alx8439]

You're the most enjoyable software developer youtuber to watch. Not only you are doing amazing job, but you also share it with community. Because of guys like you I'm gaining back my trust to humanity. Thanks a lot!

[@NicholasRenotte]

Ohhh thanks so much @alx84!! Means a ton 🙏

[@KhushiGarg-xb9bp]

from where i can get the dataset used in this video?

[@ronaktawde]

It was awesome deep learning learning experience. Its a fantastic tutorial and the results are TP. its a ...BOOOM.....BOOOM...BOOOM situation after completing this tutorial. Thank you so much Nicks.

[@TanmayPrakash-gt5ll]

Hi @NicholasRenotte !...Beginner In ML here, I have a doubt in the last dense layer: model.add(Dense(char_to_num.vocabulary_size()+1, kernel_initializer='he_normal', activation='softmax')) we doing : char_to_num.vocabulary_size()+1, why are we adding '+1'...... char_to_num.vocabulary_size() includes the oov token implicitly

[@gogyoo]

What kind of data collecting are we talking about if we want to generalise to like, the 1000 most used English words? Could we build the model so that it can recognise phonemes from the preprocessed videos? Then the model associates the most likely phoneme separation to convert to the output string.

[@jerbky]

Pioneering in data science and machine learning! love the video

[@NicholasRenotte]

🙏 🙏

[@Seriosso]

You are the best! thanks Nic for these super quality videos!

[@NicholasRenotte]

Thanks so much for checking it out Siraj!!

[@addisonpratt8916]

This is amazing!! Would it be possible to do live predictions using a camera?

[@NicholasRenotte]

That's the gameplan! I'm still mapping out how to do it most efficiently...but definitely!

[@tamizhmalar9843]

@@NicholasRenotte pls do buddy, this is really amazing!

[@mikemurphyWebSurfinMurf]

Any update on real time camera effort?

[@ghostplays5291]

hey nick really need real time lipreading @@NicholasRenotte

[@samanthaeaton414]

​@@NicholasRenotte Were you able to map this out?

[@gdavisiv]

I was wondering where you went man!! Welcome back! :D

[@NicholasRenotte]

Ayyyy! Just took a few months off at the end of last year to get the course done. Back at it at full steam now!!

[@ghostplays5291]

​@@NicholasRenottehey Nick my epoch execution takes hours eventhough I have rtx 3050 what to do

[@CreatingUtopia]

Will this work in real time on any talking video or realtime@@NicholasRenotte

[@anjanb607]

Nick can u give the versions of tf and keras u used in the video not only me everyone are getting errors at epoch some are getting in load video functions please give the versions that u used nick i beg u olease

[@vinayshukla3800]

Awesome, your way of explaining is incredible

[@NicholasRenotte]

Ohhh thanks @Vinay, means a ton!!

[@curious_haldar]

This is really cool 😎😎😎....Would love to watch tutorials like these in future. Keep up the good work

[@vamsik1453]

Hey Nick! How were you able to find out the exact co ordinates of the lip region in the frame to extract them? Thanks for helping.

[@FireFly969]

I think just trial and error, he does it manually, but in real world, we need to make sure the model can also find and track the lips movements.

[@satyajeetshashwat4115]

How to get the dataset?

[@truthseeker457]

Read the description of the video

[@katu406]

This f***ing awesome!!! 🔥🔥🔥

[@NicholasRenotte]

🙏

[@zenfestics]

@@NicholasRenotte bro add time stamps to the video

[@lifeofcode]

This is why I am a subscriber, for amazing content like this.

[@NicholasRenotte]

YESSS, and I am soooo glad to have you here Jimmy!! More intense projects planned this year!

[@Jaybaas]

Thank you so much!! This is absolutely phenomenal 🔥🔥👏🏾

[@NicholasRenotte]

JABULANI!! Thanks so much for checking it out!!!

[@nazarzaki44]

Thank you, Nich! Building the App will be very helpful.

[@derekvolt920]

This is such an important application of ai. I subscribed right away and can see it being one of the most important advances to enery one.

[@RobertThomsonDev]

Hi Nick! I would love to purchase the tutorial but the link is broken and I can't seem to find it on your site?

[@rishabhbhardwaj3893]

in the model.fit part I am getting an error suggesting that the shape of theb frame is incorrect is there any solution to it ?

[@AreebaAmin52]

Hello Nick. Thanks for always uploading such insightful tutorials. If my model training starts with loss=inf (i followed your tutorial throughout), will it get better or their is something else wrong with my code/model? please guide

[@نشامي]

Imagine that in all KZbin this is the only video detailed at how to make this project lipnet lip reading. Just shows how this topics are not easy to found

[@2155-samikshashardul]

i tried following this tutorial but i faced an issue in this line - frames,alignments = data.as_numpy_iterator().next()

[@olu_the_ai_guy]

Awesome 🤯🔥 thanks alot nick

[@NicholasRenotte]

🙏 thanks a mil @Olowu!!

[@arshvahora4224]

Hey! I got some doubts in the project. Adding to that, the code you provided also has errors in it. Would you please share updated code?

[@chaitanyasaibeesabathuni928]

most awaited video ....thanks mate !!!

[@NicholasRenotte]

🎉 anytime @Chaitanya!!

[@NoorNoor-ki5dd]

We really need the app! 🤯

[@bvdlio]

Hey Nick, great video as always. I was wondering why the input is normalized, but the tokens representing each character in the output are not.

[@Madankhatri-ot1zw]

packages for this project: imageio version-2.23 tensorflow version-2.10.0 tensorflow-gpu verison-2.10.0

[@FireFly969]

This project is making me go insane, and iam very much happy too by its results again, its facinating, its even hard for me to undertand all those words that guy says in the videos. And whats crazy is it learns from the lips movements. Its just crazy, but i think if we want to make it be able to work with any video, to first be able to detect the mouth in the video we give it then be able to read the lips movement. ❤ Project and crazy ideas like this makes me fall in love with neural network more and more, thank you for this wonderful projext, have a nice day nicknacknock

[@ditsyc6939]

That was fantastic....fabulous brooo.......u are a real geniusthe god father of Machine Learning, deep learning

[@Football_moments8]

Hey nick can you make a tutorial on how methods are used

[@NicholasRenotte]

Like applications for this kind of tech?

[@Observer168]

Can you build a language model and python model? Would it take a entire team of engineers to do that or can one single person do it?

[@NicholasRenotte]

Ahh, I’m so far behind LLMs but I’m going to give a crack this year!!

[@srikanthkoltur6911]

Can you make short video covering jupyter notebooks tricks and extensions to easily use notebooks like execution speed time and other etc

[@NicholasRenotte]

Done, added to my list for this week!

[@srikanthkoltur6911]

@@NicholasRenotte sorry for not mentioning but i was looking forward for lip read video it is fabulous thanks a lot for you efforts ❤️

[@NicholasRenotte]

Ayyy, all good man!! Love that you enjoyed it!

[@c016smith52]

Absolutely awesome, mate!

[@NicholasRenotte]

Cheers Chris 🙏

[@manan5]

THIS CAN BE A BLAST IN FORENSIC ANALYSIS! We just need more and more dataaa! and fine tune it for different angles

[@NicholasRenotte]

I’m thinking of crowdsourcing some more data then throwing it up on Kaggle so we can make it bigger!! I’m going to get v1 of the new data model up soonish!!

[@manan5]

@@NicholasRenotte yay less go!

[@siddhanthbhattacharyya4206]

im getting an invalidargumenterror during the fitting of data into the model at the 1st epoch!

[@MrPhili10]

That's awesome! Do you think it would be possible for it to, instead of sentences, I train it to detect individual phonemes? That should actually work even better, since there are only 44 phonemes in the English language.

[@yasmeennour47]

amazing tutorial , helped me a lot quick question though, how can I use f1_score metric or accuracy metric for example with this network ??

[@ghostplays5291]

My epoch execution takes hours pls anyone solve my problem ❤ Me using rtx 3050 graphics card

[@maheshwarikuntal1493]

Great effort man.

[@tejassrivastava6971]

This is insane 😶‍🌫️🔥🔥. Can we get a tutorial for streamlit app of the same? Please.

[@NicholasRenotte]

You got it @Tejas!!

[@ProjectSaraswati]

Next Tutorial please Nicholas !!! Loved your content... Subscribing for greater cause....

[@mehulkumar6504]

Why did you choose fps=10 specifically for imageio statement? How to determine what fps value we have to put in?

[@rayyansyed2998]

tf.config is not working,What to do any idea...😥😥

[@kevynkrancenblum5350]

Damn ,That crazy ! Thank Nick for an other fantastic video ! Ur the best 🙏🏻🙏🏻

[@NicholasRenotte]

Ohhhh mannn, thanks a mil Kevyn!

[@SVSingam273]

@Nicholas Rennotte. Can you please do a full tutorial on YAMNet

[@ahmadshabaz2724]

Nick you are legend man.

[@NicholasRenotte]

Thanks a mil @Ahmad!! Appreciate it man!!

[@solomon_leo_27]

Brooo, you made my day! Thank you so much!

[@a1mm288]

Hi Nick , I have just started learning about machine learning and dp. What mathematical skills are essential for me to have a solid understanding in this field?

[@NicholasRenotte]

Top three are: linear algebra, probability and calculus!

[@a1mm288]

@@NicholasRenotte Thanks for the advice! I've completed linear algebra needed for deep learning and am now learning probability. Your help is much appreciated!

[@not_humorouss]

Hey, could you please do a video of this model working on a custom dataset?

[@harsh_the_walker]

Hey Nick can you tell me. how they video can be annotated?Which tool are they using?

[@pranav-patil]

Please create a streamlit app for this lipnet in which user can get output of lipnet of his custom video

[@NicholasRenotte]

You got it!!

[@happy-mo1qc]

what is the accuracy of your this model sir please reply someone please reply

[@joaobentes8391]

Could u make a tutorial on how to use TPUs from google colab to run reinforcment learning for example on solving like cartpole or some other envieemnet more complicated i guess. Thanks a lot!! Keep up the excelent work !! i share all ur vids in my team group in university! = )

[@VũQuân-u1n]

I have a small question, what is the differences between the DNN model you use above and RNN ? I'm doing a college project, hope you can answer me soon, thank you very much.

[@martinsilungwe2725]

Thanks for the content, I'm learning a lot from your videos sir... but I need help with translating the models into Tflite, with labels.

[@morgomi]

That looks neat!

[@NicholasRenotte]

Thanks a mil @Enes!!

[@deepakgahlot6879]

It would be great if you could create a tutorial on creating a custom data set.

[@YogeshKendre]

numpyiterator object has no attribute as_numpy_iterator

[@sunilb533]

Can you please do a tutorial on training on our own dataset/ videos..

[@thebusinesscentre]

we gave you a follow since you are doing good things for the whole

[@nithyashreej]

Helo sir may I known which dataset is used in this project...

[@deeppatel_]

how much time it is taking while training the model ?? btw its great project and i am gonna use it in my Smart Interview System

[@raj4624]

Hi NIck. can you make videos on MLOPS (series for begineers or as per your convience).?

[@NicholasRenotte]

Done, will see what I can do this week!!

[@namangoyal8477]

really liked the video, thanks for the tutorial, make a streamlit app too.

[@Sumonms]

Hey Nicholas can I try this pipeline to my custom dataset?

[@NicholasRenotte]

Sure can, with custom videos it should be fine. You might just need to tweak the load_alignments function to handle how your text is structured!!

[@abishekbudhathoki8222]

what should be done to such errors? FileNotFoundError: [Errno 2] No such file or directory: 'data/alignments/s1/.align' [[{{node EagerPyFunc}}]] [Op:IteratorGetNext]

[@jeyasrisenthill]

same error ..did you figured it out??

[@abuprincewill9205]

Thank you so much for this.

[@NicholasRenotte]

Anytime @princewill!!

[@meisuci4708]

Is it possible for other languages?

[@NicholasRenotte]

Definitely, would just need alternate language transcriptions!

[@wisdom_Lane]

If anybody is facing this error '_NumpyIterator' object has no attribute 'as_numpy_iterator' Just remove the .as_numpy_iterator() It worked for me.. For instance...test.as_numpy_iterator() will just be test after removing .as_numpy_iterator Good luck

[@narsimhareddy4742]

It's not working can you help me out

[@Video13393]

on this line: frames, alignments = data.as_numpy_iterator().next() im getiing this error "FileNotFoundError: [Errno 2] No such file or directory: '/content/drive/MyDrive/data/alignments/s1/.align' HOW TO FIX THIS? PLS TELL

[@jeyasrisenthill]

This is not working.

[@mugunthking8735]

that's one of the awesome tutorials

[@NicholasRenotte]

Ayyy thanks so much!!!

[@christopherjr.maranan2146]

Good Day!! I am having error at frames, alignments = load_data(tf.convert_to_tensor(test_path)) it says here ValueError: TypeError: object of type 'RaggedTensor' has no len()

[@NicholasRenotte]

Double check test_path is a string with \\ for windows or / for Linux

[@savibatti1370]

@@NicholasRenotte it's still not working tried windows path as well

[@ghostplays5291]

Hey Nick I am running it on my gpu but I got unknown error in loading frames and alignment but I run it on CPU there is no error but I want to run on GPU what to do

[@feridazakov8887]

Hi, that is a great work, I am trying to implement it on Colab. However, in Data Pipeline section, in frames, alignments line I get an error indicating no such file or directory 'data/alignments/s1/.align'. I could not solve it, I see the files when I check data directory. I do not know what is the problem. Can you give any idea? what is the problem?

[@jannatulfirdous1459]

I'm stuck there too

[@jannatulfirdous1459]

Please let me know if you found any solution

[@feridazakov8887]

@@jannatulfirdous1459 not yet, I asked my friend to check it , as soon as I solve I will tell you.

[@jannatulfirdous1459]

Hey, I got the solution. So in load_data function change '\\' to '/' in file_name line In test_path line below this function, do the same. In tf.convert_to_tensor line do the same woth the slashes. Try it , hope it helps you, it solved my error.

[@feridazakov8887]

@@jannatulfirdous1459 thanks, I will try

[@FireFly969]

I learn pytorch, and iam planning into making this same project but in pytorch, and i dont know why i feel that tensorflow is co much easier compared to pytorch

[@sadiaafreen1519]

Hey! Superb Work. I have been working on this model following your tutorial but the epochs are taking forever to load. I just completed my first epoch after 9-10 hours. Moreover the colab gets timed out and it starts all over again🙁 is there anything i can do?

[@gsettu7255]

Did it complete I'm currently doing it's still running from past 13hours

[@OfficialGamer-kc4dz]

@@gsettu7255 Hey did you solved that epoch issue

[@OfficialGamer-kc4dz]

@sadiaafreen1519 Hey did you solved that epoch issue

[@wasgeht2409]

THANKS !!!

[@NicholasRenotte]

Anytime!!!

[@solomon_leo_27]

Nic! Can I use this model to predict a video without alignments?

[@NicholasRenotte]

Yep! That's the beauty of using the CTC loss function!

[@dclxviclan]

Real cool calculating

[@adithya7064]

Your work on this is awesome..and the explanation is also great🙌..I have found on one explaining every step in this way..But, one thing In extracting the Lip region using static values..it doesn't work for other input videos right?? Also the total number of frames from the input video will be getting error which is not matching with the convolutional layer's input_shape

[@adithya7064]

What to do for the error which is getting because of convolutional layer's input shape is not matching with the given input video's shape..can you provide the solution for that??

[@CreatingUtopia]

@@adithya7064 did you find any solution bro, I think I am facing same issue

[@adyantpandey7099]

INVALID_ARGUMENT: Only one input size may be -1, not both 0 and 1 i m getting this error to train model {model.fit(train, validation_data=test, epochs=100, callbacks=[checkpoint_callback, schedule_callback, example_callback])} in this line of code anyone pls resolve this ??

[@pranav-patil]

which exact python and tensorflow version version are you using ?

[@NicholasRenotte]

3.8 and 2.9.1

[@parthrangarajan7474]

Hi. This is great. How do i load a custom video and try to perform lipreading on that?

[@Video13393]

on this line in data pipeline: frames, alignments = data.as_numpy_iterator().next() im getiing this error "FileNotFoundError: [Errno 2] No such file or directory: '/content/drive/MyDrive/data/alignments/s1/.align' HOW TO FIX THIS? PLS TELL

[@pavanp1950]

did u fix this problem??