Real Time AI HAND POSE Estimation with Javascript, Tensorflow.JS and React.JS

[@akshitdayal2689]

I'm so glad I found your channel out here! Everything you post is extremely useful and beginner friendly!!

[@woutvandesompele8310]

THIS is what you call a flawless tutorial!! Thank you so much!

[@NicholasRenotte]

Flawless?!? What no way 😅, I'm so used to there being bugs now.

[@ifvictorthen10]

Great video!! For those who will run this code: the load model shouldn't be done every frame and the webcam properties can be saved in some local state (hook) for performance.

[@NicholasRenotte]

Yup this ^. Code was updated in the GitHub repo for optimised performance: github.com/nicknochnack/GestureRecognition

[@diegocaumont5677]

Love to follow along with ur tutorials man, clear, concise and informative. Great stuff.

[@NicholasRenotte]

Thanks so much @Diego Caumont! Anything else you'd like to see or learn?

[@diegocaumont5677]

@@NicholasRenotte Well i'm still kinda new to the Tensorflow domain. However text generation using Keras and Tensorflow would be quite cool. (just an idea)

[@NicholasRenotte]

@@diegocaumont5677 done, I've got it on the roadmap now!

[@diegocaumont5677]

Nicholas Renotte Dope ! 👍❤️

[@nobodyeverybody8437]

Thank you for your tutorial, I could go along with it, however, it is just a prototype and not an actual app. So, you may talk more about how to make it responsive to the size of the browser and device and how to make it run faster without any performance problems. tanx!

[@igor_cojocaru]

Thank you! Two days trying to set up this in react and finally found your video! Cool! Subscribed

[@NicholasRenotte]

Yes @Igor Cojocaru! Awesome, glad to hear you got it up and running. Real time body segmentation is coming out tomorrow!

[@igor_cojocaru]

@@NicholasRenotte Cool stuff I found this finger pose detector which ads new feature of fingerpose detection openbase.io/js/fingerpose/documentation Take a look at it, can be the second part of this video

[@NicholasRenotte]

Woah, thanks @Igor Cojocaru that's pretty cool! Let me dig into it, might be a Pt2 coming soon!!

[@hiteshsharma3425]

Hi Nicholas, the drawn hand and actual hand in video were not coming aligned for me. I figured that's because teh width and height of the canvas and webcam element were hardcoded to 640x480. Changing them to videoWidth / videoHeight did the trick. Works great !

[@NicholasRenotte]

YESS! Awesome work @Hitesh, that's the right approach, would need to scale it to your webcam. Awesome work my guy!

[@hiteshsharma3425]

Hey @@NicholasRenotte , really appreciate you sharing these hands-on tutorials.

[@ashishhameed2647]

hey nicholas whan I cloned the repo and ran it in my browser it was very slow. Do you have any idea on how to optimize it?

[@sandysathya4996]

The handpose is drawing perfectly but i have a problem init like its drawing below hands not on my hands its tracking all movements but the mesh is position below my hands posjtion how to fix it? Any idea

[@NicholasRenotte]

Heya @Sandy, might be a typo somewhere in the styling. Can you compare to the baseline code: github.com/nicknochnack/HandPoseDetection

[@sandysathya4996]

@@NicholasRenotte thnx for the reply went throught the file a yes there was few typos on app.js my bad😅

[@NicholasRenotte]

@@sandysathya4996 😂 it happens! Glad you got it working!

[@Dara-gq6jl]

Very nice video learned alot but I had one question how do i make ctx draw both hands? thanks

[@NicholasRenotte]

I believe this is a single hand model. Check out mediapipe handpose for dual hand models!

[@petitblou]

At the stage to run to train the model, it shows me: MacBook-Pro RealTimeObjectDetection % python Tensorflow/models/research/object_detection/model_main_tf2.py --model_dir=Tensorflow/workspace/models/my_ssd_mobnet --pipeline_config_path=Tensorflow/workspace/models/my_ssd_mobnet/pipeline.config --num_train_steps=10000 Traceback (most recent call last): File "Tensorflow/models/research/object_detection/model_main_tf2.py", line 30, in from absl import flags ImportError: No module named absl While I'm stilll having the absl package installed Could you please help

[@NicholasRenotte]

Double check you're running these commands in the same kernel that you've installed in. If you run !pip list from your notebook absl needs to be shown there!

[@elineclaeys8586]

First I want to say that it really was a super interesting tuterial. It is working! But I still don't really understand how all the files are linked. How the html is linked with the javascript. Maybe you can explain it more or make a video about it?

[@NicholasRenotte]

Awesome! I'm stoked you enjoyed it ANNNND got it working! Definitely, so the large majority of the code is done using React.JS, this sort of blends HTML and Javascript by allowing us to build apps using a slightly modified version of Javascript called JSX. When we're using the element for example we're still setting up a HTML element that eventually gets rendered to the page. The React.JS hello world example gives a great explainer on the basics: reactjs.org/docs/hello-world.html I'll add a crash course explainer to the video schedule as well 😊!

[@kalyanchakrig]

Thank you so much . The way you explain is simply super. Can we do the same for wrist to do try on for watches.... Thanks again.

[@NicholasRenotte]

Oh YESSS, Virtual Try On, been researching this lately.

[@hligfl]

Oh man you did it ! What is the fps ? That was what I needed to combine it with a VR web app 😈

[@AMRUTHAK-qu6sl]

Thank you for the wonderful explanation. Could you please give a tutorial on hand gesture recognition using LSTM model

[@NicholasRenotte]

Sure, let me take a look!

[@mycode0]

How would you do this without hardcoding the position of the webcam and the handpose

[@riturajsingh2234]

Hi Nicholas. I understood all of the video and wrote the code following you but i didnt understand only one thing. i would be really nice if you explain why you did this // Set video height and width // but why is this important webcamRef.current.video.width=videoWidth; webcamRef.current.video.height=videoHeight;

[@NicholasRenotte]

Great question @Rituraj Singh, I haven't had a bunch of time to dig into why this is needed yet. I think tfjs may depend on the video.width and video.height property. We're setting width (videoWidth) and height (videoHeight) as part of styling, so (afaik) we're extracting the styling properties and using it to set the ones tfjs depends on.

[@saeidrezatalaeikhozani9466]

Hello, Thanks for your great video. I have a question, when I try to use the app on the cell phone browser, it couldn't detect the the points on the hand correctly. The points are not attached to the hand and totally not working well on the browsers of mobile phones. Can you let me know how I can fix it please. Thanks again :)

[@mutaherkhan2161]

what is the difference between fingerpose and hand pose estimation? Is both are work for solving same problem i.e sign language prediction

[@NicholasRenotte]

fingerpose is the library that allows you to detect gestures, hand pose estimation is the deep learning model that allows you to detect where your hand and joints are in the frame!

[@영상일기-h6h]

Thank you for the tutorial! this is really cool. I just have one question. When I do it until 25:00 where I can see the landmark points appear on top of my hand, the landmark somehow appears bigger than my hand(whereas your landmark is the same size as your hands). Do you know how I could fix this problem? Thanks in advance!

[@NicholasRenotte]

Heya, you can try changing the size values being used in the arc method inside of utilities.js e.g. // Before ctx.arc(x, y, style[i]["size"], 0, 3 * Math.PI); // After ctx.arc(x, y, 2, 0, 3 * Math.PI);

[@midor_bare6513]

could I use real-time hand, body, and face recognition all at the same time for creating a real-time moving 3d produced avatar?

[@NicholasRenotte]

I guess you could some what @Nanii Nanii! the pose detection model is just detecting a few keypoints so it wouldn't necessarily be full 3D though.

[@wgalloPT]

QUESTION: I know its been a while youve done this, but my question is what if I want to add the forearm segment to the hand for the wrist joint? any tutorial that youve done that could help me with that? Thank you!!

[@cristeenajoseph7145]

Hi, Can we do multi-hand detection with this tensorflow model

[@NicholasRenotte]

I think this particular one is single hand but I've done some Python stuff on the channel which is dual!

[@きさま-x3v]

was a greatly nice stuff you saved my time

[@jaydave791]

can this be used to create virtual try on augmented reality applications(like trying on rings on different fingers)

[@NicholasRenotte]

With a bit of additional work, I believe so @Jay!

[@kenmiles5027]

Thanks for these tutorials! I have just one doubt, suppose I wanted to do these predictions only when we press a button, i.e. the webcam and predictions process starts and stop when we press a button. Can you suggest how to implement this?

[@NicholasRenotte]

Heya @Shakib, you could use the getScreenshot method rather than running the pipeline live. This is an example of how to use React-Webcam to get a single frame: codepen.io/mozmorris/pen/gOOoqpw

[@kenmiles5027]

@@NicholasRenotte Thanks for the suggestion 😇

[@NicholasRenotte]

@@kenmiles5027 anytime!! Good luck, let me know how you go!

[@innocentntuli6995]

is it possible to senseboth of the hands simultaneously

[@NicholasRenotte]

Not in this particular model but with other hand models there is that capability. Check this out: kzbin.info/www/bejne/rIK9ZXysd5Vqm8U

[@amirhosseinebneroomi7280]

really liked it!!! Can we do the same thing with React Native?

[@NicholasRenotte]

Sure can, haven't done a vid on it but defs possible @Amir!

[@aadityaknv3990]

Hi Nicholas, Thank you. This was very helpful. Ive been stuck at wanting to get the distance from the hand to the camera? Is there a way to do that?

[@aadityaknv3990]

how do we retrieve the x and y coordinates of a keypoint lets say keypoint 12 (end keypoint of middlefinger).

[@NicholasRenotte]

Thanks @Aaditya Knv, you can grab it from the hand object, for example: // Get x and y coords const x = hand[0].landmarks[12][0] const y = hand[0].landmarks[12][1]

[@NicholasRenotte]

Definitely, I don't cover it in this video but the technique you're looking for is called Depth Estimation. I can grab some papers on it if you'd like? I've added it to the videos I'm planning as well!

[@aadityaknv3990]

@@NicholasRenotte that would be great! :D

[@rjcrabbe1210]

Amazing tutorial. Very helpful for novices like myself. However I am having a problem with the visualization around minute 19 I get this error. Unhandled Rejection (Error): The implicit shape can't be a fractional number. Got 1 / 3 Please help :_(

[@NicholasRenotte]

Heya @RJ, no stress, can you share your code?

[@rjcrabbe1210]

@@NicholasRenotte yes for sure

[@rjcrabbe1210]

thanks soooo much Nicholas !!!! Here is the code that I added btw import React, {useRef} from 'react'; //import logo from './logo.svg'; import * as tf from "@tensorflow/tfjs"; import * as handpose from "@tensorflow-models/handpose"; import Webcam from "react-webcam"; import './App.css'; function App() { const webcamRef=useRef(null); const canvasRef=useRef(null); const runHandpose = async () =>{ const net = await handpose.load() console.log('Handpose model loaded.'); // Loop and detect hands setInterval(()=>{ detect(net) }, 100) }; const detect = async (net) =>{ // check data is avaliable if ( typeof webcamRef.current !== "undefined" && webcamRef.current !== null && webcamRef.current.video.readyState === 4 ){ //Get video properties const video = webcamRef.current.video; const videoWidth = webcamRef.current.video.videoWidth; const videoHeight = webcamRef.current.video.videoHeight; //Set video height and width webcamRef.current.video.width = videoWidth; webcamRef.current.video.height = videoHeight; //Set canvas height and width canvasRef.current.width = videoWidth; canvasRef.current.height = videoHeight; // Make detections const hand = await net.estimateHands(video); console.log(hand); // draw mesh } }; runHandpose();

[@rjcrabbe1210]

Alos, I think my problem happened when I ran tensorflow/tfjs and tensor flow handpose in terminal. I realized that I had newer versions of the programs, and I think that was the biggest problem. I really appreciate the help and your videos. You are the best!!

[@NicholasRenotte]

@@rjcrabbe1210 anytime! Started taking a look: 1. Just did a comparison of your code to mine and it all looks okay, in the error trace, where does it say the error is occurring? 2. Can you share the full error that you're seeing? 3. Are there any differences between your utilities code and the sample code (github.com/nicknochnack/HandPoseDetection/blob/master/src/utilities.js)?

[@atifasadkhan]

In windows I used ubuntu downloaded from windows store and then performed all of these, but it take 10 mins to load shouls I do these in linux ?

[@NicholasRenotte]

Heya @Atif khan, you can do it directly in windows 😀 , no need to spin up containers or vm's as long as you've got Node.JS nodejs.org/en/download/ you should be good to go!

[@atifasadkhan]

@@NicholasRenotte ohh oki doki

[@atifasadkhan]

Can we build a test web app where gestures are the inputs, like in MCQs test app where we give inputs using the hand gestures.

[@NicholasRenotte]

Definitely @Atif khan, stay tuned, this weekend I'm releasing how to extract gestures from the hand poses!

[@atifasadkhan]

@@NicholasRenotte thanks 😊 yes subscribed and liked and the bell icon is pressed eagerly waiting for the next video 😁😁

[@venkatsangeeth6984]

Hey Nicholas Cool Video! Can u make a video on feet detection if possible ? i'm really interested in it ! and also how to render a 3d object/model in the detection ?

[@NicholasRenotte]

Heya @Venkat, feet detection can be done with the Object Detection series (studio.kzbin.info/door/HXa4OpASJEwrHrLeIzw7Ygplaylists), I've definitely got come 3d modelling stuff planned!

[@venkatsangeeth6984]

@@NicholasRenotte Thank you so much it was really helpful ! i'm waiting for the 3d modelling .

[@NicholasRenotte]

@@venkatsangeeth6984 awesome! Glad it was useful!!

[@codinginminute]

hi can u show how can use mediapipe in Javascript

[@MejiMaru]

Wow awesome tutorial bro... how to make this detection as mouse control on web apps? thank you.

[@NicholasRenotte]

Oooh, let me think about that @Meji Maru! Will get back to you or make a video on it!

[@MejiMaru]

@@NicholasRenotte Great... thank you so much.

[@muhammadnawaz496]

Hey, thanks for amazing video! But i want to detect specific objects in an image, how i can find it?

[@NicholasRenotte]

Check out the object detection tutorial on the channel :)

[@ozyozk9466]

Does it tell you the angle between each joint?

[@NicholasRenotte]

Not by default nope, @Ozy, You could calculate it with a little trigonometry through!

[@LightningXBolttt]

Hey can you create a video for sign-language recogntion. I am really interested to know how to do it?

[@NicholasRenotte]

Heya @Swarnav, hell yeah, buckle in it's coming tomorrow!

[@LightningXBolttt]

@@NicholasRenotte Yuhuuu..Thank You 😁

[@LightningXBolttt]

@@NicholasRenotte Hey...Waiting for your video !! 😄😄

[@NicholasRenotte]

@@LightningXBolttt it's rendering right now!! Should be up in about an hour!!!

[@NicholasRenotte]

It's out! kzbin.info/www/bejne/pnW7lZ-OocZ2mdE

[@rjcrabbe1210]

thanks for uploading such an awesome video... I followed it step by step and even copied your repo, but React was unable to draw the mesh

[@NicholasRenotte]

Heya @RJ, any errors displaying?

[@midnightblue914]

@@NicholasRenotte I have the same issue, no issues when I look at the console or the terminal.

[@midnightblue914]

Btw really appreciate your videos!!

[@rjcrabbe1210]

@@midnightblue914 try comparing code on github to yours line by line. If that doesn’t work just copy and paste segments of his code to yours. Also try saving your code program and refreshing ur page :)

[@NicholasRenotte]

Great suggestion @RJ and thank you so much @Navya! Did you get it up and running?

[@jayanthsaikiran6624]

How do i create same in react native ?

[@zaidiqbal9671]

can we do this in react Native

[@NicholasRenotte]

Sure can, TFJS should work there as well. Haven't had a chance to play with it yet though.

[@marco.scaligeri]

thank you sir! this is so wonderful!! Is there any way to record in a bvh or fbx format for animation purposes?

[@NicholasRenotte]

This is super interesting, first I've heard of bvh and fbx. bvh looks super useful, like it could be used for rigging?

[@marco.scaligeri]

@@NicholasRenotte bvh is a file format to record just motion capture and it can be applied to any previously rigged character or in this case hands. I'm a pianist animator and I own a leap motion but it's quite unreliable in terms of performance capture. I'm looking for alternatives and the Ai seems a great option. what do you suggest?

[@NicholasRenotte]

@@marco.scaligeri definitely, check out RigNet as well: zhan-xu.github.io/rig-net/ I've got a vid planned on this sometime in the future.

[@marco.scaligeri]

@@NicholasRenotte awesome, looking forward to it! Thanks for your reply! Cheers

[@marco.scaligeri]

​@@NicholasRenotte I just checked, Rignet is only an automatic body rig generator from a 3d character, adobe mixamo does it already efficiently. But i'm only interested into hand/fingers motion capture with a phone maybe or camera while i'm playing the piano and be able to record that as a fbx file so I can apply it to any 3d model. do you have any suggestion for that? thanks!

[@tylerbradbury8444]

Why does it only capture one hand?

[@NicholasRenotte]

Heya @Tyler, it's a single hand model. There are dual hand models out there, let me know if you'd like to see it in action!

[@tylerbradbury8444]

@@NicholasRenotte I would love to see it in action!

[@NicholasRenotte]

@@tylerbradbury8444 perf! Added to the list!

[@rolandfilho]

that's a very good material.

[@NicholasRenotte]

Thanks so much @Roland!

[@hrushikeshpardeshi4184]

Great Video Sir 🖤 ! Can you also make video of drawing by handpose on canvas 🥺??

[@NicholasRenotte]

Heya, this is drawn on the canvas :)!

[@gustavojuantorena]

Great content!!

[@NicholasRenotte]

Cheers @Gustavo J!

[@JustKiwoong]

thank you teacher, i love you

[@NicholasRenotte]

Anytime, right back at you!

[@aashishkalra9997]

Can you make a video over Freihand dataset ?

[@NicholasRenotte]

Oh didn't even know this dataset existed, yep, added to the list @Aashish!

[@aashishkalra9997]

@@NicholasRenotte please do. Facing a lot of trouble installing those packages.. specially 'opendr'.. Thanks in advance

[@NicholasRenotte]

@@aashishkalra9997 opendr? Where about's are you getting the error?

[@aashishkalra9997]

@Nicholas Renotte I get an error while installing opendr package.. I have tried 'pip install opendr' but it gives errors which tell the url doesn't exist anymore.. As per the git page, I've tried to run the sample file nd have got hand keypoints. But i also need to get 3d hand pose, which is only possible by installing opendr.. so please help.. This is the git page link: github.com/lmb-freiburg/freihand Thanks for responding back :)

[@NicholasRenotte]

Just checking, this isn't from this tutorial right? I don't think opendr is used as part of posenet?

[@shashankkumar851]

Bruh....make more content on tensorflow.js and it's application like controlling the mouse cursor on desktop screen using ✋ pose model

[@NicholasRenotte]

Already in the pipeline ✌

[@owenizedd2830]

For experienced developer, real things start at 5:50

[@NicholasRenotte]

Go get em!

[@vikashchand.]

Can you do a sign language recognition with tensorflow.js tutorial? 👌

[@NicholasRenotte]

Oh this is a brilliant idea @Vikash Chand, I'm doing finger sign detection this weekend will add sign language estimation to the schedule!

[@vikashchand.]

@@NicholasRenotte nice 🤘🤘😂

[@NicholasRenotte]

Heya @ Vikash, it's with Python but covers basic sign language detection: kzbin.info/www/bejne/pnW7lZ-OocZ2mdE

[@vikashchand.]

@@NicholasRenotte yoooo, it's dope! I just saw 🔥

[@NicholasRenotte]

@@vikashchand. yeahhya 🙏!

[@pauliuspuskunigis4721]

You're amazing!

[@NicholasRenotte]

🙏 ohhh thanks so @Paulius Puskunigis!

[@atifasadkhan]

do u have a whatsapp group where we can share our issue or fb group?

[@NicholasRenotte]

Not just yet @Atif khan, I'm working on it, will give you a heads up as soon as it's up! ✌

[@atifasadkhan]

@@NicholasRenotte got it !! waiting for it

[@Akosiyawin]

Sadly, I don't like react in my project.

[@fastmenu695]

A+ Tutorial

[@mummyskitchen5311]

Hahahahaha Hello Thanos 😜😜

[@NicholasRenotte]

🤣 @Aman Rathi, wanted to use the avengers theme so bad!!

[@THE-MNG]

why it's so smoth

[@NicholasRenotte]

That's just the way it renders, we're detecting with a reasonable FPS rate and able to draw it to the screen at level which appears smooth!

[@THE-MNG]

​ @Nicholas Renotte So, Can we customize with mobile

[@NicholasRenotte]

@@THE-MNG build it for a mobile app?

[@THE-MNG]

@@NicholasRenotte yes

[@NicholasRenotte]

@@THE-MNG yep, I'm working on a React Native example atm.

[@matheusserraouchoa8222]

Monster

[@NicholasRenotte]

Cheers!

[@jazodnem]

Hello Nicholas! Been following you for while quite recently, great vid! I'd like to ask if, for example, I want to create a bounding box instead of a hand skeleton that follows the hand on the screen? How should would you it? I had an idea to first take the x & y coordinates of the 21 keypoints and load them inside an array? Is there an easier method for that? ps: the bounding box I'm referring to is similar to the one found in this video of yours: kzbin.info/www/bejne/kIW2g42qZZl9obs Many thanks and looking forward to more of your videos!

[@Lucas-yn7ow]

bemsa :pray:

[@NicholasRenotte]

🙌

[@donche6]

cheater

[@NicholasRenotte]

Thanks? 🤣