Free Motion Tracking plugin for TouchDesigner: Face, Hand, Pose Tracking with MediaPipe Plugin

[@arthoefmann]

Can I just say this is brilliant work, thank you so much for making it accessible, as an artist and designer working and exploring touchgdesigner tools like these make it so much easier and more fun to learn and experiment!!

[@feelzlab]

wow! I just get to know about MediaPipe - thank you so much for creating this and making this tools more accessible for individuals like me who have visual ideas but are not as technical as one can be.

[@blankensmithing]

Thanks! that manes a lot. Excited to see what you create with it!

[@gregderivative2647]

o my, brilliant tracking without special hardware. great explanations.

[@devicenotfound]

Thanks for sharing such a helpful video! I never thought TouchDesigner could do this, and it's truly inspiring. Your video also has shown exactly what I've been looking for - how to superimpose TouchDesigner output to OBS.. Your effort is greatly appreciated!

[@AliTanUcer]

this is phenomenal!!... i was about the tackle the same thing and wishing to find time to port everything to GPU.. now we have it.. amazing.

[@nightshining5662]

Thanks for the plugin! For anyone having issues with the MediaPipe tox loading forever, the last version of 2022 (.35320) works for me on an M1 mac. I just avoided having to upgrade my commercial license for another $300 over this.

[@OzanTezvaran]

This looks amazing, cannot wait to check it!

[@pagarreton]

Wow this is incredible! I am really happy it also works for Mac computers!

[@saucemonster_the_movie]

he is using a mac

[@yoli5222]

i would be really grateful to understand the spoutcam for windows in depth for the mediapipeline plugin. Thank you

[@Treybradley]

Same. was having issues with spoutcam. it worked briefly at first but with a strange aspect ratio. then was showing noise. opened up the spoutpanel and spoutsettings and just randomly tinkered around until things got going

[@jimkuehn]

Sick sick sick. Am rolling into a project that might benefit from your great work. Perfect timing

[@christopheluez1860]

Incredible content ! Verry smooth, everything is working perfectly ! (Macbookpro 2021 M1pro, lastest TD version)

[@borjonx]

When looking for channel names(like around @20:37 & earlier) you can also just add a SELECT and click the black triangle next to "Channel Name" for a full list.

[@markus_knoedel]

WOW, I will be off grid now for some time to figure all of this out. Thank you. So helpful.

[@BossCODINGplease]

This is truely amazing, thanks for all the hardwork

[@eldelbaar]

Thanks a lot for making this project public! it helps a lot for people that is starting

[@gianlucacolia3431]

This plugin is amazing, I hope you keep updating it. thank you so much for sharing it!!!

[@knast7093]

Hey guys, first of all: huge thank you for sharing and working on this - This is soo sick! Everything is working as intended for me (2023 version of TD) But I want to overlay the image segmentation with my original webcam-in and I'm having issues with it just showing a 'black screen'. Even if I try with the SpoutCam I get a black screen or saying 'No Active Sender Found'. I hear that you have to use TD 32bit version for SpoutCam to work, but is that true? Maybe there's a way to get the camera and the image segmentation out on the same time to composit or overlay the two? All the best.

[@RusticRaver]

350£ kinect 2 is good for the bin!

[@sadiaa.3227]

still useful for pointcloud projects

[@jagvalverde]

According to the workflow logic of my project, I would need the image input to be a pre-recorded video, but without using OBS, only using TouchDesigner. Would that be possible? I want to create different automations with different videos, and I don’t think OBS will allow me that configuration. I have no words to express my gratitude for your contribution. It’s incredible. Thank you so much.

[@blankensmithing]

@@jagvalverde OBS will allow you to get video input into the MediaPipe component. Connect your video from within TD to a Spout/Syphonout TOP. In OBS pick up that video feed and turn on the virtual camera. Your video will now show up as a webcam device for MediaPipe. (I wish we could just connect a TOP to MediaPipe)

[@cele_stin]

This is unbelievable, thank you guys!

[@denizz_kurt]

Awesome tutorial Torin!

[@j.adambrinson8415]

Fantastic plugin-I think I can pack up my old Leap Motion Controllers now. Question: Is there a way to permanently have h1 be the left hand/h2 as the right? My experience now means that whichever hand is detected first becomes h1, regardless of whether it's my left or right hand. Specifically referring to the normalized data from Output 1 in the hand tracking component. Appreciate any advice on this, as well as the awesome tool.

[@EMERGENCIASTATE]

you are the real master bro ! keep doing it !

[@StrawberryBattlefield]

thank you so so so much for the amazing works!!!

[@carlosgaston3985]

you two are so friking awsome! thanks a lot !

[@bradcatlett5215]

Excellent tutorial and plugin. Very well made. I'm trying to apply the face feature points with a 3d model to animate the face according to the user. Any tips on how best to accomplish this? I keep running into blockers or dead ends.

[@saralopez7873]

This is incredible! Thank you so much

[@fuegofinder2015]

Awesome. How is the performance of this vs the kinect?

[@TrevorNew]

This is amazing, thank you both for making this!

[@sebastianosabo]

Hi guys, first of all I wanted to thank you for this great tool! I am working on a project where, using my hands, I reconstruct images as if I were painting them and I wanted to ask you a question. Is it possible to remove the neck part and keep only the hand part when working using output 1 of the image segmentation 21:33 (the body skin colour one)? In the preview both hands, and part of the arms, are coloured pink/purple but as all the skin is detected the neck, which always remains uncovered, is also detected, ruining the effect. Thanks again and congratulations on everything!

[@roberty9057]

More than amazing ! Thank you so much

[@MikeKluge-o3u]

Amazing, thanks for your hard work on this!

[@pixelasm]

Oh wow, just stumbled upon this. As I just recently used a #LeapMotion in #TouchDesigner it seems like #MediaPipe provides a great replacement as no dedicated hardware is needed anymore.

[@diyacorrea1023]

you guys are amazing thank you so much!!!

[@spindlewhorls9764]

thank you for your work it's real fun to use this and a lot of possibilities. I have one question: would it be possible to use the mediapipe plugin in td with an Infrared webcam?

[@jjeon0923]

Thank you so much! Such a game changer! Is there a way to out put the wire frame in a top? Trying to just out put stick man figure without the source video underneath it. Again, thank you!

[@benjavides]

This is incredible! Thanks for publishing it ❤

[@alexfarac873]

I tried to do the obs set up but for some reason my virtual cam on the media pipe node gives me a black screen. I have tried various resolutions in obs preferences, the input video, my camera's resolution, and 1280x720 but it isn't working. Any recommendations?

[@oligest4lt]

This has been an amazing plugin to work with! I was wondering if there's a simple way to get the finger point tracking coordinates to reset to the centre of the screen when they are not visible? Currently they go to 0,0 in the bottom left so it constantly pulls anything I have mapped to it to this point if it can't see the intended finger. Thanks!

[@光亮张]

Thanks for publishing it

[@borjonx]

how are you getting the z position of your fingertip from a non-3D camera? ty for sharing, this looks incredible - I've never had such a powerful tox up & running to quickly. Only issue was I had to download the latest TD.

[@junweichan-n9i]

Why does mediapipe report an error when I use it?

[@mustaTraceur]

Hi, I hope you can help me. I just download the mediapipe plugin, so I open the main project MediaPipe TouchDesigner.toe then I Select my main camera and it doesn't show any image, but when i use a videodevice_in it automatically use my main camera. Therefore almost every subsequent node show me some kind of error Im using a macbook air M1, so, the main camera is a facetime camera. Also, when I drag and drop some of the .tox at an empty Touchdesigner project it just doesnt drop at the workspace. Any idea if Im the one thats doing something wrong or someting else happening?

[@adpokem]

How is it that you are able to have both the mediapipe node and video device in run at the same time? I can only ever have one working at a time because my webcam doesn't like it when multiple things try to use it at once. Is there any workaround?

[@neokortexproductions3311]

We appreciate you!

[@notweirder]

Love the plugin! For some reason the latest version (0.32) shows the facePoints chop as an x,y,z chop (with green, yellow and red) as opposed to the time-slice style chop in the video. Any idea how to convert to the latter? Thanks!

[@kuanfuweimu]

u can try to use analyze CHOP i thought

[@notweirder]

@@kuanfuweimu Perfect, thank you so much!

[@lee_sung_studio]

Thank you. 감사합니다.

[@NoahEckert-u5s]

Hey Torin! Thanks for the great video and plugin! I was wondering if there is a way to track multiple people at once. especially with the pose tracking? or is this not possible with MediaPipe?

[@blankensmithing]

Hey at the moment Media Pipe only has pose tracking for 1 person. It does look like multi-person pose is on the way soon though since their web based demo has multiple person detection(those examples are usually head of what they release to the public).

[@NoahEckert-u5s]

@@blankensmithing ok great thanks for your reply! Diffrent question: How can i use the Plugin with a higher resolution? I want to use the video output in 1080p but i can not use my Camera Input in a Video Device In TOP when i allready selected it in the MediaPipe Container.

[@blankensmithing]

@@NoahEckert-u5s The ML model scales down the webcam size, so using a higher resolution won't give better results. If you need to use a higher res cam you can use Spout Cam, or Syphon w/ OBS to change the resolution of your device and pipe it back into TD as a virtual webcam

[@moonth]

It's amazing! Thank you

[@shiangchia4337]

looks cool! thanks for sharing!

[@Ardsadprm]

Hi! I got a problem, only loading and media pipe logo, can't detect the face in the camera, what is the solution?

[@HarryStemmer]

Is it possible to use ndi directly as input

[@QISUN-ms5rn]

太谢谢惹！！！！！好棒好棒的组件！ 😁

[@AtelieristaMHP]

I am trying to make particlesGPU react to several hands in the same time, but it doesn't seem to work, only with one hand at a time. is there a setting I can change that?

[@Rodrigo-h7q]

cool Torin, thanks a lot 4 da work work work. , do you know if i connect 3 webcam to TD, can i use mediaPipe, using each one to track their onw user's hand?

[@blankensmithing]

Hey, thanks! Yes you can just duplicate the MediaPipe component, make sure to change the Websocket Port and the MediaPipe Port to something different than the original component and you can select a different web cam. You can do this with however many webcams you have. I'd recommend turning off all the models you're not using for speed

[@doridori8916]

Thanks! i have a question! can i add my hand gestures? (not something that already exists like Closed Fist, Open palm, Point Up ... )

[@jaspersegers]

Hey, this is a super great plugin. I have a question though, There's no way to keep a consistent left hand and right hand. Everytime I remove one hand from the frame, the remaining hand becomes Hand_1. This makes it impossible for me to create a controller using 2 hands :(. Does this have an easy fix or is this functionality coming in the future? :)

[@moonmountain]

Thanks so much for this amazing upgrade to working with mediapipe in TD, I was wondering if there is any benefit to using smaller resolutions than 1280 and if the spout method is the only way to achieve that downsizing? With the previous workflow of mediapipe I was able to get good hand coordinates at resolutions as low as 240x135 with an increase of roughly 8-10 fps. Although I am totally unfamiliar with the webbrowserCOMP and how/where it gets the webcam feed. Thanks

[@blankensmithing]

Hey! yeah the models run off a very small resolution feed, so there's no need to use a large video format. The only way to do it at the moment through spout or obs, which would add an additional delay

[@lilyhsiao9315]

Thank you so much for your plugin!!! I have a question about the bound box data, there is an "angle" from the mediapipe face bounding box value list, but in my project, it always shows "0".

[@blankensmithing]

Hey! which model are you looking at? Is it the face_detector, object_tracking, face_tracking component?

[@lilyhsiao9315]

​@@blankensmithing thank you for your immediate response! I'm looking for a face detector. I wanted to get the face/head rotation value, and then I saw an "angle" property, but it always shows"0".

[@lilyhsiao9315]

Hi, I just would like to follow up on this if I did some wrong setting. Thank you so much🙏

[@LealRaulAndres]

Looks amazing, but i could not get it to work, i get an error that resolutions is up to 720p. Tried putting the resolution down but still no working. Working on a MAc pro with sonoma TD 2022.33910

[@mattphillips3024]

WOW! incredible

[@vxkhwz]

Hey, thanks! Question: so there is no way to keep up tp date with git pull atm? So I will have to manually download updates?

[@xswl5164]

COOOOOOOOOL!!

[@daveydonaldson2750]

what is the best way to use the face mask and replace it with a custom face model, IE make the face track onto a scary ghost or something fun instead of the base White mask. Trying to figure out the best way

[@7cd664]

Did you find a solution?

[@chrismills8404]

I feel like your intro gets louder every vid and its a slow prank :D

[@blankensmithing]

haha It's always the same volume, but may have to do this now :)

[@C0llinsW0rth.]

hmmmm the mediapipe component just keeps saying "loading" i don't think i did anything different than the tutorial, any thoughts on troubleshooting?

[@zablok69]

Hey guys! Anyone knows if there is a workaround to use Mediapipe TD for vertical video / stream?

[@federicaintelisano5639]

Can you connect cameras like arri?

[@alchemist.D]

nice job

[@BACCAJIMIN]

hello thank you so much for the plugin!!! I just have a problem of saving the file. Everytime I save my file and exit, when I open the file again, all the work I've put disappears.. if the file is not saved in the same folder of the toxes folder, the work is still there but it says the file is having a trouble loading components(probably looking for the toxes files) How can I solve this? :(

[@blankensmithing]

Hey if you've lost any progress just put the tox files in a folder called toxes that's in the same directory as your .toe file and re-open your .toe file. In the future there's a toggle you can turn off on all the mediapip tox files in the parameter page under Common called "Reload .tox on Start"

[@BACCAJIMIN]

@@blankensmithing omg thank you so much!!! I tried turning the toggle off from mediapipe.tox file and still had the same problem, I had to actually turn off all the toggles from rest of the tox files in the tox folder and convert them to tox files instead of toe files to make it work!! Thank you so much💗💗💗

[@AliTanUcer]

Another option is NDI in NDI out..

[@mlnima]

I am trying to use it but it gets extremely laggy and fps drops on camera even I have 3090 gpu and I9 cpu, can anyone help?

[@blankensmithing]

Try turning off the models you're not using. It's not intended to have all of them running.

[@eshu1023]

Hello.. does this work on TD non-commercial as well? I subscribed to your patreon and downloaded some TD sketches.. none of my webcams seem to work (just blank white page)and I am wondering whether it is the TD version! mediapipe downloaded from your Github has the same issue. My MacOS is 14.0 Sonoma. Thanks in advance.

[@blankensmithing]

Hey what version of TD are you using. You don't need a commercial license. It should run on the latest version of TD, but loading into much older version won't work

[@eshu1023]

@@blankensmithing it's the most recent version. I downloaded it this week.

[@yosukeDVJ]

Amazing! This is huge. How can I get Syphon to show up in the cameras dropdown? I'm using someone's proprietary OpenFrameworks Syphon App to get a KinectV2 working on my M1 Mac. I only get the video feeds tho so no CHOP data. Incorporating this mediapipe plugin into my workflow would nearly complete finally having a fully functioning kinectv2 in touchdesigner on apple silicon!

[@yosukeDVJ]

never mind! Using ndi.

[@kamiusu2360]

Hi! I have a problem, when i open a patch all connection from MediaPipe base in disconnected. And i have red circles with cross. How i can connect them again?

[@blankensmithing]

Hey please follow the steps here: kzbin.info/www/bejne/eamXdp-ils9poc0si=jzo-JCOVxF89jOpS&t=139 so I can help debug. Make sure to attach screen shots. If you see a red x on a component, hover over it and include a screenshot of the text that's displayed in the error message

[@kamiusu2360]

@@blankensmithingThank you for you help! Now it works good on my Apple laptop, but on PC it stucks in loading Media Mapper when i connect usb webcamera.

[@kamiusu2360]

I resized usb camera resolution in OBS to 720p and it works now! Thank you very much!✨💜

[@amanpathak2338]

This is my first time working with a plugin on TD, I followed the steps like you said, and referred to the github page but it doesnt load up media pipe for some reason, I selected my Webcam, checked it as well but MediaPipe doesnt finish loading, what should I do?

[@jhahnaaa]

I was facing a similar issue. On downloading the current version of TD (2022.33910), it worked well!

[@blankensmithing]

Hey please follow the steps here: kzbin.info/www/bejne/eamXdp-ils9poc0si=jzo-JCOVxF89jOpS&t=139 so I can help debug. Make sure to attach screen shots. If you see a red x on a component, hover over it and include a screenshot of the text that's displayed in the error message. Make sure to list what version of TouchDesigner you're running as well

[@amanpathak2338]

@@blankensmithing Hi, so I was able to figure out the problem and now it works just fine. Excited to create with it. For anyone facing a similar issue, I used OBS studio instead of spoutcam for the webcam stream [PC] . In the video input of Mediapipe I used OBS Camera (which was scaled to 720p) and thats how it started to work. I created a new scene in OBS and simply added my webcam to the scene. Keeping OBS open I restarted TD and I could see "OBS Camera" as an input in the drop down. Hope this helps anyone facing this issue. Cant wait to get started with this plugin finally:) Thanks to the creators again!!!

[@blankensmithing]

@amanpathak2338 Hey! I'm glad you got it working :) Thanks for the tips so we can pass it along to other people

[@NA-cw4pj]

Hi, thank you for this beautiful project, but I am having an issue, why the MediaPipe box is always on Loading mode, and it print an image of Loading. I tried reseting the process but still doesn't work for me. Can you help me? Thank you!