Free Motion Tracking plugin for TouchDesigner: Face, Hand, Pose Tracking with MediaPipe Plugin

[@arthoefmann]

Can I just say this is brilliant work, thank you so much for making it accessible, as an artist and designer working and exploring touchgdesigner tools like these make it so much easier and more fun to learn and experiment!!

[@feelzlab]

wow! I just get to know about MediaPipe - thank you so much for creating this and making this tools more accessible for individuals like me who have visual ideas but are not as technical as one can be.

[@blankensmithing]

Thanks! that manes a lot. Excited to see what you create with it!

[@rudybutler9729]

Absolute Legends!!! As a Mac user, i've been tearing my hair out trying to get motion capture to work in TD beyond just a standard camera input.

[@gregderivative2647]

o my, brilliant tracking without special hardware. great explanations.

[@AliTanUcer]

this is phenomenal!!... i was about the tackle the same thing and wishing to find time to port everything to GPU.. now we have it.. amazing.

[@devicenotfound]

Thanks for sharing such a helpful video! I never thought TouchDesigner could do this, and it's truly inspiring. Your video also has shown exactly what I've been looking for - how to superimpose TouchDesigner output to OBS.. Your effort is greatly appreciated!

[@borjonx]

When looking for channel names(like around @20:37 & earlier) you can also just add a SELECT and click the black triangle next to "Channel Name" for a full list.

[@nightshining5662]

Thanks for the plugin! For anyone having issues with the MediaPipe tox loading forever, the last version of 2022 (.35320) works for me on an M1 mac. I just avoided having to upgrade my commercial license for another $300 over this.

[@pagarreton]

Wow this is incredible! I am really happy it also works for Mac computers!

[@saucemonster_the_movie]

he is using a mac

[@yoli5222]

i would be really grateful to understand the spoutcam for windows in depth for the mediapipeline plugin. Thank you

[@Treybradley]

Same. was having issues with spoutcam. it worked briefly at first but with a strange aspect ratio. then was showing noise. opened up the spoutpanel and spoutsettings and just randomly tinkered around until things got going

[@jimkuehn]

Sick sick sick. Am rolling into a project that might benefit from your great work. Perfect timing

[@KeyFur_NYC]

I know it is possible as anything with the correct knowledge and determination can be accomplished however, I am still pretty new to touch designer but I have ideas. Does someone happen to know how to utilize media pipe facial landmarks or any other touch designer tool to map a projected image to a persons face. I want to project fractal videos or even just a splash of color onto a mask I would be wearing while DJ'ing so I would be moving a little but not like running around a room. So i'm just trying to figure out how to map a video out source from a projector. Does anyone have any suggestions to accomplish this? Please and thank you.

[@christopheluez1860]

Incredible content ! Verry smooth, everything is working perfectly ! (Macbookpro 2021 M1pro, lastest TD version)

[@mustaTraceur]

Hi, I hope you can help me. I just download the mediapipe plugin, so I open the main project MediaPipe TouchDesigner.toe then I Select my main camera and it doesn't show any image, but when i use a videodevice_in it automatically use my main camera. Therefore almost every subsequent node show me some kind of error Im using a macbook air M1, so, the main camera is a facetime camera. Also, when I drag and drop some of the .tox at an empty Touchdesigner project it just doesnt drop at the workspace. Any idea if Im the one thats doing something wrong or someting else happening?

[@OzanTezvaran]

This looks amazing, cannot wait to check it!

[@eldelbaar]

Thanks a lot for making this project public! it helps a lot for people that is starting

[@markus_knoedel]

WOW, I will be off grid now for some time to figure all of this out. Thank you. So helpful.

[@gianlucacolia3431]

This plugin is amazing, I hope you keep updating it. thank you so much for sharing it!!!

[@adpokem]

How is it that you are able to have both the mediapipe node and video device in run at the same time? I can only ever have one working at a time because my webcam doesn't like it when multiple things try to use it at once. Is there any workaround?

[@BossCODINGplease]

This is truely amazing, thanks for all the hardwork

[@knast7093]

Hey guys, first of all: huge thank you for sharing and working on this - This is soo sick! Everything is working as intended for me (2023 version of TD) But I want to overlay the image segmentation with my original webcam-in and I'm having issues with it just showing a 'black screen'. Even if I try with the SpoutCam I get a black screen or saying 'No Active Sender Found'. I hear that you have to use TD 32bit version for SpoutCam to work, but is that true? Maybe there's a way to get the camera and the image segmentation out on the same time to composit or overlay the two? All the best.

[@cele_stin]

This is unbelievable, thank you guys!

[@alexfarac873]

I tried to do the obs set up but for some reason my virtual cam on the media pipe node gives me a black screen. I have tried various resolutions in obs preferences, the input video, my camera's resolution, and 1280x720 but it isn't working. Any recommendations?

[@RusticRaver]

350£ kinect 2 is good for the bin!

[@sadiaa.3227]

still useful for pointcloud projects

[@tor5515]

Question for 22:58. How do you have two video input devices using the same camera? I'm trying to use a video device in and the mediapipe plugin, but only one will work at a time

[@blankensmithing]

@@tor5515if you’re on windows you can only use 1 at a time. In that case just spout cam like how Dom shows in the video

[@EMERGENCIASTATE]

you are the real master bro ! keep doing it !

[@jagvalverde]

According to the workflow logic of my project, I would need the image input to be a pre-recorded video, but without using OBS, only using TouchDesigner. Would that be possible? I want to create different automations with different videos, and I don’t think OBS will allow me that configuration. I have no words to express my gratitude for your contribution. It’s incredible. Thank you so much.

[@blankensmithing]

@@jagvalverde OBS will allow you to get video input into the MediaPipe component. Connect your video from within TD to a Spout/Syphonout TOP. In OBS pick up that video feed and turn on the virtual camera. Your video will now show up as a webcam device for MediaPipe. (I wish we could just connect a TOP to MediaPipe)

[@sebastianosabo]

Hi guys, first of all I wanted to thank you for this great tool! I am working on a project where, using my hands, I reconstruct images as if I were painting them and I wanted to ask you a question. Is it possible to remove the neck part and keep only the hand part when working using output 1 of the image segmentation 21:33 (the body skin colour one)? In the preview both hands, and part of the arms, are coloured pink/purple but as all the skin is detected the neck, which always remains uncovered, is also detected, ruining the effect. Thanks again and congratulations on everything!

[@carlosgaston3985]

you two are so friking awsome! thanks a lot !

[@oligest4lt]

This has been an amazing plugin to work with! I was wondering if there's a simple way to get the finger point tracking coordinates to reset to the centre of the screen when they are not visible? Currently they go to 0,0 in the bottom left so it constantly pulls anything I have mapped to it to this point if it can't see the intended finger. Thanks!

[@Alexxi-generativeartist]

Hi, I want to ask a question that I have never been able to solve. Why does Mediapipe become unusable and report an error every time I restart? I need to delete and reload it before it can be used normally?

[@Ardsadprm]

Hi! I got a problem, only loading and media pipe logo, can't detect the face in the camera, what is the solution?

[@junweichan-n9i]

Why does mediapipe report an error when I use it?

[@notweirder]

Love the plugin! For some reason the latest version (0.32) shows the facePoints chop as an x,y,z chop (with green, yellow and red) as opposed to the time-slice style chop in the video. Any idea how to convert to the latter? Thanks!

[@kuanfuweimu]

u can try to use analyze CHOP i thought

[@notweirder]

@@kuanfuweimu Perfect, thank you so much!

[@fuegofinder2015]

Awesome. How is the performance of this vs the kinect?

[@AtelieristaMHP]

I am trying to make particlesGPU react to several hands in the same time, but it doesn't seem to work, only with one hand at a time. is there a setting I can change that?

[@janinevenema4201]

does someone know if I can do a movie file in instead of a noise with the image segmentation? Im trying to create my own visuals on the body!

[@TrevorNew]

This is amazing, thank you both for making this!

[@bradcatlett5215]

Excellent tutorial and plugin. Very well made. I'm trying to apply the face feature points with a 3d model to animate the face according to the user. Any tips on how best to accomplish this? I keep running into blockers or dead ends.

[@StrawberryBattlefield]

thank you so so so much for the amazing works!!!

[@saralopez7873]

This is incredible! Thank you so much

[@MikeKluge-o3u]

Amazing, thanks for your hard work on this!

[@roberty9057]

More than amazing ! Thank you so much

[@QISUN-ms5rn]

太谢谢惹！！！！！好棒好棒的组件！ 😁

[@HarryStemmer]

Is it possible to use ndi directly as input

[@spindlewhorls9764]

thank you for your work it's real fun to use this and a lot of possibilities. I have one question: would it be possible to use the mediapipe plugin in td with an Infrared webcam?

[@diyacorrea1023]

you guys are amazing thank you so much!!!

[@doridori8916]

Thanks! i have a question! can i add my hand gestures? (not something that already exists like Closed Fist, Open palm, Point Up ... )

[@LealRaulAndres]

Looks amazing, but i could not get it to work, i get an error that resolutions is up to 720p. Tried putting the resolution down but still no working. Working on a MAc pro with sonoma TD 2022.33910

[@daveydonaldson2750]

what is the best way to use the face mask and replace it with a custom face model, IE make the face track onto a scary ghost or something fun instead of the base White mask. Trying to figure out the best way

[@7cd664]

Did you find a solution?

[@benjavides]

This is incredible! Thanks for publishing it ❤

[@光亮张]

Thanks for publishing it

[@j.adambrinson8415]

Fantastic plugin-I think I can pack up my old Leap Motion Controllers now. Question: Is there a way to permanently have h1 be the left hand/h2 as the right? My experience now means that whichever hand is detected first becomes h1, regardless of whether it's my left or right hand. Specifically referring to the normalized data from Output 1 in the hand tracking component. Appreciate any advice on this, as well as the awesome tool.

[@Rodrigo-h7q]

cool Torin, thanks a lot 4 da work work work. , do you know if i connect 3 webcam to TD, can i use mediaPipe, using each one to track their onw user's hand?

[@blankensmithing]

Hey, thanks! Yes you can just duplicate the MediaPipe component, make sure to change the Websocket Port and the MediaPipe Port to something different than the original component and you can select a different web cam. You can do this with however many webcams you have. I'd recommend turning off all the models you're not using for speed

[@lilyhsiao9315]

Thank you so much for your plugin!!! I have a question about the bound box data, there is an "angle" from the mediapipe face bounding box value list, but in my project, it always shows "0".

[@blankensmithing]

Hey! which model are you looking at? Is it the face_detector, object_tracking, face_tracking component?

[@lilyhsiao9315]

​@@blankensmithing thank you for your immediate response! I'm looking for a face detector. I wanted to get the face/head rotation value, and then I saw an "angle" property, but it always shows"0".

[@lilyhsiao9315]

Hi, I just would like to follow up on this if I did some wrong setting. Thank you so much🙏

[@jjeon0923]

Thank you so much! Such a game changer! Is there a way to out put the wire frame in a top? Trying to just out put stick man figure without the source video underneath it. Again, thank you!

[@romacicada8121]

Guys you are amazing

[@federicaintelisano5639]

Can you connect cameras like arri?

[@neokortexproductions3311]

We appreciate you!

[@borjonx]

how are you getting the z position of your fingertip from a non-3D camera? ty for sharing, this looks incredible - I've never had such a powerful tox up & running to quickly. Only issue was I had to download the latest TD.

[@zablok69]

Hey guys! Anyone knows if there is a workaround to use Mediapipe TD for vertical video / stream?

[@denizz_kurt]

Awesome tutorial Torin!

[@shiangchia4337]

looks cool! thanks for sharing!

[@moonth]

It's amazing! Thank you

[@jaspersegers]

Hey, this is a super great plugin. I have a question though, There's no way to keep a consistent left hand and right hand. Everytime I remove one hand from the frame, the remaining hand becomes Hand_1. This makes it impossible for me to create a controller using 2 hands :(. Does this have an easy fix or is this functionality coming in the future? :)

[@vxkhwz]

Hey, thanks! Question: so there is no way to keep up tp date with git pull atm? So I will have to manually download updates?

[@NoahEckert-u5s]

Hey Torin! Thanks for the great video and plugin! I was wondering if there is a way to track multiple people at once. especially with the pose tracking? or is this not possible with MediaPipe?

[@blankensmithing]

Hey at the moment Media Pipe only has pose tracking for 1 person. It does look like multi-person pose is on the way soon though since their web based demo has multiple person detection(those examples are usually head of what they release to the public).

[@NoahEckert-u5s]

@@blankensmithing ok great thanks for your reply! Diffrent question: How can i use the Plugin with a higher resolution? I want to use the video output in 1080p but i can not use my Camera Input in a Video Device In TOP when i allready selected it in the MediaPipe Container.

[@blankensmithing]

@@NoahEckert-u5s The ML model scales down the webcam size, so using a higher resolution won't give better results. If you need to use a higher res cam you can use Spout Cam, or Syphon w/ OBS to change the resolution of your device and pipe it back into TD as a virtual webcam

[@BACCAJIMIN]

hello thank you so much for the plugin!!! I just have a problem of saving the file. Everytime I save my file and exit, when I open the file again, all the work I've put disappears.. if the file is not saved in the same folder of the toxes folder, the work is still there but it says the file is having a trouble loading components(probably looking for the toxes files) How can I solve this? :(

[@blankensmithing]

Hey if you've lost any progress just put the tox files in a folder called toxes that's in the same directory as your .toe file and re-open your .toe file. In the future there's a toggle you can turn off on all the mediapip tox files in the parameter page under Common called "Reload .tox on Start"

[@BACCAJIMIN]

@@blankensmithing omg thank you so much!!! I tried turning the toggle off from mediapipe.tox file and still had the same problem, I had to actually turn off all the toggles from rest of the tox files in the tox folder and convert them to tox files instead of toe files to make it work!! Thank you so much💗💗💗

[@C0llinsW0rth.]

hmmmm the mediapipe component just keeps saying "loading" i don't think i did anything different than the tutorial, any thoughts on troubleshooting?

[@moonmountain]

Thanks so much for this amazing upgrade to working with mediapipe in TD, I was wondering if there is any benefit to using smaller resolutions than 1280 and if the spout method is the only way to achieve that downsizing? With the previous workflow of mediapipe I was able to get good hand coordinates at resolutions as low as 240x135 with an increase of roughly 8-10 fps. Although I am totally unfamiliar with the webbrowserCOMP and how/where it gets the webcam feed. Thanks

[@blankensmithing]

Hey! yeah the models run off a very small resolution feed, so there's no need to use a large video format. The only way to do it at the moment through spout or obs, which would add an additional delay

[@kamiusu2360]

Hi! I have a problem, when i open a patch all connection from MediaPipe base in disconnected. And i have red circles with cross. How i can connect them again?

[@blankensmithing]

Hey please follow the steps here: kzbin.info/www/bejne/eamXdp-ils9poc0si=jzo-JCOVxF89jOpS&t=139 so I can help debug. Make sure to attach screen shots. If you see a red x on a component, hover over it and include a screenshot of the text that's displayed in the error message

[@kamiusu2360]

@@blankensmithingThank you for you help! Now it works good on my Apple laptop, but on PC it stucks in loading Media Mapper when i connect usb webcamera.

[@kamiusu2360]

I resized usb camera resolution in OBS to 720p and it works now! Thank you very much!✨💜

[@eshu1023]

Hello.. does this work on TD non-commercial as well? I subscribed to your patreon and downloaded some TD sketches.. none of my webcams seem to work (just blank white page)and I am wondering whether it is the TD version! mediapipe downloaded from your Github has the same issue. My MacOS is 14.0 Sonoma. Thanks in advance.

[@blankensmithing]

Hey what version of TD are you using. You don't need a commercial license. It should run on the latest version of TD, but loading into much older version won't work

[@eshu1023]

@@blankensmithing it's the most recent version. I downloaded it this week.

[@pixelasm]

Oh wow, just stumbled upon this. As I just recently used a #LeapMotion in #TouchDesigner it seems like #MediaPipe provides a great replacement as no dedicated hardware is needed anymore.

[@amanpathak2338]

This is my first time working with a plugin on TD, I followed the steps like you said, and referred to the github page but it doesnt load up media pipe for some reason, I selected my Webcam, checked it as well but MediaPipe doesnt finish loading, what should I do?

[@jhahnaaa]

I was facing a similar issue. On downloading the current version of TD (2022.33910), it worked well!

[@blankensmithing]

Hey please follow the steps here: kzbin.info/www/bejne/eamXdp-ils9poc0si=jzo-JCOVxF89jOpS&t=139 so I can help debug. Make sure to attach screen shots. If you see a red x on a component, hover over it and include a screenshot of the text that's displayed in the error message. Make sure to list what version of TouchDesigner you're running as well

[@amanpathak2338]

@@blankensmithing Hi, so I was able to figure out the problem and now it works just fine. Excited to create with it. For anyone facing a similar issue, I used OBS studio instead of spoutcam for the webcam stream [PC] . In the video input of Mediapipe I used OBS Camera (which was scaled to 720p) and thats how it started to work. I created a new scene in OBS and simply added my webcam to the scene. Keeping OBS open I restarted TD and I could see "OBS Camera" as an input in the drop down. Hope this helps anyone facing this issue. Cant wait to get started with this plugin finally:) Thanks to the creators again!!!

[@blankensmithing]

@amanpathak2338 Hey! I'm glad you got it working :) Thanks for the tips so we can pass it along to other people

[@mattphillips3024]

WOW! incredible

[@mlnima]

I am trying to use it but it gets extremely laggy and fps drops on camera even I have 3090 gpu and I9 cpu, can anyone help?

[@blankensmithing]

Try turning off the models you're not using. It's not intended to have all of them running.

[@lee_sung_studio]

Thank you. 감사합니다.

[@AliTanUcer]

Another option is NDI in NDI out..

[@xswl5164]

COOOOOOOOOL!!

[@alchemist.D]

nice job

[@yosukeDVJ]

Amazing! This is huge. How can I get Syphon to show up in the cameras dropdown? I'm using someone's proprietary OpenFrameworks Syphon App to get a KinectV2 working on my M1 Mac. I only get the video feeds tho so no CHOP data. Incorporating this mediapipe plugin into my workflow would nearly complete finally having a fully functioning kinectv2 in touchdesigner on apple silicon!

[@yosukeDVJ]

never mind! Using ndi.

[@chrismills8404]

I feel like your intro gets louder every vid and its a slow prank :D

[@blankensmithing]

haha It's always the same volume, but may have to do this now :)

[@NA-cw4pj]

Hi, thank you for this beautiful project, but I am having an issue, why the MediaPipe box is always on Loading mode, and it print an image of Loading. I tried reseting the process but still doesn't work for me. Can you help me? Thank you!