Tello drone and computer vision: selfie air stick

The Tello drone is a lot of fun by itself. But it is also the opportunity to apply some well-known computer vision techniques to expand its capabilities !
Github: github.com/geaxgx/tello-openpose
I want to thank all the people who wrote and shared the great libraries/programs I used here :
- github.com/hanyazou/TelloPy : DJI Tello drone controller python package,
- github.com/CMU-Perceptual-Com... : Real-time multi-person keypoint detection library for body, face, hands, and foot estimation. This is an amazing library !
- github.com/Ubotica/telloCV/ : Ubotica wrote a code for the Tello to follow a color ball. Instead of starting from scratch, I used his code. It makes me saved a lot of time for UI !
Music credits:
- The Place Inside - Silent Partner • The Place Inside - Sil...
- Cello Suite #1 in G & Your Call : Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License
creativecommons.org/licenses/b...

[@GenshinPlayerBR]

How does someone dare to hit dislike button in such video? This project is absolutely amazing! 😲

[@ianbenicio1035]

you probably dont care but if you guys are bored like me during the covid times you can stream pretty much all of the latest movies and series on InstaFlixxer. I've been binge watching with my brother these days xD

[@bodietripp8218]

@Ian Benicio Definitely, I have been using instaflixxer for years myself =)

[@tahaanwar5224]

I've watched your other videos too, and as a Vision professional myself i'll say this....you sir are on a whole different level...these combination of ideas of yours are Classic

[@geaxgx1]

Thank you sir ! I really appreciate your kind comment.

[@aaratmajithia445]

Excellent project! Thank you for sharing.

[@RakeshNair007]

Really impressed by your project!! Keep up the good work, subscribed to your channel as well.

[@rgdi]

Just Amazing! Congratulations, impresive project!

[@sebastianreal4363]

You are genius, incredible utilization of the tello

[@cnjieyang]

Really love this video and the codes! You demonstrated perfectly what can be done even by a toy drone.

[@MatteoFrancia]

Impressive job! Thanks for sharing your work and a bit of behind the scenes! 👍🏻

[@attilabpc542]

I'm absolutely amazed. This opens totally new horizons. Thanks.

[@r.3424]

Congratulations ! great job, very impressive

[@oregomm]

Fantastic work! Thanks for sharing. This is definitely going on my list of projects to try out

[@kadumaverick7013]

Congratulations! It's so cool man, I've loved it. Definitely inspiring.

[@user-jy6kd7xg4d]

as someone who learn machine vision, watching your video feels like when a painter discover bob ross for the first time. beautiful!

[@geaxgx1]

Thank you for your nice comment !

[@ShredderJT]

@@geaxgx1 How do you get the video so fast? I am on windows, and have only been able to get the video at very slow fps with lots of delay

[@geaxgx1]

@@ShredderJT The fps you can get depends on your GPU. I was using a gtx 1080ti.

[@yc67125]

Using simple tello drone to acheive that is amazing!!

[@zensic]

Man, this is so cool! Thank you for sharing!

[@LorenzoGoriniFrisbee]

genius. really great ideas and thanks for the inspiration and the code!

[@dougson_]

That is huge! Thank you!

[@zonethin7211]

So inspiring! Thank you, Sir!

[@PERTHWESTOZRC]

This is Amazing, well done and thanks for sharing!

[@jcurwen31]

Incroyable! What you did is amazing Thank you for sharing

[@kcchiok]

this is solid! good job!

[@AlbertPhan]

Very slick!! Nice work.

[@lforte01]

Amazing project... Simply Fantastic!!

[@DarianCraciun]

I love you. :)) You have some of the coolest programming projects out here.

[@minesti3725]

this video is so well explained I finally can picture now how to create an object tracking algorithm!! thank you so much !!

[@Augmented_AI]

Really amazing!! Love it

[@chillacombs8790]

smart project sir...i was seriously looking for such,thanks for sharing.

[@philippmittmann2188]

Awesome! Very good idea & video. Impressiv!

[@Ree1981]

That was seriously impressive. Thanks. :)

[@AndruXa]

Great work!

[@brainjohnaboze3659]

Amazing work, building on it

[@hashbrown5164]

This is amazing!

[@rafael2010tlax]

Good job, regards from Mexico

[@pereiaslav_dzhura_club]

thanks a lot for code Sir! I will teach kids on your examples!!

[@maggichannel2526]

Wonderful job sir 👍👏

[@LoboBRT]

Great work

[@gavinxiaozu]

This is the most awesome project I have seem!!!

[@Draxen]

Absolutely awesome!!

[@simolx]

Really cool, i want to make my own now

[@culturerajasthanka1540]

You have done an excellent job it's amazing ✨😍🙌

[@dwightyou5546]

Wow!! Excellent!!

[@JulioCesarAyalaMendoza]

Wow! excelent job :)

[@seobin9489]

WOW, Amazing!!

[@serhatb.5703]

Amazing project!

[@henkoegema6390]

Well done !!!!😄

[@vinhdang678]

love all of this, love you sir

[@EdjeElectronics]

This is so cool 😁😁😁 nice work!!

[@geaxgx1]

Nice to see you here, Edje !

[@blubberflutsch]

Excellent!

[@sucrose]

Subscribed, you've inspired me to do something similar, thanks!

[@geaxgx1]

Thx, it is rewarding to read that !

[@hassanaleem1975]

great stuff! subscribed!

[@gavinknight8560]

One ML engineer to another, dude, I am not worthy, great job!

[@tuttoto]

Thank you for sharing... nice music ;)

[@jamesg6625]

cool, gonna try it.

[@fabiocosta3003]

impressive!

[@keshav2136]

Woah!!! 😍 Amazing 🤩🤩

[@vxkhwz]

amazing!

[@weirduserid]

Fabulous!

[@cryptoevolution5250]

Amazing

[@vkcvkc8910]

Awesome

[@jeremyalfananda8659]

You're excellentl!

[@rklauco]

This is just brilliant! I have the drone already, not used too much, but this would be perfect. If this is able to run on RPi zero W or something like that - that would be amazing...

[@geaxgx1]

Thx! Unfortunately, to run Openpose, you need something more powerful than a Pi.

[@Raven3one]

What about a cellphone? Like snapdragon 855?

[@geaxgx1]

Well, you can run a pose estimator on a phone (if it is not OpenPose, you can use another one like Posenet maybe lighter?). But I am not sure it can run fast enough to use it to "comfortably" control the drone. Imagine you can only decode 2 frames/sec, you will need to limit the max speed of the drone to a low value otherwise the chance to hit a wall is too big. In any case, you will need to tune the settings of the PID controller accordingly to the decoding rate.

[@zoombapup]

Nice! one thing you might try, using the tello to take images for photogrammetry. I might buy one and give it a go myself.

[@geaxgx1]

Thx! I have already tried photogrammetry with the tello. It was outside and using frames from the video stream, for a poor result, probably because of the low resolution of the frames. I will try again but this time by taking 5M pictures.

[@zoombapup]

@@geaxgx1 Best advice for photogrammetry is to take as sharp focus pictures as you can. I live in the UK so there's a lot of overcast weather so its almost perfect for photogrammetry (avoid hard shadows). Having a programmable drone that can take images around an object seems like a perfect use-case though.

[@xenophilie]

Very cool.

[@Alexander-em7kn]

Respect sir.

[@yoriarif]

I want to try when I have the Tello.. 😁

[@ropopal1094]

Epic!! i love computers, tech, and quadcopters... do you think you can make a video or explain somehow how to do all that? i mean how to install all what is needed on the tello to do these hands tricks? thank you :)

[@geaxgx1]

You don't need to install anything on the tello itself. But you need a PC with a powerful nvidia GPU and install a few packages/libraries as explained in the github : github.com/geaxgx/tello-openpose

[@gorabramyan]

Hey, man! It's a great job. Thanks. I'm trying to figure out, can i program drone to fly every 15 minutes from the base, check for small trash on the table, pick it up if there is a trash and throw it to the garbage, and then return to base for charging?

[@alchemyx]

What an awesome project. I wonder how it will behave when it sees more than one person?

[@geaxgx1]

Thx! If there are multiple bodies in a frame, the program currently selects the one which appears to have the bigger size and ignore the others. The size is evaluated by measuring the length of some segments of the skeleton. Often it is the body which is the closest to the tello.

[@anuradhajayatilake1093]

Super

[@Jayhhardy]

Amazing. Truly. This could replace cameramen. Lol.

[@maloukemallouke9735]

Hello sir, Thank you for sharing, I want to ask you which graphic card are you using for training the models of Deep learning ???

[@BUDIVLOGER]

good

[@alexisdelforges235]

When you search for a project with the drone and openCV and a guy did all the hard work :D

[@alexmb16]

Amazing project ! Is it possible to move the drone camera from 0° to 90°?

[@geaxgx1]

Thx! No, on the Tello, the camera is fixed.

[@goktugakn8254]

Is there any drones that more powerful (big, has sensors etc) than this with SDK support? How do you overcome the stabilization ? Great job!

[@geaxgx1]

Thx ! I am not an drone specialist, but yes there are bigger drones that have their own SDK support, like the DJI Mavic (developer.dji.com/) or the Parrot drones (developer.parrot.com/). The Tello is more like a toy but it has its own stabilization mechanism and it is working great ( I don't have to care about it).

[@RomIlay]

can you guide me to how you did the pose recognition? did you use ai for that or jsut check which points are connected to each other and in want locaion? plesae someone answer if you know

[@geaxgx1]

I am not using AI for pose recognition but just do computations on the landmark locations. For instance, to determine if hands are crossed, I can check if the distance between hand landmarks are below a threshold.

[@marcelf.284]

Wow, great project! I am thinking of buying a tello drone as well in order to learn programming that way. I already have a basic knowledge of coding with python and would like to start an ambitious project for myself. Therefore I would appreciate, it if you could answer me following question: Would it be possible to go outdoors and make the drone follow you while running for lets say 50 or 100 meters recording you from the side? I'm thinking of wearing a colored cap for example in order to make the drone follow me by flying sideways and recording me running.

[@geaxgx1]

Thx! From my experience, detecting a color to follow an object in a image is not reliable when you don't control the lighting conditions. Depending on the sun, the clouds or the shadows from the tree, the original color can look very differently in the image. Using a person detector is probably a more robust option.

[@ramsu8220]

Hello, can I please know how to make openpose work, everything is just fine except that..

[@habilpekdemir5703]

i will try to do

[@RamboLastManStanding]

Great work! With RTX 2070 Super GPU I am only able to achieve 30 fps before running openpose and 10 fps after. The frame rate seems to cap at 30fps and 10 fps respectively and does not change even if i downscale/upscale the images.

[@udstv.8751]

have you tried this project ?? i am having issue in the last stage, after running the code , window appears that shows video ,fps ,etc but with in a minute it stucks. no errors it just stucks

[@PeakedInterest]

Hi there, saw your video on Reddit and it prompted some questions that I'd like answered if you'd be so kind... 1. Can it be programmed to focus on one target? 2. What happens with multiple targets in the room? 3. Can more expensive drones be programmed this way?

[@geaxgx1]

If there are multiple bodies in a frame, the program currently selects the one which appears to have the bigger size. The size is evaluated by measuring the length of some segments of the skeleton. Often it will be the body which is the closest to the tello. If 2 bodies appear to have close size, the tello may "oscillate" between them. To make the tello focus on one particular target, we could use face recognition, so that the tello can distinguish between several faces. Face recognition alone wouldn't be enough because it works well only when the face is facing the camera. By using tracking in complement, we should be able to keep the focus when the face is not facing the camera anymore. I am not very familiar with drones but I know that some more expensive drones can already do that kind of things (body tracking and gesture recognition), and in a more efficient way than mine (because their program is embedded in the drones and they certainly don't have the processing power of a gtx 1080ti on board :-)

[@PeakedInterest]

@@geaxgx1 thanks for taking the time to respond. It was a great video with an interesting concept

[@TheMisterBobization]

Congratulations and nice work! I'm wondering about: - Two people are present in the same frame, what's the behaviour of the drone? - How can you avoid collisions with walls eg behind the drone looking at you? Thanks!

[@geaxgx1]

Thx ! - Two people are present in the same frame, what's the behaviour of the drone? The drone only follows the one which appears bigger in the image. It means that if the people are moving, the drone can switch from one guy to the other one, which is probably not the behaviour we would like. - How can you avoid collisions with walls eg behind the drone looking at you? I can't ! The Tello has no sensor to "see" behind or on the sides. So the guy who is "piloting" the drone has to be careful :-)

[@goktugakn8254]

@geaxg1 Hi. Is there a more powerful drone that we can program it with this SDK like you did? For example a drone that has sensors, and much bigger than this ? Great job!

[@ehsandildar2550]

Dear sir i installed all the libraries and openpose is in running condition i also run the examples but how will i open your project file of tello_openpose_master in openpose

[@killerluiz89]

Would I be able to measure your distance without using OpenPose? I have a project that is to follow any object that has been classified on a CNN. The only idea I have is whether the detection box is large or small, which would show if it is approaching or moving away from the drone.

[@geaxgx1]

You have probably observed that the size of the bounding box can vary a lot from one frame to the other even if the object is almost not moving. So you should probably use a moving average of the box size to decide if you are approaching or moving away.

[@airdog7566]

I can only repeat: absolutely amazing - Very creative and impressive solution Do you have any experience in Xcode/Swift as development environment in this area ? (And thank you very much for the additional info at Video)

[@geaxgx1]

Thanks ! Ah no, I have zero experience on Swift.

[@claudiopascarelli3159]

Great video! Are you using a "normal" Tello or the EDU version?

[@geaxgx1]

Thx! The "normal" Tello.

[@ivanprojic3082]

Wonderful project! You earned a new subscriber! :-) Btw - Do you think this project can be done through Google Colaboratory in case the GPU poses a problem?

[@geaxgx1]

Thanks ! It is a good question but unfortunately I don't know if it is possible. The tello drone needs to communicate with a process that is on the local tello wifi network, and this process would also communicate with google colab.

[@andreasspiracopoulos5628]

Hi there , very good indeed. I am Trying to install everything on windows . Python 372. PyAv gives a bunch of syntax errors Though . Struggling with ffmpg . Is it necessary and how do I go about it ?

[@geaxgx1]

I am sorry I can't directly help you on windows. But pyav is just used to decode video frames sent by the tello. There may have alternatives. Maybe you can take inspiration from these examples : github.com/hanyazou/TelloPy/blob/develop-0.7.0/tellopy/examples/keyboard_and_video.py (which relies on mencoder), or github.com/dji-sdk/Tello-Python/tree/master/Tello_Video (which relies on h264decoder). Good luck !

[@viju85]

Hi. This is super amazing and inspiring. I am trying to build openPose for python on mac and really struggling. Also when compiled openPose and ran using CPU_ONLY mode, FPS is very poor (0.4) on the latest MAC. The FPS in your demo is 25+. Was there anything different than the specified configuration in the openPose github to get it to 25+?

[@geaxgx1]

Thx ! I was using a powerful GPU (Gtx 1080 ti) in order to get such FPS. If I had to make this project today, I would not used Openpose, but a recent model like Blazepose or Movenet that can run much faster on a CPU.

[@viju85]

@@geaxgx1 thanks. Just tried blazepose mediapipe sample and it is way better. Getting around 15 FPS. BTW this video is really good. Have shared it with my friends too. Especially the Morse code touch was brilliant.

[@claudiocimarelli]

@geaxgx1 thank you for sharing this video! Did you use the normal Tello (SDK1.3) or the Tello EDU (with SDK 2.0)?

[@geaxgx1]

The normal Tello. I don't rely on the official SDK1.3 to communicate with the Tello, but on the python package github.com/hanyazou/TelloPy

[@claudiocimarelli]

@@geaxgx1 Thanks for your answer. So you wouldn't say there is the need for the EDU version for controlling the drone. Because I can only find the normal Tello around here in the shops.

[@geaxgx1]

Exactly, no need for the EDU version. Actually I don't even know if TelloPy would be compatible with the EDU. A more critical point than the tello version, is to have a good GPU to run Openpose fast enough.

[@claudiocimarelli]

@@geaxgx1 ok thanks. I will buy one for computer vision educational purpose :)

[@zonabugar]

unbelievable! a

[@edwardthebest9958]

hello. I wanted to ask if this can be used once you cloned it or do you need to train it or something similar

[@geaxgx1]

First all pre-requisites need to be installed. And after the cloning, it depends on your hardware configuration and at how many frame per second you GPU can do Openpose inference : the parameters of PID controllers in the code were tweaked to work well on my config and should probably be tweaked again for another config.

[@ItHackBytes]

Everything is perfect, but i want to know which hardware inside drobe is being used for the coding, or for the motion controller... (Means which hardware is used for communication of drone with openCv or OpenPose )

[@geaxgx1]

Thanks! I don't know what is precisely the hardware inside the drone, but actually I don't really need since I rely on the Tellopy python package that manages the communications between my program and the drone (via wifi).

[@ItHackBytes]

@@geaxgx1 okay, i know , python or languages made communication between operator and drone, but the thing is, in which hardware has been this code is being implemented ? ... But i am expecting u don't know, but can u prefer from where i will get my answer ? Because i really need to know the hardware for my project that contains this coding ..

[@geaxgx1]

@@ItHackBytes If you ask what hardware is inside the drone, this page says the tello uses a Intel VPU Movidius Myriad 2: gobot.io/blog/2018/04/20/hello-tello-hacking-drones-with-go/ Not sure if it answers your question :-)

[@ItHackBytes]

@@geaxgx1 i will read it, Thanks a lot for u time and u told what u know..

[@juallive6678]

i see for tracking is very good realtime and fast, but how if in front drone stand 2 person, its will lock both person or first person find?

[@geaxgx1]

In the current version, if it detects several persons, it will consider only the "biggest" person in the image, the one which have the largest shoulder width for instance. It is not a very satisfying solution because if 2 persons are close to each other, the focus may oscillate from one person to the other depending on their movement and their position relative to the drone.

[@sebastianreal4363]

Do you think some sbc could handle it? I'm not thinking in raspberry pi, but Nvidia Jetson nano or his bigger brothers, maybe another one I didn't know

[@geaxgx1]

I have no experience with IA single board computers. Nvidia claims they get 14 FPS iwth their Jetson Nano on a lower resolution Openpose (developer.nvidia.com/embedded/jetson-nano-dl-inference-benchmarks). 14 FPS should be enough. Personally, if I had to use one of these sbc, I would have a try with the Coral edge tpu (on its dev board or even in complement of Jetson nano), which seems to have much better perf on classification or object detection tasks (blog.raccoons.be/coral-tpu-jetson-nano-performance)

[@prithvikrishnaalluri8652]

can u share the process of how to do the face detection with tello drone...

[@geaxgx1]

In this project, I don't directly do face detection with the drone. Instead I rely on pose estimation to localize the different parts of the human body, including the head.

[@jackotatty]

My tello drone keeps moving to the left even though I'm not commanding it to do anything. How can I stop this?

[@quim2040]

Hey, great demo! Were you able to teach the drone to keep distance with you?

[@geaxgx1]

Thx! I can have an indirect and very rough approximation of the distance by computing the ratio shoulders width measured in the image / image width. For instance, for the drone to keep a constant distance to me, it just need to keep that ratio constant by moving forward or backward. Of course, it does not work anymore if the drone sees me from the side view, because it will think I am further away than I actually am, but that was enough for the demo :-)

[@quim2040]

@@geaxgx1 wow that's clever! So facing the situation that you want the drone to follow you if you're turning left/right, would the tracking box detect you or it would lose you? Have you tried it? Thanks in advance!

[@dhruvshindhe4671]

Great work!! How are you classifying the pose ?

[@geaxgx1]

Very easily, by using the coordinates of the keypoints given by Openpose. For instance, to take a picture, I cross my hands under the neck. To recognize this pose, I just need to check that the keypoints for the right wrist, the left wrist and the neck are close to each other. To ask for an immediate landing, the left wrist keypoint needs to be close to the right ear keypoint. Just a bit more complex to make the drone move (forward, backward, left, right), because I need to look at the angle between pairs of keypoints. I am lucky here because I just have 6 poses in this project. If I had more poses to recognize, I would probably need to train a classifier.

[@dhruvshindhe4671]

@@geaxgx1 correct me if i am wrong here , you are calculating the euclidean distances and checking if they are are below or above a threshold ,you are not using a classifier.

[@geaxgx1]

Yes, exactly ! Euclidean distances and that kind of stuff. No need for a classifier here :-)

[@dhruvshindhe4671]

@@geaxgx1 thanks a lot!!

[@dariusdarius9506]

Sir can I get the source code i am working on this project in which drone detect the face with pose estimation and follow and using gestures it does some tasks it will really help me in FYP if you share the source code