Coding an AI to Play Fruit Ninja

In this video, I use Python and the Ultralytics module to train a custom YOLO V8 CNN for object detection to automatically play Fruit Ninja. If you have any ideas for improvements or things to teach an AI or develop next, leave a comment below; you might find a video with your idea. If you enjoyed this video make sure to like, subscribe, and leave a comment below.
The idea for this video came from a recommendation to watch "Using Image Recognition to DESTROY Fruit Ninja" by CodeNoodles.
Link to the video:
• Using Image Recognitio...
Link to more in-depth details about CNNs:
stanford.edu/~shervine/teachi...
Link to the code:
github.com/TwoPoint0200/Fruit...
Thank you for watching!

[@Froothy]

since the background stays still you can compare the current frame to the still snapshot of the background. now you have an object mask. with this mask you can choose to only detect bombs and if its not a bomb then slice it, if it is a bomb then ignore it. this would only work for the fruits that dont zoom the camera in

[@TwoPointCode]

Yes that would be another solution to this problem. Something I noticed while working on this project is that I could only move the mouse so fast before the movement wouldn’t be registered. With an object mask the sliced fruit, “critical” text and “combo” text would also be detected as something to slice which could cause problems with this speed limit. Would be interesting to see how it compares though!

[@mathew6041]

I am glad to see such a good implementation of the project. But right I'm crying my eyes out like crazy. I made the same project inspired by the same video using the same architecture word to word, but left it half way. You have beaten me to it. Good job TwoPoint Code

[@mohammadazad8350]

When I watched that video, I was disappointed by the AI's low score and wanted to see a better version. I guess I finally got one.

[@samsterjam5190]

Amazing! Fantastic job explaining the use of the CNN and code while keeping it entertaining! This should blow up

[@rafaelsantiagoaltoe6606]

Amazing! I am just starting to study AI, hope to soon be able to do stuff like this. I really liked the video, I liked how you explained what you did in a concise manner, but still without hiding the needed information.

[@mofalkmusic]

Rly enjoyed this

[@foxstamp]

Great video!! It’s a shame that you don’t get more recognition

[@TwoPointCode]

Thank you!

[@DefGamer-ek1zt]

Very nicely done and well explained!

[@jotaro6390]

Amazing video, I really enjoyed it. I was thinking about scoring combo cuts by slicing once through multiple fruits. I had an idea of predicting trajectories of fruits by first frames, then calculating the the point in time so that the program could cut exactly where fruits are aligned. But I am not sure if it is practical as the program in the video already makes few mistakes. I don't have much of experience to implement it, but I would definitely want somebody to do this.

[@TwoPointCode]

Thank you, I really appreciate it! The idea of tracking the trajectory of the fruits and bombs to get better combos would be interesting. One thing I noticed while developing this is that the bombs don't always follow an arched trajectory and seem to sometimes move erratically. This can be seen with the rightmost bomb between the times of 14:35 and 14:37, which could make tracking difficult.

[@bluemax6668]

Yes i also want to do that since i saw codeNoodles video, like when a fruit is about to fall down out of the screen we find a path that cuts all the fruits, i think that calculating trajectories is pretty doable but i think you could do that more simply. You look at the current velocity of each fruit and you cut along this velocity vector, so even if the fruit moves a bit it will still be in the cut trajectory.

[@bluemax6668]

and to generate the trajectory maybe do a bezier curve with fruits positions and a point just outside the bombs bounding boxes, and if the trajectory goes through a bomb you select an other point

[@Lucas-pj9ns]

Excellent, love the optimizations u did!

[@paulrouxel7526]

Excellent video !!

[@blue_birb]

I've got an idea to make the pathfinding better. *When 1 fruit is on screen, find a corner of its bounding box and jump the mouse from the corner to the opposite corner *When 2 or more fruits are on the screen, calculate the center of all the fruits and jump the mouse from center to center. *When 2 or more fruits and a bomb are on screen, calculate the straight line between the centers of the fruits that you are about to jump between, if they intersect with the bounding box of the bomb, find the closest point on the line to the center of the bomb and calculate a vector from it to the closest corner of the bomb's bounding box, multiply it's magnitude by 2, and make another "ghost fruit" at the new point to add an extra stop between the fruits If I wasnt clear about the last case then what I meant it basically find if the path between the fruits intersects with a bomb, if it does, find the closest point on the line to the center of the bomb, and push it out 2 times its distance to the closest corner towards the corner to make sure it gets far from the bomb and add that new point as another checkpoint. *2 or more fruits and 2 or more bombs, perform the same thing as last case but do it for every bomb again and again until a solution is found All this should take like 2 milliseconds or something if implemented correctly

[@TwoPointCode]

I like it! If I’m understanding correctly, one thing I’m not sure about is when there are two or more bombs, how do you guarantee that a “ghost fruit” doesn’t get placed inside a different bomb? Or, if it does, how do you handle ignoring the “ghost fruit” and going around the bomb? Depending on the speedup, this could be worth implementing and using for zero or one bomb as it is a very common scenario in this game.

[@delfini8135]

Great Video!

[@joeljacob4546]

Cool video

[@heret1c385]

Great video

[@MegaTpyn1]

Seems like the model doesn't really like strawberries, I noticed that 4 times it recognized strawberries just a little too late, thus failed to slice them. And in some cases it recognized them late but still was able to slice. Maybe with some tuning in that direction it could achieve an even better result

[@TwoPointCode]

100% agree. I'm not exactly sure why it struggles with strawberries, but I assume it's because the model uses lower quality images for faster inference and has a hard time distinguishing the difference between uncut and cut strawberries. Could definitely use more examples of strawberries.

[@robert_trirop]

Good job :)! I tried to do the exact same thing but on my iPhone. My plan was to film the iPhone with a webcam and slice the fruit using an ESP 32 board emulating a Bluetooth mouse. The mouse emulation works but I did not test the performance (speed/latency) yet. I want to use a GStreamer pipeline to capture the camera stream, process it and do inference. GStreamer can also be used to capture the computer screen and there are pipeline elements (plugins) from NVIDIA to do TensorRT inference right in the speed & latency optimized pipeline (DeepStream SDK). The inference results can be fed into a python script. I already trained the model with about 1500 images. Took quite some time to label them manually. I got surprisingly good results using the smallest sized model (nano). Which size did you use? Mine ran with 5ms inference time on my 1080Ti. Maybe one could predict the future position of the fruits to compensate for the latency. Have you analyzed why your code fails eventually? Is it because the model did not detect a fruit in the few frames where it is visible?

[@TwoPointCode]

Thank you! Oh, wow that sounds interesting. I also used the Nano version of YoloV8 for this project. I wonder if predicting the future position of the fruit is viable, and if so, how much it improves the score. I almost feel like the time it would take to predict the path could end up harming the performance. Based on my testing, the code eventually fails due to either randomly long inference times, fruits not being detected correctly, or fruits being too close to a bomb when above the screen threshold I created.

[@LUVVEOUS]

Could there be used a tragectory prediction for fruits to optimize the cutting path?

[@TwoPointCode]

Yes, this could be done to improve the cutting accuracy! I decided against doing this due to the complications in finding the same fruit between frames and predicting the downward acceleration as it would need to be done very quickly to see the benefits.

[@LOKI101-zt2dy]

Bro give the code for it. I want to try it out.😁😁

[@TwoPointCode]

Wasn't going to release the code due to the dependency on capturing specific parts of the screen and the problems that can arise with different Cuda, Python, and Pytorch versions, but I have just updated the description to include a link to the code. If you decide to use it, the screen coordinates will most likely need to be changed to work on your monitor!

[@LOKI101-zt2dy]

@@TwoPointCode You are the man 😍😍 But I am not into any coding.I thought I can execute your given code directly on my pc.But you said I have to tweak some of it which I don't think I am capable of. But anyways you shared your code without any hesitation made my day...THANKS👌👌👍👍

[@cklangat7118]

Where can I find the code?

[@TwoPointCode]

Originally wasn't planning to release the code, but because you and another person are interested in it, I have added a link to it in the description. If you decide to use it, the screen coordinates will most likely need to be changed to work on your monitor!

[@reallight_]

Seems like you stole this from CodeBullet

[@TwoPointCode]

I’m actually not aware of a CodeBullet video about fruit ninja. What is the title of it?