Using Image Recognition to DESTROY Fruit Ninja

To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/CodeNoodles/. You’ll also get 20% off an annual premium subscription.
This project has been a long time in the making. I set out last year to automate Fruit Ninja with image recognition, but I ran into a couple of road blocks along the way. But luckily, this year I decided to try it again, and it worked out fantastically to say the least!
This video was sponsored by Brilliant
LINKS
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
Support the channel: ko-fi.com/codenoodles
Itch.io: codenoodles.itch.io/
GitHub: github.com/OfficialCodeNoodles
MUSIC
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
Shoutout to @qumumusic for making wonderful remixes of some of my favorite video game songs! Check out their channel if you haven't already!
• The Legend of Zelda - ...
Roboskater by Audionautix is licensed under a Creative Commons Attribution 4.0 license. creativecommons.org/licenses/...
Investigations by Kevin MacLeod is licensed under a Creative Commons Attribution 4.0 license. creativecommons.org/licenses/...
█▀ █ █ █▄▄ █▀ █▀▀ █▀█ █ █▄▄ █▀▀
▄█ █▄█ █▄█ ▄█ █▄▄ █▀▄ █ █▄█ ██▄

[@CodeNoodles]

To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/CodeNoodles/. You’ll also get 20% off an annual premium subscription.

[@deltadeltagrand2842]

I feel like something that could improve it is if you had the code specifically look for any black spots that would mark a Bomb and label areas around them as dangerous, that could help the code avoid bombs as well as make sure it doesn’t try to hit fruits too close to bombs. I would also make it so that it doesn’t scan too close to the bottom of the screen as a bomb could come up right after or at the same time as a fruit and get hit.

[@gaco577]

if object = bomb is bad else is good

[@CodeNoodles]

Basically

[@catWithFunkyFace]

if only it was this easy

[@IgsihziysPleazure]

That looks like something from scratch.

[@cybi124]

if (object = bomb) { donttouch } else{ destory }

[@Duckless137]

==*

[@lukeseaman2994]

The somewhat unoptimized nature of the program gives it a lot of personality and comedic value 10/10

[@CodeNoodles]

Thanks, that makes me feel better 😆

[@IntrovertSinceBirth]

Programming also makes mistakes like humans do?!

[@Dusti_buns]

​@@CodeNoodlesGreat video. I loved it

[@johnsimpsen5]

It would be cool if the program waited a while before the fruits were on screen and then calculate how many there are. If there are more than one, it tries to slice them in one go instead of a bunch of slices.

[@Rjciralli]

I can already imagine that constantly catching bombs in the big slices

[@dylanherrera5395]

​@@Rjciralliperhaps a pathfinding algorithm to avoid the bombs, although this might be too slow

[@advance64bro]

The program does what’s efficient

[@TheStolenBattenberg]

@@advance64broit uses python…

[@inakiorozco881]

@@advance64bro the program does what is designed to do, It can be efficent with fewer slides if its designed that way

[@rockinggamerdude]

The problem I see is it doesn’t wait for all the fruit to make sure to get a combo for a points bonus

[@mateuszpragnacy8327]

IT is possible to train ai that way just Smart genetic algorithm or other

[@advance64bro]

It doesn’t even know how the game works, it’s a program that does what is told

[@mateuszpragnacy8327]

@@advance64bro yes but no. You can train it just genetic algorithm and train 10 ai at time and ai with biggest score replicates like natural selection and there is a chance that ai learns to mąkę a combo

[@Cozumii]

Isn’t that lost score made up by how many critical it gets?

[@ethanperry8]

@@mateuszpragnacy8327that would take a long ass time and processing though

[@zobiah6263]

This feels like a strong candidate for reinforcement learning imo. Just give it the average color values you have already collected, a reward function based off the in-game score system (obviously make bombs a high negative), and watch it go.

[@4_real_bruh]

Could also just use a single convolution operation with a filter of 7x7 since the bombs always have a minimum size, then compute the average of the colors as he did and if its not white or black, slice the fruit by going 10 pixels in either direction from the pixel

[@soothingstationW]

@@4_real_bruh Yeah i had the same idea. I thought about going by size as well. He could add this on top of the color to prevent more false positives.

[@ryans3979]

Being somebody who works in the computer vision field, I feel like it would've been simpler for you to convert the image to HSV, take the value (lightness) channel, and then binarize the image by checking if it's less than a certain threshold. From there you can see how many connected pixels there are, and if there are more than say 1000 black pixels, its a bomb.

[@cvabds]

This could be better optimized if you didn't based it on color recognition but actually on moving pixels on the grid you created. Based on that principle, you should need to only recognize the black color of the bomb, everything shouldn't be biased on only color. I saw it bug on the background splashed fruit some times

[@ryans3979]

This would add a small delay because he'd always be one frame behind (need to take the current frame and subtract the previous frame pixels that didn't move), but I do agree that the optimization would probably speed it up enough to be worthwhile. He should also not be processing the full RGB image, especially because he's in python, that's obviously going to be slow. Turning it into a grayscale or HSV (lightness channel) 2D array and doing some sort of processing to check for the darkest pixels would definitely be faster.

[@cvabds]

@@ryans3979 nice! Thanks for the input! That makes sense!

[@ilovecoffee4978]

​@ryans3979 could he not make it identify the position and type in one frame, wait like 3 frames, and then identify the same objects position for average velocity, then swipe afterwards based on delay?

[@madhuchowdary8826]

Wont that be pretty hard, as the program needs to calculate a path to slice, a path which does not include any bombs in between

[@emanuelemanzoni5191]

how would you detect moving pixels

[@shadow_blader192]

You are writing functions without space between them 😭

[@Trevorus1]

May I ask what’s happening in your profile picture with the mimi sentry?

[@aaronking2020]

​@@Trevorus1 mini sentry meets big thing

[@trystankitty5393]

@@aaronking2020why

[@romancatholicgameing]

​@@Trevorus1 It takes someone with experience to recognize that pfp's image and what happened in it... I don't know if I should be horrified or impressed.

[@letter_o_hyphen_letter_o]

is bro's argument invalidated by deranged pfp?

[@sbd2639]

I have made a program in python to automatically complete tasks in among us and wrote code for almost all tasks for the first map. I faced the same problem in some tasks ( like clean vent, clear asteroids etc) where the image recognition would not work properly due to random rotations of the sprites on screen. Your solution to the problem might be perfect in my program and I am gonna try that soon. It might be even better suited since there is even less chance of false positives ( which was caused by the splattering of the fruits on the wall in fruit ninja ).

[@TheFurry]

awesome project! Are you going to do a video on it?

[@sbd2639]

@@TheFurry ​nah probably not. I just made it because my USB mouse broke and using laptop's mousepad was very annoying. If you are interested, I could give you the code but since I made it just for me, I didn't take the screen resolution into consideration, so it only works on the monitor with the exact resolution of 1920x1080.

[@HarshWeave9487]

Interesting.... Would you mind if I could see the code? If you can share the link, I would be grateful.

[@sbd2639]

​@@HarshWeave9487 This is my first time using github so sorry if there is any mistakes.. Also, since I just made it for me, almost every pixel coordinate is hardcoded so there are just random numbers everywhere. If you have any questions, you can just ask here, I am online pretty regularly.

[@sbd2639]

​@@HarshWeave9487 I commented the link here but I think it got deleted by youtube. Are there any other way to share the link???

[@McTuber42]

The rhythm of the slicing syncs surprisingly well with the music after 8:25

[@-ShoeCat]

i haven’t heard about this game in a long time, when i saw this, it reminded me about how i got around 1390 while playing this in a daycare

[@CodeNoodles]

That's impressive!

[@-ShoeCat]

@@CodeNoodles thanks

[@ElementEvilTeam]

would've been useful 14 years ago

[@ShockedTaiLung]

Aw yeah it’s noodling time

[@TVDemonstro]

Loved the part where he said "noodling time" and noodled all over the place

[@GreatDynamics]

I also loved the part where he said "noodling time" and noodled all over the place

[@gummybread]

8:37 8:59 *RUUUUULES OF NAAAATURE*

[@lingeprune2373]

yeah it's swinging at the speed of raiden crackhead mode

[@SuadoCowboy]

why don't you just instead check if it's a bomb or not? as far as i know the bomb is the most different from them and by only checking the bomb you could optimize this code alot and then you only pass the mouse on things that are on movement except the recognized(s) bomb(s) maybe you could even try using grayscale images or something idk

[@melody-v-]

IF ONLY

[@advance64bro]

Dude, do you not know how hard that is

[@SuadoCowboy]

@@advance64bro do you?

[@advance64bro]

@@SuadoCowboy yes

[@SuadoCowboy]

@@advance64bro and don't you think it's worse checking each type of fruit instead of just checking if it's a bomb or not?

[@ODISeth]

Oh wow you've been at this one for quite some time, excited to see how it turned out!

[@trystankitty5393]

Hi

[@ODISeth]

@@trystankitty5393 hi?

[@trystankitty5393]

@@ODISeth hi?

[@ODISeth]

@@trystankitty5393 you replied to me lol

[@trystankitty5393]

@@ODISeth nope

[@affegpus4195]

The color recognition is a clever idea. The fact it is fast it can be used in combination with other methods for increasing accuracy

[@wChris_]

I thought you would use image recognition with a neural net for object detection, but maybe that could be v2 if you ever want to do that again.

[@JamesTDG]

I feel like this system could be expanded more if used right. Like for example, if the system detects high red pixel counts in a region, have it take a screenshot and use image recognition to see if it can detect a red arc at around 75% completion, use an or function with it so it can also check the image for an X. If either result comes true, it will determine the region to be dangerous for the next x-amount of frames. Should help improve the system by a bit, and even can use fewer resources if it is able to know the danger region before doing the bomb check.

[@vitatreat9037]

I feel honoured to get this video on my feed. This video made me excited for sure 👍

[@CodeNoodles]

Thanks, I really appreciate it!

[@CarlosRoxo]

You can try using something like a yolo network for the fruit recognition. Most recent models run very fast with a medium capacity gpu and are pretty accurate

[@ofridaniel2127]

Exactly what i thought

[@juliusblomgreen]

I'm trying to do this right now. My problem is the slicing function. I can't seem to make the mouse fast enough. I'd like to see how he did that

[@skmiraj1549]

I don't know what is going on but I love it

[@Mossy_Dahlia]

I’m not too savvy on machine learning, but would it be possible to extract the UV maps from the fruit and use that to differentiate them from the bombs? The UV map has their whole texture so would it be possible to use that to understand and recognize a fruit at all angles?

[@Jayrenzyx]

I gotta admit that segue was clean

[@KirbyCoder]

Now I wonder what it’d be like to train an AI to play this game

[@Bodoczky]

That Temple of Nadia soundtrack hits hard ❤

[@mateuszpragnacy8327]

You can lower the screenshot resolution so network can have whole screen as input and train it if game is laggy while training just make game run slower

[@Mehbem]

Bro being a student working on segmenting quantum dots with dog noise this is so relatable 😭

[@fadlanal-amsi9839]

Its magical to see a fresh half brick games content these days

[@never_dre]

The music 😮 it took me a minute to understand that something was sooo familiar here!

[@linkmastr]

2:50 nice explanation of kernels in an image recognition model

[@midtech168]

If I had this back in 2011 I would have seen so popular in school lol

[@yanglightheart]

Now that's an actual fruit ninja, my personal best is 694 though... in the arcade mode.

[@krischarles4326]

How is your IDE not SCREAMING at you…? PyCharm would be kicking in my door at the first lack of whitespace 😅

[@mrcooltree3566]

Why not have the ai focus on the background color and bomb color and when thag color is over the screen it does not attack there? Sure you couldn’t get combos but I think theoretically it would work easier faster and longer since instead of picking out a fruit it is just picking out a difference in what is normally there

[@nasirgrant4271]

Never thought I would see TAS for Fruit Ninja

[@I_am_Itay]

I think it would be fun to optimize it, here is sime idea that would maybe improve it- Waiting with slicing and calculating where each object is not just frame by frame so it would include the speed, direction, where the object would get in the next frames and I think it would give better accuracy + would give the compatibility to plan combos with smooth slices instead of just spamming

[@advance64bro]

It does what is efficient, don’t be that harsh just because you’re dissatisfied

[@I_am_Itay]

@@advance64bro I am not harsh, I really like it and the video and I just want to add suggestions so maybe he would do another video and improve it or to inspire someone else who interested in this for example I started building my own version to this bot

[@advance64bro]

@@I_am_Itay doing something like that would have to make the fruit identifying program to be more complex which you already saw was really hard to do

[@I_am_Itay]

@@advance64bro Of course it would be a bit more complex but it's well within the capabilities of code noodles, he did much harder things. The preprocessing he did would really help, it would be more work on how the bot play than on the identification

[@real1cytv]

5:23 this code is my absolute nightmare... Why didn't you put newlines inbetween the functions but put them around the while loop?

[@harkiratsingh3967]

It made me remember a scene from tge dark forest where the droplet is destryoing the the space fleet , its drscribed as God 's scribbling as the droplet is ramming objects taking sharp turns which are zero in accordance to our aerodynamics

[@racingfarmer053]

Would it be possible to code it so if it detects any black on am object at all to steer clear of it. It not because of other dark spots on the background and such

[@gdclemo]

Very good. Next project: build a robot arm with a samurai sword that chops up fruit you throw at it (but not bombs)

[@equalent]

your naming convention in Python should be illegal

[@kelvinluk9121]

Would it be easier and more accurate to fine tune an image segmentation model for this task?

[@ramadhanisme7]

The chopping is unreal 🤣🤣

[@lukraniom8731]

You could also just hold your finger in one spot and if a fruit passes by it it splits

[@Ramzi65ok]

Combos would be so cool

[@jordancrouse714]

Yay! Excited to see you post 🥰 only place I'm interested in code 😂

[@notchpoodles5864]

0:45 I️ remember this from CodeBullet (an angry Australian man who also programs robots to play games for him)

[@zeroinfinius]

You could have created a program to identify colours other than brown and black hence it could identify a new colour- fruit

[@L_tlu]

1:44 me: just use machine learning

[@fishygaming2554]

One of the best games.

[@SleepyV3rt]

Nice program. Had you considered using the indexed version of screenshots, might improve the accuracy. Also maybe look into the YOLO model, it's open source and pretty good with image recognition.

[@angelfire9622]

if you have a good enough pc you could have used YOLO for object recognition and have a better accuracy more easly

[@bachoulekikoune]

yes, small yolo models can even run on crappy pcs pretty well

[@jasonorjoshlee7607]

The bot even got a 5 fruit combo😮

[@paradoxcorporated2906]

Bro is code bullet 2.0

[@GimmeMyHandleBack]

Could you edit the color mixing program to favor black more heavily than other colors? That should help with the bomb identification

[@ExploringNew1]

How did you get a video of what the program sees?

[@CodeNoodles]

I wrote some code to create a video file where I could draw objects on top of the image to display information.

[@ranarehanqaisar2266]

Bro is going to be a genius one day.

[@shivanshmathur]

Wait, is that bgm from the old Pokemon game?? Feels like remnants from a core memory

[@user-js4xl7pw7l]

You can change the background, your blade color and splashes on the dojo in Srttings, as well as try to instantly revoke any area containing black pixels or i dunno, just (if any of this place in a screenshot has black bad No black Good, slice)

[@AuraAcorn]

Final program wasn't a fruit ninja, it was a fruit samurai

[@noahnash7265]

Would there be a way to remove the fruit juice in the background to make it perform better?

[@cinnamonsugarcourtney6073]

Oh hey, I was wondering why you didn't just use color averages at the beginning, looks like you figured it out though! :3

[@feffy380]

I'm thinking the color sampling code could have been massively simplified by just scaling the image down, which is a highly optimized operation in most image processing libraries

[@elicooper5233]

we need code bullet to do this

[@pierpaolopinna6301]

considering an impunt based on 2 succesful screenshot (instead of 1) would make the program too slow? good job thought, nice project

[@memeboi6913]

Finally, the first Fruit Ninja TAHS (Tool Assisted High Score)

[@godofall332]

Can we use Ultralytics YOLOv8 for this? I understand the processing time would increase, and it would need some program optimizations (what you did at 6:11), but it might be able to avoid bombs completely, making it better at play. Creating a dataset for this is easier than normal; let me know your take on this. By the way a great video, it helped knowing how you approached this problem from a problem-solving point of view 😊.

[@konothomasda7430]

Maybe implementing sound extraction might help?

[@cosmilitebar6772]

For screenshots i tent to use mss library, its much faster than pyautoguis screenshot method. Maybe you could do a v2 with machine learning, yolo 5 or really any version of it would probably perform much more accurate. And also extremly fast as that stuff runs on the gpu. Nice Video, its cool to see games being automated.

[@cruncyart]

bro update your profile picture that's an extremely old cosmilite sprite

[@rbbr08]

Bro, just use deep learning. If you want it to be fast you can use some mobilenet network along with some rpn detection heads... Faster R-CNN or something like that. Like a 2 stage network.

[@user-px2we6di1w]

using A* pathfinding can be good choice for finding the right path for the mouse

[@charlierogers98]

Make a bot to destroy Candy Crush and win every level without doing any single micro-transaction.

[@lonelyPorterCH]

Has been a long while since I played this^^

[@robertstevensii4018]

7:55 watch the left side of the screen and imagine a very angry Beatrix Kiddo

[@collincatmull1369]

Code noodling time :) I always have loved your videos.

[@CodeNoodles]

Thanks!

[@collincatmull1369]

@@CodeNoodles Wow, Ive never had a youtuber near this popular even get close to talking to me. Thanks :)

[@piercegames4577]

even in the image that had a blur effect I could tell you were using Visual Studio and Python

[@Happy_Koopa_boi]

You should do an image recognition thing but for suika game

[@silver2797]

your fruit killing skills are remarkable

[@niji4894]

The most real part in this video. 5:20 the naming

[@lolnt6103]

3:47 is that a gd reference

[@king_ian_yt]

its coincidence

[@lolnt6103]

@@king_ian_ytwoah really, i could never have guessed it was a coincidence!!!!111!!!!11!

[@user-fh3wg8oj6q]

Indeed, there is no need in ML. Pixelperfect analysis is enough

[@strawsEXE]

Can you make a video about using image recognition to get the best possible scores on the human benchmark website?

[@CodeNoodles]

That's a good idea!

[@kmyau_2583]

2:44 The background music sounds familiar 🤫

[@wewuwewu]

fruit ninja says "G太ME OVER"

[@makxs4818]

what is the font You use on 0:16

[@ontoya1]

You should tag the dude that created Fruit Ninja, he's on KZbin!

[@some_guy1990]

Toriel got stuck in the dummy room

[@navoddakshina767]

Can you bring a video of how to make an auto clicker for the Bloom game(air drop)

[@usptact]

You can try to use computer vision algorithms for object detection to detect and track objects. Look into detectors like SIFT for example. Warning: it’s a DEEP rabbit hole.

[@inciampare9083]

8:27 funny green splatter

[@ghb323]

so the secret difficulty mode is AUTOMATING the gameplay

[@ExplosiveGames]

Nice, super cool bro

[@CodeNoodles]

Thanks!

[@00_ontop]

The animatronics do get a little quirky at night…

[@Bostumo]

PLEASE I'M BEGGING YOU I NEED TO KNOW WHAT 2:45 MUSIC IS I REMEMBER IT FROM MY CHILDHOOD BUT CAN'T PUT MY FINGER ON IT, THE MUSIC SECTION DOESNT SAY ANYTHING!

[@ThegreatLynix]

Now,Do it with snake eating fruit game 😈

[@danveverita3077]

The edges of the bomb is always red, you could yes that as an identifier

[@salvadormilanesbraniff3511]

That second song and the fact I recognize where it from 💀

[@SmokeyyYT]

it sounds so familiar but cant remember from where

[@e_gorrr]

No one will say that the program chops pomegranates like in anime?