AI Destroys a Mobile Game in Real-Time with OpenCV (Object Detection)

Let's create AI that beats a simple game of ZigZag. Can't get that Ultra Rare Trophy? During this video, I show you how to create a powerful AI that can top high scores in the game ZigZag. We will use OpenCV to empower our bot with AI vision and make decisions based off the game inputs!
github.com/ClarityCoders/ZigZag
Want to chat with me and other programmers join our discord!
/ discord

[@blazeinhotwings]

The amount of people confused by the definition of ai in the comments and “calling him out” really makes me feel bad for the video creator

[@maxiimized]

It's annoying when people don't read up on something at least a little before making stupid comments

[@ClarityCoders]

I really appreciate this comment. Honestly though if it gets more people into it I can take a little heat. No brain no pain haha... Thanks for watching and commenting.

[@ClarityCoders]

Thank you!

[@FireDotExe]

I actually thought this was just a well made program too, i knew that an Ai is something that needs training such as machine learning and involving generations, so i went to the comment to find an explanation since i'm just a begginner in the programming world, so could you tell me what makes this an Ai?

[@ClarityCoders]

@@FireDotExe Of course. Although note I"m not claiming to be a leading expert in the field so take this with a grain of salt. AI is simply the simulation of human intelligence in machines. So in this program... playing the game using simulated mouse inputs! We did this using computer vision and rules. Machine learning on the other hand you are using more of a black-box approach. So you're not telling the computer how to do the job it's learning how to do it by itself.

[@Noobificado]

10:57 69421 gems We were this close to perfection 👌

[@ClarityCoders]

I have never been more disappointed with myself. I did however have to move on and code again.

[@gsas3012]

😞😞😞😞

[@cowch3916]

Top 10 saddest anime moments 😢

[@amogus7]

train the ai to get 69420 points and pause in all games

[@ultralaggerREV1]

Him: “turn off internet to prevent ads!” The fu$&ing game a few seconds later: **play this tank game**

[@ClarityCoders]

Changed my mind don't turn off internet keep watching my videos. Thanks for view and comment 😉.

[@hintofanimage1004]

Yea it like stores ads for a bit

[@Squishy_yhsiuqS]

Apps like this download 2-3 videos to play if your Internet is off

[@gorb9944]

Also some say that you need to the internet in order to play but its only for the ads

[@thedevildarksoul9727]

Guy you may be, but them cheeks gonna feel what it means to get balls to the walls.

[@impurpl2]

no way. i've been playing this game for years and have wondered how an ai would do in it. thanks for the great vid!

[@ClarityCoders]

Great minds. Thanks for watching and commenting.

[@YoungPutter]

Why have you played zigzag for years?

[@lukeyxo]

yes i too always play games and wonder how ai would do in it !!!!!!!!!!!!!!!!!!!!!!!!

[@gabrielandraderosa8622]

Thanks youtube recomendations, underated channel

[@ClarityCoders]

Honestly comment means a lot thanks so much.

[@TurtleMaster326]

This is an underrated channel, it’s nice to see small channels that also produce quality content

[@ClarityCoders]

Thanks that really means a lot to me! Thank you for watching and commenting.

[@Pitiflu909]

11:02 it recognizes the "C" on score as a circle but not the "O" xD

[@ClarityCoders]

Darn AI.

[@jagaya3662]

The first R in "RETRY" also had a circle, the second one however only a line. I think you'd include a minimum distance two objects must have to be considered different objects. You can also see early on how it didn't detect an edge up to the point where it touched the ball but actually had a gap there.

[@chris_aka_CP]

Thanks KZbin recom, i really liked the vid and the fact that i see so many answered comments, Shows that you care for community. Subbed

[@ClarityCoders]

I honestly really appreciate every view and sub. Thanks for commenting and subscribing.

[@allak1n]

Subbed - I'm a dev (work on boring business stuff) but always been curious about vision AI. I think your videos are well explained. Keep it up, I want to see you grow, you deserve it.

[@ClarityCoders]

Comment really means a lot to me. Thanks for taking the time to watch and leave me a message.

[@noobmasterr69]

Great video..for someone getting started with OpenCV this was very entertaining to watch. Subbed!

[@ClarityCoders]

Thank my man! Really appreciate the sub.

[@JamesTDG]

It's a pretty good concept, would be interesting to see more visual-based AIs...

[@ClarityCoders]

I'm on it. Thanks for watching and commenting.

[@BlobsoR0ck]

Disappointing that i got recommended this video after 6 months, very interesting to see this AI. Although im not a programmer I've always been interested in programming, modeling, and coding a game. Very cool video.

[@ClarityCoders]

Thanks! I really appreciate the comment and view.

[@warmpianist]

Still wondering, the slope of the lines at the boundaries should be just around 0.5 and -0.5 right? I think you can use edge detection and filter all lines everything that has different slope than that.

[@ClarityCoders]

This is a VERY smart comment. A few people have mentioned the idea and it crossed my mind very late in making this video. I decided I didn't want to invest the time to try it but I think it would work.

[@kbabe3915]

I think that on top of this you could detect the intersection point between the left and right lines (if you just made the lines go off to infinity before drawing), and with a static distance from that intersection point determine the ending point of the line. This would in theory fix the opencv "interference" behavior and draw more consistent lines. It'd also make the damn thing perform perfectly

[@feha92]

Yeah, I was surprised when he used a hough-transform, as something similar to Sobel with different kernels (or straight up, what was its name, canny?) should have sufficed quite well (and not resulted in infinite lines). Would be edges and not lines, but it seemed like he used the lines for edge-detection anyway (just painted "hazards" [with binary directions] for the algorithm to trigger direction changes with).

[@godman1117]

Didnnt even know about the open cv module, might have start learning that, looks pretty useful

[@gubiithefish]

This is amazing man;-) Subbed!

[@ClarityCoders]

Thanks for sub Emil means a lot.

[@Gaellka]

Congrats man. the god of KZbin algorithm has chosen you.

[@ClarityCoders]

Thank you KZbin overlords. Thanks for watching and commenting my man!

[@Gaellka]

@@ClarityCoders no problem ^^

[@Caterblock]

Listening to Rick rn

[@Pheonix1328]

I play a game called "Cat Jump" and I wonder if this is how people get those insanely high scores... I'm still in the top 1% with my own skills so I'm not too mad if they are cheating xP

[@ClarityCoders]

I do feel bad for those people part of the reason why I picked stupid not so popular games.

[@unpaidintern6652]

@@ClarityCoders I'd say the people who are able to bot those games applied their own skill to the game.

[@matthewstewart5113]

I Do not understand it, but I can see the future value in it. Keep going. Great video-Thank you

[@ClarityCoders]

Awesome I will! Thanks for watching and commenting.

[@tenzinsherap4049]

Really thankful for the youtube reccomendation for showing me this video

[@ClarityCoders]

hey thanks for watching I appreciate it.

[@ozzy1987mr]

muy buen video explicando detalles importantes... lastimosamente mi ingles es malo pero el traductor algo ayuda... tengo poco tiempo incursionando en este mundo de la IA y tus videos ayudan mucho

[@lunaroja369gimenez6]

Hola, hablas español? Me dirías si pudiste hacer funcionar este proyecto? Tengo que hacerlo para un juego casi igual

[@jkl9984]

That's a very nice video. Was interesting to watch.

[@ClarityCoders]

Thanks JKL that really means a lot. I appreciate you taking the time to comment.

[@adam4478]

it might be interessting to only keep the lines with a specific angle since each wall are parallel

[@ClarityCoders]

Honestly this crossed my mind very late in this project. I think it's a good idea! Thanks for watching / commenting.

[@Kenshinse312]

Probably the easiest way is to simply remove all vertical lines, all lines that are connected to the underside of the vertical lines. After cleaning the lines, you can also remove the gaps in the edges, I think this will also help. As the next steps, you can also check that there is a full path from the current point to the end of the screen. And clean everything between the two edges ( artifacts from gems for example ).

[@annesophie4037]

I'm not really good with the terms or really anything mentioned in the video but it's really interesting to see what an AI can do.

Thank you for sharing, I like it. :D

[@ClarityCoders]

I'm glad you like it. Should be plenty more coming soon.

[@alohaboomboom8913]

This is what i wanted to learn thanks

[@kema2240]

This is amazing! One quick question: Can you please share the position & size (in pixels) of your ldplayer? I'd like to adjust my ldplayer accordingly so that I don't have to tune parameters such as those for sct.grab(). Thanks!

[@ClarityCoders]

Dang I wish I had thought of that before posting it was on my old laptop. Next time I will post that as well.

[@rasmustagu]

Absolutely awesome, coding is something way over my head and I love seeing stuff like this!

[@ClarityCoders]

Thanks! Never to late to get involved start small and work your way up!

[@fabianbleile9467]

nice video - well done! :) i am new to the topic and wondered if it would be sufficient to check for the direction of the lines plus a min length to get the relevant lines. this should eliminate the diamond lines instead of masking them out. what do u think?

[@ClarityCoders]

That's a really good idea someone else mentioned it as well. Towards the end of the project I thought about finding all lines and only keep ones at the angle of the track. It was late and I was over it at that point so I never tried.

[@RonaldoSF]

This app is so useful to find these values

[@ClarityCoders]

I love playing around with it. Thanks for the comment.

[@tazgirl_]

Seeing people who don’t understand programming argue with you over what an AI is feels bad man, feels like asking any question on stack overflow. Nice video

[@ClarityCoders]

Thanks! That's a spot on analogy haha.... Overall though I appreciate the comments and views even the ones complaining about my haircut 😂😂.

[@ZombieChicken-X]

This looks like a fun project Imma try it

[@ldoubleprime1654]

As a young dood i am rly interested in coding and tech im glad yt recommended me this yt channel ^w^

[@ClarityCoders]

Thanks dood. Keep coding!

[@volttheprotogen1893]

uwu

[@ldoubleprime1654]

@@volttheprotogen1893 uwu

[@incine8908]

Hey man, love ur vids. I was wondering if it would be possible to create an AI that can actually read captcha codes (dw its not for anything bad, just a curious mind). Ive seen a couple bots do it and wondered if its possible so went to try and see what they used but didnt really get that far. So could u make a vid showing us how to beat a captcha game with an AI that can read distorted letters from an image? much thanks man, great work on ur vids.

[@unpaidintern6652]

Probably boat loads of tweaking and huge training sets. But it is definetly possible and the programs became better at doing these text based captchas than humans so we switched to Googles weird meta data and image recogintion ones. And im 100% convinded that Google uses those "select all X" pictues as a pretense to build image recognition sets. They can do it just fine without us having to cllick anything

[@SamiTheAnxiousBean]

this is really Interasting Great job

[@ClarityCoders]

Thanks that means a lot. I got more cool stuff coming out soon!

[@SamiTheAnxiousBean]

@@ClarityCoders im exited for that

[@djentslayer]

4:00 Haha, you said "balls". All dumb humor aside, this is a really interesting video! I hope this channel gets bigger, you definitely don't have the number of subscribers you deserve, my dude :)

[@ClarityCoders]

Thanks KZbin has been fun for me doing stupid projects. I'll stick around if a few people keep watching! Thanks for comment/view.

[@vinusanker8767]

Discord is fun to hang out and ask doubts people

[@ClarityCoders]

Thanks for watching and commenting!

[@vinusanker8767]

@@ClarityCoders sure, y not?

[@basilicon.]

Can you make an AI that plays Geometry Dash and learns how to get farther after each death? I think it might be really interesting to see the computer's solution to each level.

[@2-_]

something like sethbling's marIO would work for that i think

[@ClarityCoders]

Yup probably need to go deep learning but it could be done. I'll put it on my list.

[@bmanb2099]

Instead of wait maybe a month to see it happen why do you try making it in about a year! Much faster

[@JohnMazz]

I've always wanted to automate some mobile games, cool vid thanks!

[@ClarityCoders]

It's a fun challenge for sure. Thanks for watching and commenting.

[@JohnMazz]

@@ClarityCoders I always comment on videos I want to see more of. Got to train that algorithm!

[@masondaniels8675]

Nice video 👍 Neat project

[@ClarityCoders]

Thanks! I appreciate the view and comment.

[@cosmilitebar6772]

You could try to make an piano tiles ai. That is really simple but still funny to look at. I managed to get around 30 tiles/s with mine.

[@AntiDroidZ]

Great video 😎

[@ClarityCoders]

Thanks Zack means a lot.

[@Bubs.]

my record on there is like 2000- i used to always play that game while listening to audiobooks. probably have logged a ton of hours

[@deiviselenis9106]

Great video!

[@ClarityCoders]

Thank you for watching I really appreciate it.

[@daddyofalltrades]

Sweet 🔥

[@ClarityCoders]

Thanks!

[@tiger-mt4ju]

I'm very glad that you had enough thought to buy all of the balls. Thank you.

[@ClarityCoders]

I'm all about the balls.

[@1996Pinocchio]

Using opencv to find the ball is nice, but you could also search for black pixels on a set height (presumably on half height). That would be way faster I suppose

[@tttam96]

It's just a thought I'm curious of why you didn't say to avoid the white and stay on the colored path would it be a lot harder to tell it to do that instead of doing it what you just did. For instance as I'm trying to say is you program it to avoid the white void equals death type of thing so it would be focused on staying but you did it in a reverse way

[@akerman_]

Cool vid dude

[@ClarityCoders]

Thank you!

[@ductr2hn]

this bring back my memory about dancing line

[@explosivemonkey4596]

That game’s still going I think, god it had such bangers, the design of each stage only got better as you went along! That Easter egg in the maze was heartwarming too

[@ductr2hn]

@@explosivemonkey4596 ok but still can't play it because i using android

[@explosivemonkey4596]

@@ductr2hn I pray for your loss.

[@TilDrill]

@@explosivemonkey4596 they flooded it woth ads at some point it got unplayable for me

[@linus855]

this is very a very amazing video

[@MsTatakai]

i don't know if it your code is using GPU or not but if you change the saturation on gpu drivers wouldnt affect the code? if i ramp down the saturation down the screen turns black and white... wouldn't that help instead of detecting edges and drew lines instead? black, white and gray is easier to program... maybe not i don't know ...

[@vinusanker8767]

Hey, jake, it's me vinu from discord, hope u r well

[@Lebensgott]

cant you improve it with some prediction of the lines? i mean they are always the same angle and lenght (the longer are the one lenght times two or times three)

[@Resursator]

Video: turn off the internet to prevent ads!! Me: okay Video: stops

[@ClarityCoders]

Whew thought I lost you there glad you got it back! haha

[@Mlafster]

Hi, I'm a bit late to the party but I believe there is a version of this game where going left/right is dictated (followed) by changes in music, do you think making an AI/script that plays the game (or any rhythm based game) in a similar fashion is possible? And do you think it would be harder or easier? Audio is easier to process than video for sure but it has its challenges. Just a thought :P

[@ClarityCoders]

I'd love to try that game! haha we could use the sound as input right?

[@Mlafster]

@@ClarityCoders Yeah of course! Search "Dancing Line" on Google play. This got me thinking about the connection between the visual representation of music and the AI's ability to "see" things.

[@Cod3rMax]

I did implement the same concept on c# i detected the ball and the edges but how can i detect if the ball is close to the edge?

[@458stanley]

Loved the video, although I am a bit curious if this could lead to any legal issues. You can probably pay for the diamonds, so is this like a cheat to get around a pay wall that could lead to a lawsuit?

[@ClarityCoders]

You might break terms of service and get banned but nothing about it is illegal. Note I'm not a lawyer and this is not legal advice. 😉

[@paulp1204]

That's really impressive. I am not sure this is actually a form of AI though, more a well automated script, essentially.

[@ClarityCoders]

By definition it's artifical intelligence though it's not machine learning. I would agree with your point though it's not learning anything.

[@somdudewillson]

AI does not require any form of learning.

[@SurrealBackrooms]

1. AI 2. Machine Learning (a type of AI) 3. Deep Learning (a type of machine learning and AI)

[@Maric18]

anything that makes decisions to solve a problem is intelligence, and if it is created by humans we call it artificial your phone has several ai systems, even route planning is AI the edges of the definition of ai is fuzzy, but for example the game itself is not ai, although if there are enemies that try to hurt you, or even if the track generator uses tiles you die on more often later in the level and less often earlier, that is ai

[@Laszer271]

@@Maric18 as you said the term AI is fuzzy and what we perceive as AI also changes with time. It's similar to technology. Some time ago a pair of scissors could be considered a technology but today we have a much different idea of what technology is.

[@WeatherStone]

hey, i made a game called Ready, Set, Roll, for android, its also about a ball, but you need to collect coins and crystals to finish a level, can you beat it with AI?

[@honeynutios]

Ooo can u do an idle fishing game where you drag ur cursor around the screen to catch fish? Id love to see an ai just immediately catch everytging as soon as it pops up

[@ClarityCoders]

Probably what's the game called?

[@legend0.]

this video was really cool

[@ClarityCoders]

Mean a lot! Thanks for commenting Legend!

[@GageMason]

There was a game where you jump a bird side to side avoiding spikes that I used to love. I'd love to see it done by ai.

[@tmnnn]

flappy bird?

[@Skellitor301_VA]

what's the song at the part that's showing off the bot getting over 1k?

[@toprakdikici9459]

cool project

[@ClarityCoders]

Thanks! Really glad you enjoyed it. Lots more coming soon.

[@mrwait2425]

hi can you make a tutorial that an object trace a line that curves

[@gauravrewaliya3269]

please provide a detailed tutorial... you have a lots of knowledge... 🔥🔥🔥🔥🌟🌟🌟🌟🌟

[@ClarityCoders]

Thanks! I've been wanting to do a line by line one for a project like this I'll put it in the works.

[@lincolnh2348]

Flappy bird would be a cool one to automate. I don’t think it’s on the App Store anymore though

[@deezerload]

I'm sure your AI had so much fun

[@ImCxoi]

*visual confusion*

[@ClarityCoders]

It did! It slept so well that night all tired from it's big day.

[@Rex_Vulcan]

What is the name of the music in the background at the beginning?

[@cg7407]

i think the game called "Rolling Sky" (which is a game commonly played in schools) would be interesting although i dont think it would be very easy since almost every level has different textures especially the event levels and there are many objects that can interact with each other. (You could probably brute force it since the levels are always the same but that would be boring)

[@angelben24]

You could try and show us 😉

[@ClarityCoders]

I'll check it out and put it on my list!

[@Bencehh]

Does anyone know what is the music name which is playing in 10:06 at this video? I really need the name of this song.

[@ClarityCoders]

Fast and Run - Nico Staf. Thanks for watching and commenting!

[@NECarp]

Could you try Dunk Shot by Ketchapp really fun game that uses trajectory

[@koko_noob8078]

Make a video on how to capture the game screen. please

[@RandomBubble]

My personal score was 1700 in June 2021. I played it after a few years and instantly got that score. I guess I can’t be better than an AI being lol

[@ClarityCoders]

You're certainly better than me.

[@agentx7138]

I play the game Scrap II a merging game with multiple upgrade menus that I have wondered if it would be possible to automate.

[@XaviIntheHouse]

the answer is yes

[@PastyMancer]

Scrap II is pogging so hard. 10/10 besties!

[@juulleke]

holy f this thing is good

[@rikisu.]

This video was great by itself but what I'm appalled by is that you still reply to comments... The amount of patience and dedication! Damn

[@ClarityCoders]

I'm amazed people watch my videos let alone comment on them. I truly appreciate every one of them even the ones making fun of my unibrow😂. Thanks for the comment and the view!

[@Crashthatch]

How long did it take you to build/test this? Hours? Weeks?

[@ClarityCoders]

Probably 8 hours spread over a few weeks? Basically it was really simple to create the rules and automate it to play repeatedly but dialing in parameters is tough. So I would change some things then let it run for 100 games then change some things.

[@Diego-Garcia]

@@ClarityCoders The next step: use a genetic algorithm to look at the best parameters hahahahah

[@carldressler3262]

@@Diego-Garcia Then we would enter machine learning territory!

[@paulyguitary7651]

I like how Open Source rhymes with Free Course

[@gicudoi2771]

What are the limitations of ai other than processing speed if you optimise the ai to play a 2d fighting game could the ai do it if it had the processing speed

[@entegi]

An AI that is written in C could do it, but Python is too slow to process that much data in rt

[@ClarityCoders]

Great question. Even with processing full image data in some games I was able to achieve better than human reaction times. Although as someone mentioned there are faster language choices out there if speed is an issue.

[@kasiorap]

let me grab my "here before this channel gets 1m subs" chair

[@ClarityCoders]

Hope you're right 😉. If not glad people are enjoying it! Thanks for watching and commenting.

[@Brandonzv]

What program are you using that is showing each step of the line detection live?

[@ClarityCoders]

OpenCV

[@Brandonzv]

@@ClarityCoders Thanks! I was thinking it OpenCV was only a library or something.

[@silvix3137]

or you could just do the big where you press on play again and the edge of the screen and you can literally go outside the map and just spam for points

[@enderinvader]

I'm trying this right now with a game, but I can't seem to get the mouse to click even though it moves to the correct location

[@ClarityCoders]

Can you click on anything? Try something other than the game maybe.

[@enderinvader]

@@ClarityCoders I actually figured out there is a setting in the emulator that wasn’t letting pyautogui click

[@ayushxsharma]

Pls tell how can I run this code and play it

[@zaclanderos1624]

If you want a bigger challenge, you can try Cosmo Run ;)

[@zaclanderos1624]

Amaizing job btw

[@ClarityCoders]

I'll put it on my list of games! Thanks for watching / commenting.

[@AyAylinJJ]

Very cool

[@ClarityCoders]

Thank you!

[@lineage254]

Awesome video and about the comments debate, this is an AI, a basic one ,but an AI nonetheless Machine Learning or Deep Learning are just "smarter" types of AI, you can think of them as ppl with higher IQ that can learn complicated task or things on their own with out guidance, so you dont have to "teach" (Code) the AI what to do, they are smart enough to learn on their own.

[@devangsharma336]

Hello sir I try this code to run in my computer with same emulator and game but it is showing following error: Traceback (most recent call last): File "D:\photo\ZigZag-master\HighScore.py", line 147, in x = x + 1 NameError: name 'x' is not defined

[@ClarityCoders]

If it doesn't find an x value for the circle this area will pop up. Basically it didn't find the ball on the screen. Make sure your screen grab is covering your emulator area it won't be the same as mine. scr = sct.grab({ 'left': 0, 'top': 390, 'width': 440, 'height': 50 })

[@rasmussiira9151]

What is the song in the start

[@YukiAttano]

Imagine the old days, where you simply search with a thrid party software for the score value, increase it to something incredible high and die after the first sec. But you got all the ingame money, the highest possible score and be done with the game. in 5 min.

[@ClarityCoders]

Those were the days. Cheat engine baby.

[@res.1123]

how to train in wave mode on geometry dash

[@Roxst4r17]

Would it simplify things if you searched for hue lines in a specified angle using a simple slope formula to store the initial slope values of the sides of the track? The angle of the path doesn’t seem to change. Furthermore could you specify that it only look for hue lines in the specified angle previously stored. You could further perfect it by storing the width of the hue line just as the angles change on the track. This point would be (what I assume) is the width of the track. Using that value divided by 2 to set your pixel distance out from hue lines and search for hue lines a set distance out past that. I’m not really a programmer and don’t have knowledge of this program specifically just asking based off what I assume is possible. I think this could reduce computational load as well.

[@ClarityCoders]

Yeah I've had this comment a few times and you guys are correct. I think that would be the easiest way to improve the project.

[@Roxst4r17]

@@ClarityCoders honestly though thank you for responding I had been wondering. Great videos too! Edit: now subscribed

[@faddy_is_ur_daddy]

1:41 bold of u to assume that

[@killertigergaming6762]

Seems cook

[@ClarityCoders]

Cooking the competition.

[@mrfoucs7574]

whats the soundtrack at the end of video?

[@ClarityCoders]

Fast and Run - Nico Staf

[@ayushxsharma]

Bro how do I run this code

[@athanasiosp.]

I remember that game. I played it at my free time when I did not have enough time for something bigger like on the pc. I actually reached the 1000 point mark and played until I got all balls and stopped haha.

[@ClarityCoders]

You are a stud. I literally never improved I must not have the attention span or reflexes in my old age.

[@athanasiosp.]

@@ClarityCoders Its super repetitive in a bad way, like wait until the turn and just tap once or you lose and when multiple turns appear you can easily lose track of how many times you have to tap. I just like everything that can be challenging and try it until I win it or at least know how to play it well. The code was more interesting as always compared to me just mastering this game hahaha.

[@burak1382]

Try the not much known game named Shapez (No Shapez io, only shapez)

[@mqhasrul5326]

How to use this on Android