Using Image Recognition to Automate More Mario Minigames

[@CodeNoodles]

To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/CodeNoodles/ . The first 200 of you will get 20% off Brilliant’s annual premium subscription.

[@bowzert]

Hi! I did the moles minigame in C# a few months ago! The video on my channel : kzbin.infowjhrn0oKg8g

[@lolathepinky4857]

Woohoo I'm one of the first 200 ! :D Love your videos Noodles !

[@Kaishidow]

I won't even be mad if you decide to milk this concept, love it so much

[@bigwinnergamer]

U ain’t Ted ed

[@Giorgodacat]

Potato cheese

[@underrated1524]

Fun fact: it's physically impossible to hit all the Montys in Whack-A-Monty, even if you're TASing. The game simply can't register touch screen inputs fast enough for you to hit them as fast as they show up near the end.

[@CodeNoodles]

That makes me feel a lot better about my script 😆

[@Bryce_the_Woomy_Boi]

He didn't automate these, he's just really _really_ good

[@CodeNoodles]

Oh no, you caught me 😆

[@cinnamoone3553]

i swear, i see you everywhere now-

[@danielgysi5729]

I'd love to see some further optimizations to these. I know image recognition gets clicks or whatever, but for whack a monty, all you really need to do is check the color values at specific pixels. If you wrote a C program that used the Win32 API, you could grab a bitmap of the ds screen, check certain pixel indices for the right color, and simulate an input event at each location. I reckon it'd be blazing fast compared to your python script with all that pyautogui and image recognition overhead.

[@leonfire99]

Yeah, i was thinking that needed more optimization. Watching the playback it really missed the bottom row a lot

[@cerulity32k]

Rust has rsautogui, and processing data in-language is way faster. Using it, I made a 7 line script that uses Tesseract as well to get 7500wpm on the Human Benchmark typing test.

[@li5up6]

Overkill. The processing almost certainly does not take longer than 30fps or whatever sm64ds runs at.

[@AlmostPGLiam]

Great video as always. I watched it at 82x speed, just to make sure I could comment early.

[@CodeNoodles]

That's a great skill to have!

[@sumynona.01]

how did u get 82x speed?

[@chairwood]

​@@sumynona.01is actually possible with entering some JavaScript into the f12 console. I used to watch videos for class on 4x speed xd

[@vk8a8]

@@sumynona.01he’s a hacker

[@bobosaurus331]

@@sumynona.01Skill issue

[@EpicLPer]

For the last minigame I'd suggest using color recognition instead of images, since the colors are not really close together. It also seems to run very slow judging by the jittery mouse movement so maybe it improves with that then :)

[@FlutterBug]

yea I was thinking all it really needs to do is check for anything bright orange, that'd include the walls and the flame breath

[@MightyMemeKing]

That's what I'm doing with my autohotkey bot that automates a mobile game I play. I'm digging through it and replacing every imagesearch command I can with pixelsearch commands that only check the color of a single pixel.

[@gooseylucy2158]

Thrilled to see a sequel to this! There’s at least 2 really close calls in Danger Bob-Omb, so this hit box must be really interesting. Thank you for making another!

[@burnfire4617]

For the last mini game with the fireball. What about this strategy: you view the fireballs as electrons (negatively charged) and the player as a proton (positively charged) such that the movement of the proton is dictated by the sum of the electric forces of the electrons.

[@CodeNoodles]

That's a really cool idea! I actually made a magnet simulator a few years ago, and I could definitely see that doing fairly good with this minigame.

[@Imaproshaman7]

The first image recognition video is what helped me discover this channel in the first place and it was great, so I'm glad to see it revisited! These are always so cool.

[@gamingplus2884]

It'd be cool if after you finish up all the mini games. for SM64 DS if you could go on to do Mario Party minigames. or maybe make a program to play through all of certain mario games

[@PWPictures67]

That score is higher than my highest score on Whack-A-Monty. My highest score is 79 Moles. I don't remember my highest score on Danger Bob-Omb Danger, I haven't played that game in ages. And I never play Snowball Slalom because I could never finish it. I think that I made it to the end once or twice though.

[@zdelrod829]

Would love to see the project continued to the point where all of the mini games in NSMB and SM64DS are automated, and would love to see how you tackle the Poker and memory games.

[@pokeli2728]

If you could automate the board section of Mario Party DS that would be really cool. Then you could combine it with the automated mini games to make an entire game of Mario Party played by an AI. That'd be really cool to see!

[@daniiii888]

It would be more like "played by a TAS bot" since AI would have to learn every aspect of the game instead of being programmed to do exactly what it needs to do.

[@theAstarrr]

This is my favorite thing on your channel It's so fun and also gives me nostalgia.

[@limbus66]

I'd love to see you take a crack at the Luigi Poker game. Working probability analysis into it would be interesting

[@TheTechn9neforlife]

absolute legend. my siblings and i played these when we were children and seeing them broken has me in awe. thank you for this!

[@luigichief2200]

The nostalgia plus the computer science is a great duo! I'd love to see more.

[@sallystrings]

Yessss this is so good!! I love your other videos on minigame automation, thank you for doing more!

[@xYamakaze]

Based on how the Bob-omb minigame was going, you have to make it play and beat Touhou next.

[@lukethompson5492]

I don't leave comments on videos ever, but this is one of my favorite series and I would love so much to see more of this!

[@CodeNoodles]

Thanks, I'm glad you enjoy the series!

[@SylasTheShadow]

I cannot stress how much I love these videos, and how badly I need more of them

[@marcosbarrios1625]

We need more of this for sure

[@beardalaxy]

that bob-omb one made me wonder.... touhou?

[@MontyMoleLoreMaster]

We are just going to ignore the fact that the top screen of Whack-a-Monty shows all of the Monty Mole corpses the player smacked to death.

[@niemandit]

You reminded me of playing these minigames during the holidays and made the current ones a little better. Thank you for the great video :)

[@Foxy-yr8wq]

I love these videos so much It's awesome to see these little mini games from my childhood again xD

[@RudyGGames]

Looks fun to code this. Looking forward to seeing more!

[@Blazico]

Thank goodness, computers still can't beat me at the final 2 minigames.

[@GarlicRoastedSalmon]

Hey, there's this game called Backpack Battles on steam which is basically an auto battler where you have to buy items that synergize best with each others and defeat your opponents in a total of 18 rounds. It's apparently NP-hard and the well known knapsack problem to make an AI. Would be a real challenge to create an AI that can defeat humans?

[@jizosgoescrazy]

love the style of the video! Amazing ideas and oh man... the music in the background, so nostalgic, takes me back to the good old DS days

[@YeetoTheYeetafic]

Alternative Title: Using Image Recognition to make the Nintendo Ninjas find me

[@SmudgeAlot]

There is a shooting range game in kirby superstar ultra for the ds and I would like to see the ai get a perfect score. Anyways this is a great video 😊

[@kennycarter5682]

I would love to see this further expanded on, aka do every minigame... i wish, ha ha

[@hithere-cz4oc]

You could try using voronoi diagrams to optimize the bowser one… you ideally want to move along the lines in the voronoi diagram, and you would want to target resting in voronoi vertices that are very large distances from any fireball. You can pathfind using the path with the largest closest distance you’ll ever be to a fireball, so you’ll be able to stay in the safest zones and take the safest path to get there. I would imagine this is very close to how a human plays the game! Except, of course, instant and never gets tired, never blinks, never gets distracted, and never loses track of fireballs

[@lemmingyellow3888]

I love this. Love the code, love the editing, love the ideas. Great work!

[@ermamu]

More minigame automation would be great!

[@maiinganbear5412]

I love my favorite mini games being cheesed with image recognition. ☺️☺️☺️

[@marckey5021]

We need more

[@qangeldratsch9643]

I am really curious, how you will handle the desert puzzle minigame.

[@MagnusBuffoonery]

awesome video! if you make another part, you should try to further optimize the snowball game! it would be awesome if it didnt bump into anything at all, even if its just lucky!

[@FridayNightStreamz]

This was nice, and hope you feel better

[@CodeNoodles]

Thanks!

[@FridayNightStreamz]

Np

[@luigiymario2xd]

Let's goooo, I was waiting for more of these lol

[@link-cq2wx]

Now do whack a luigi on the monty minigame

[@link-cq2wx]

or you do the opposite of what you're supposed to do, hilarous but probs not worth the effort doing the whole idea

[@TheFlashzap]

6:56 Carl from Aqua Teen Hunger Force looks surprisingly happy for once

[@CrimsonNova927]

Merry Christmas!

[@diamondoge]

1:00 sir that's Luigi abuse. Thats illegal!

[@RaziOrSomething]

Automate the entire mario vs luigi mode

[@chiefsofnobles]

awesome and great job, its fun watching a fellow coder solve problems, opens my mind to different ways to tackle problems i previously wont have thought of.

[@philogex]

i don't usually comment on videos, but i wanted to say, that while the video itself was interesting, your approaches were understandable, but not very optimized. you did mention it at the end, but i still think you could have improved your designs by a long shot

[@CodeNoodles]

That's completely fair. I'll try to optimize the scripts more in the future.

[@ianjohnson218]

I am very excited to see you doing more of this! I’ll be excited to see Mario party get mastered too!

[@dreeg1896]

Good remix taste my man. TND is majestic.

[@aureliamastergoomba1278]

I hate the snowball one…

[@sahilhossian8212]

Lore of Using Image Recognition to Automate More Mario Minigames momentum 100

[@DarkChaosMC]

This seems fun, I’m gonna try it myseif

[@DoctorTex]

Automate that one Yoshi flower petal game!

[@escanor_974]

really cool

[@txqea9817]

still waiting for 2 (or more) ais fighting in nsmb minigames

[@Smoet]

Sick video as always man!

[@dovgertz288]

wow great job as usual

[@CodeNoodles]

Thanks, I really appreciate it!

[@TheDMan2003]

1:00 Isn’t that the pose he makes on the Luigi’s Mansion box?

[@ImExistent405]

Nice job bro

[@djmintyfreshful]

MORE!

[@ocomolinaehain1795]

Do you plan to do Mario 64? Like, the trampoline one or the piranha plant one?

[@CodeNoodles]

I've thought about it. I might take a crack at it next year.

[@emochoo]

Lessgoo new codenoodles video

[@MccTube]

Great video!

[@GraalsDeadMonster]

What is this, a early Christmas gift? 😍

[@Jelly_shy_guy_man]

I thought you were think noodles

[@user-qv5do7vz5l]

Yes i need more

[@tanter14]

loved the video, something id suggest especially with coding or anythng creative, show some of your fails(things that just go wrong or horribly wrong) it gives the video progression and incentive for the watcher to keep watching (long story short, people love when things go wrong)

[@CodeNoodles]

I definitely make plenty of mistakes 😆

[@bratkartoffeln_]

keep the good work up

[@absin8078]

Super cool man

[@nathanlamberth7631]

Last year? Stop making me feel old

[@CodeNoodles]

Yeah, I know right 😆

[@IoIxD]

Are there any minigames you CAN'T do with image recognition? It'd be interesting to see if the entire game can be completed with scripts. EDIT: Oh, one of them requires a neural network. So maybe not.

[@Useott]

Yessss new video

[@DonSMDT]

how many minigames are even left in this game??

[@NOTuetc]

I love your content man

[@Mizai]

good job 😊

[@LexiLex421]

I’m st-epp-ing a-way From, my re-grets of yes-ter-day!

[@GmNdWtchr96]

Would these count as Tool-Assisted Speedruns/Superplays?

[@ihaspancake2331]

Cool

[@ahmedskully]

Cool videos

[@tanhrs8711]

Cool a video ! Btw when Minecraft 2d update ?

[@jknifgijdfui]

i wanna see u automate warioware and i just realized halfway thru typing that that that will be near impossible and not fun

[@Olejika_CheatEngine]

As I remember I made 5000 points in Danger Bombomb. So it's not very impressing

[@YoloYester94]

They used a python? I can’t even see sharp!

[@pokepowerz4]

The method you used for the snowball minigame seems kind of... bad. I mean it only learned to navigate that one specific track, and if memory serves me right the layout of the penguins and rocks are randomized each time you play it. Also at the end it hit the wall a few times and slowed down. I feel like the best solution would be one that can handle any track given to it and doesn't hit walls.

[@pokepowerz4]

Still love the video btw great work as always just wanted to share my opinion on your approach to that minigame

[@CodeNoodles]

Yeah, the layout is randomized. I tried a couple of different methods to react in real time to the obstacles, but none of them worked correctly. Next time, I will try and avoid using this method.

[@justacutepanda888]

Just add controller button noises, and claim you’re the best player in the world.

[@suttonfreeland6368]

I got an ad at the beginning of the sponsor 😭

[@dominicmora2559]

Where is the code for bomb danger?

[@afartingguy8541]

ye

[@c7wolfyy]

LETS GOOOO

[@DoneThatSeenThat]

Trampoline

[@bupp291]

I enjoy these videos, but from what you showed, it doesn't feel like you spent a lot of time optimizing these. Most of these look like first attempts. I'm sure you put far more effort into these than what's shown in the video, but it didn't come across that way. I would much rather watch a video on one game that you optimized the crap out of than three where you were satisfied after the first few attempts. I do love your videos though! They are very enjoyable. This is just some sincere critique for you to consider and do what you want with. Ultimately, just do what makes you happy!

[@CodeNoodles]

The last 2 minigames gave me a lot of trouble, but I definitely should have spent more time trying to optimize them. Thank you for the constructive criticism!

[@lelepao]

❤

[@BoxuTrev]

bro makes tas look weak

[@Skillez_Elias]

Lessgoo

[@explodingwolfgaming8024]

Commenting 4 algorithm

[@ErinLynagh]

"This one is extremely difficult to last long in" 😏😏😏

[@busofselfdoubt]

from mario party ds? these…. aren’t from that game.

[@ado-]

he made a nsmb video, then a mpds video, and now this is the third video, coming back to nsmb. it was weird phrasing

[@matthias916]

good