Thresholding with Match Template - OpenCV Object Detection in Games #2

Detect multiple objects with OpenCV's match template function by using thresholding. In this tutorial, we dig into the details of how this works.
Full tutorial playlist: • OpenCV Object Detectio...
GitHub Repo: github.com/lea...
OpenCV documentation: docs.opencv.or...
Official template matching tutorial: docs.opencv.or...
1:18 Understanding the result from matchTemplate()
2:25 Using the debugger in VSCode
5:08 Why is the match template result smaller than the original image?
6:18 Using numpy where() to filter the results
7:27 Converting np.where() to a list of pixel positions
9:41 Adjusting the threshold
10:11 Drawing multiple rectangles
12:27 Using TM_SQDIFF_NORMED
With thresholding, we can detect multiple objects using OpenCV's matchTemplate() function. They discuss briefly how to do this in the offical tutorial, but here I will cover it more in-depth.
Previously we used minMaxLoc() to get the best matching position for our needle image, but matchTemplate() actually returns a result matrix for all the positions it searched.
Each value in this matrix represents the confidence score for how closely the needle image matches the haystack image at a given position. The index of the outer dimension represents the Y position, and the index of the inner list represent the X position. For example, the confidence value 0.58679795 in the data [see: learncodebygam...] corresponds to Y = 1 and X = 6. When we overlay the needle image on the haystack image, such that the upper left corner of the needle is placed at pixel position (6, 1), it's match score is 0.58679795.
Note that the resulting matrix size is (haystack_w − needle_w + 1) * (haystack_h − needle_h + 1). This is because there are no meaningful match results when the needle image is partially overhanging the haystack image.
The idea with thresholding is, we want to get the coordinates of all the places where the match confidence score is above some threshold number that we set.
To do that, the documentation suggests we should use the np.where() function. This will give us all the locations above that threshold.
In the result, the first array contains the Y positions, and the second contains the X positions. So in the above example we found two matches above the threshold we set, at positions (1, 1) and (15, 5).
So we have the data we need know, but the format returned by np.where() isn't very convenient to work with. So lets convert it to a list of (X, Y) tuples: list(zip(*locations[::-1]))
In this line of code, [::-1] reverses a numpy array, so that we get the X values to come before the Y. Then the star * unpacks a list, so that we now have two one-dimensional arrays instead of one two-dimensional array. We then use zip() to merge those two lists into a bunch of new lists, each comprised of the elements from the input lists that share the same index. And because zip() actually returns a generator instead of a rendered list, we wrap it all in list() to get the final result we're looking for.
So that's how we've converted the Y and X arrays returned by np.where() into a list of (X, Y) tuples.
Now that we have a list of matching locations, let's draw rectangles around all of those locations. We'll adapt our code from part 1 to do this, and simply loop over all the locations.
Keep adjusting your threshold, and your comparison method, until you get as many correct matches as possible without any false positives. In this case, I achieved the best result by using cv.TM_SQDIFF_NORMED and setting a threshold of values below 0.17 (remember TM_SQDIFF_NORMED uses inverted results).
You'll no doubt notice that you are getting many more location results than the number of rectangles you're seeing on the result image. You'll also notice that some of the rectangles are much thicker than others. This indicates that you're getting many match results that are very close to one another, and your rectangles are overlapping each other.
Visually this may not be too problematic, but if you're trying to count up the number of some item in an image, or if you're searching for screen locations to click on, then you'll need some way to clean this up. And that's what we'll be covering in part 3 of this tutorial: how to group those overlapping rectangles into single detection results.
Read the full written tutorial here: learncodebygam...

[@AlexTubu]

At first I thought that I just wanted to cheat on the game. Then I saw your video. Then I saw the future. And now I know that game will be only the first step for me in this story.

[@LearnCodeByGaming]

Watch out, the coding might be more fun than the games themselves 😄

[@gmac9313]

And how it's going now?

[@Гусьэкономит]

It has been two years already, where is the continuation? Why did you abandon your channel? You explain better than all my teachers! You're the best! It has been 2 years already, perhaps you have improved object detection, bot performance, and their movement between points, or maybe there's something else interesting to see!

[@spencer5028]

Got hired by DoD

[@patwicker1358]

Just discovered your channel and am enjoying it very much. I really like your presentation style, the fact that you show manual pages, and that you know what you are going to say with no fumbling.

[@LearnCodeByGaming]

That's the style I'm going for. Glad you like it!

[@zen123w]

great explanation of zip(*location[::-1]). Being some what familiar with opencv, I have never come across such a clear explanation of that line before.

[@cidsantiago9577]

I am learning opencv because I love to automate tasks and I love to play online games and I have to say your channel blows my mind. I watch and re-watch all of your videos. It have been AWESOME learn with you, thank you very much spending time teaching us all.

[@Charvin]

I have been waiting for this part very eagery, and it was worth the wait. My test script was able to identify objects I show on an image. There is a sense of great satisfaction when you see something you made work. Figuring out just the right threshold where the program can find out things quite right was the part which took a bit of time & lots of trial and error (understandable). Really glad that it worked out. Thank you! Looking forward to learning from the next videos in the series as well as from the old ones on the channel.

[@LearnCodeByGaming]

Thanks, glad to hear you're following along!

[@pacman804]

I love you style of teaching so much, hands down one of my best programming channels.

[@thedeveloper643]

i've waited for the next tutorial since the first one and here it is! thank you so much for valuable contents

[@LearnCodeByGaming]

Thanks, glad you look forward to these!

[@JS-kr7zy]

super excited for this style of tutorial. Project centered learning is definitely the way to go, and there isn't enough.. I don't want to say "hand-holdy" content, but after you get to a certain point programming kind of takes a leap off the diving board into dense documentation.

[@LearnCodeByGaming]

Yeah for sure. I try to lead people to the fun stuff so they're more excited to explore on their own from there.

[@greedsloth1516]

Glad to have found this channel! Keep up the good work Ben! Stay safe amidst of the pandemic

[@LearnCodeByGaming]

Thanks, you too! Thanks for checking out my videos.

[@laboratoriya]

Perfect tutorial. This gentleman explains everything crearly.

[@DemandRS]

First time I've seen unpacking with *, really cool. Good video thanks :)

[@SquallHart05]

You're an EXCELLENT teacher. Thank you!

[@yunustalhaerzurumlu6547]

Dude your channel is amazing, your presentation is really clear, Keep up with the good work!

[@alekaditez3830]

some of the best python tutorials out there. Much appreciated and good luck to you!

[@LearnCodeByGaming]

Thanks, glad you like them!

[@nerdwolf3839]

Never stop making videos. Your videos are amazing!!!

[@ProficusLets]

Спасибо за прекрасные уроки дружище!

[@emiletheroux2317]

Trying for Path of exile, its hard but your lessons helps a lot.

[@tobias6361]

Amazing that you try that on an Albion screenshot and what a coincidence that I’m using template matching for an Albion project too :D

[@MrBru96]

Great video! I look forward to seeing this kind of object detection into a real time video. Keep it up!

[@vakula6859]

Ты делаешь очень крутые вещи для очень привычных, для многих, занятий. Большое тебе спасибо! You do very cool things for very familiar, for many, activities. Thank you very much!

[@markr87]

Just came across your videos recently and I have to say you are amazing! I have been trying to learn programming for nearly a decade but it never really grabbed my attention, and totally lost by the time i got to object oriented programming. First time ever I am actually enjoying the lessons. Subbed and crunching through all your videos!

[@keve8586]

you are the man! thank you for your content and channel!

[@LearnCodeByGaming]

Thanks, glad you like it!

[@faizaanazam]

Excellent series so far. Thanks, Ben!

[@prodffstarboy]

lovely ! trying to build a bot for a mobile RPG game emulated on Bluestacks, this has been a great help !

[@prodffstarboy]

@Xarius01 what do you wanna know

[@shamelessvideoeditor3839]

great video, thank you buddy. Looking forward to running through some of your other videos.

[@goob6433]

Man this is funny I actually came to your channel with the pyautogui video hoping to code a fishing bot for Albion online, and look what I see! A tutorial for harvesting cabbages

[@LearnCodeByGaming]

Haha, nice coincidence!

[@palette454]

you saved me so many work hours with openCV

[@javiermusso3010]

Hey, really enjoy your style in this videos! Keep it up and thanks a lot!

[@ЕвгенийСергеевич-л7с]

Всех приветствую! Бен большое спасибо тебе за твой труд! В русском сегменте ютубу, к сожалению довольно мало именно прикладного использования Opencv, информация подается в виде поверхностного ознакомления. Еще раз тебе спасибо!

[@ЕвгенийСергеевич-л7с]

Greetings to all! Ben thank you so much for your hard work! In the Russian segment of KZbin, unfortunately, there is quite a little application use of Opencv, the information is provided in the form of a superficial acquaintance. Thank you again!

[@ProficusLets]

Полностью согласен с тобой!

[@leonardochaves6569]

thanks for the content. It was everything I was looking for. You even got the game right that I would like to make a bot.

[@ColeDockter-g1t]

Would love to see a breakdown of diablo 2's pixel bot, which is coded in python & uses this library. Hope there are more of these!!

[@cedricdeege5763]

Thanks man, really clear and easy to follow!

[@Alexander_Meyer]

Cant wait for the next episode :D

[@LearnCodeByGaming]

Next video is up!

[@tymothylim6550]

Hello Ben! Thank you for this video! I think the explanations were clear.

[@aidancohen8721]

I love these videos. thanks so much these are great!

[@svampebob007]

Really interesting and nice explanation. I find that using games to learn how to program is a really neat way to learn the basics of a program. Mainly because you are already familiar with the game, but also because there's 1000 different things to "program" for in a game. I started with programing a rocket guidance system in KSP by using KOs, but also it made me learn more about Bash and SSH since I was running KSP over the network and I used SSH to connect to the game "telnet console". I'm going to hell for this one but: One eye on the code, one eye on the console :)

[@LearnCodeByGaming]

Yeah the project ideas with games are endless, which is great for beginners because it helps you get a lot more time writing code. Kerbal Space Program is fun, haven't played it in years!

[@JordhanRDZ]

Amazing series ! Can you give some spoiler for the next episodes? xD

[@LearnCodeByGaming]

Thanks! Part 3: Grouping rectangles and calculating mid/click points. Part 4: Fast real-time image capture. Part 5: Real-time object detection. Part 6: Image processing for better match results. Part 7: Integrating automated clicks and bringing it all together. Might be a few parts past that, too.

[@irynashevchenko1575]

incredible !! super!!! great work!!!

[@martinneff1681]

Great Tutorial, Thanks

[@lordraphi]

Waiting for the next one

[@LearnCodeByGaming]

Next one is up!

[@falsemessiah5515]

keep doing this! ive learned alot

[@jayadevashok2070]

Awesome tutorial bro!!!

[@DanielXavierDosReis]

awesome good! very nicee lesson, bro, keep it up, very well explained

[@MrRAk1N]

Have you considered making a comparison using YOLO?

[@Natsumi_Nakamura]

[spanish] 2023 y me ayudas un monton con todo esto. lo usare para programar bots >:D . I♥U so much

[@waynewu7763]

thanks alot for these tutorials. I am also learning as I follow along slowly. I have a question, for your for loop, would it be possible (or easier) to compact into a list comprehension? if so, how would that work?

[@rb_420]

you are amazing!! thanks for sharing so much knowlodge (:

[@Backflipmarine]

Can you use multiple images or is that more tensorflow? Whens the next vid btw?

[@LearnCodeByGaming]

You can search for multiple images, but it starts getting slow pretty fast in my experience. I filmed the next video earlier today, so I should be done editing and have it uploaded in 2-3 days.

[@dr.kingschultz]

Thank you for the videos!

[@kosmonautofficial296]

Great video! Thank you for this great tutorial I. I am hoping to use this for detecting what round I am on in nazi zombies.

[@TravisJMedia]

Amazing, thanks for your tuts! Can you show us an example on how to use opencv to let the bot detect when the bot reaches the edge of a zone? I'm trying to let my bot gather resources by walking randomly and with an unstuck feature. Only challenge is that he keeps getting stuck at the edge of the map.

[@LearnCodeByGaming]

What game?

[@TravisJMedia]

@@LearnCodeByGaming Albion

[@LearnCodeByGaming]

@@TravisJMedia Cool, I'll see if I can find a solution for that and include it in a future video.

[@TravisJMedia]

@@LearnCodeByGaming Thanks a lot man! You're the best.

[@thewdead3673]

Hey, Great tutorial! Is it possible for you to make some tutorial on how to walk your character in in a defined square? I am trying to make a bot for pokemon game farming in grass and I am not sure how to move my character in the same area without it going to the wrong way. Thanks!

[@waenurdeanwaehamad2787]

I wonder why does it have to reverse the locations, does it mean that np.where() returns the array of ys first then xs?

[@t-72k16]

Thank you so much for these, i was wondering if you can go into some better object detection that can handle camera angle changes in game?

[@LearnCodeByGaming]

Yeah that does get tricky when the camera angles change. I'm going to be covering more and more advanced topics as this series continues.

[@t-72k16]

@@LearnCodeByGaming using the other technique of for looping through each pixel and seeing if it matches a color on a list. It works pretty well but can be slow at times

[@nburn8306]

I ran into a problem with the code example at 10:23. When running that in VS Code my CPU usage spiked, the program took 5-10 minutes to finish, and the image result was different from the one in the video. For anyone having this example freeze up on them, the issue for me seemed to be calling "cv.waitKey()" with no argument (line 35). After changing "cv.waitKey()" to "cv.waitKey(0)" the code ran in a fraction of the time and produced the expected image result.

[@LearnCodeByGaming]

Thanks for pointing that out! From video 4 I started using cv.waitKey(1) myself. I don't recall why I made the switch, but I probably ran into problems, too.

[@nburn8306]

@@LearnCodeByGaming Weird, I just went to double check and can't recreate the bug anymore. waitKey now works even without an argument. Some "magic smoke" stuff going on in my computer...

[@LearnCodeByGaming]

@@nburn8306 Oh man that'll drive you crazy. Glad it works I guess?

[@magnumdavid6394]

Thank you!

[@SpieltGames]

Why do you have a black left eye, did you have any blows?

[@kevinflynn4867]

Awesome !

[@andreasas1462]

nice project dude

[@venkateshvenky5489]

Can we able to detect multiple objects of by giving two different sample templates

[@LearnCodeByGaming]

Yes, absolutely!

[@venkateshvenky5489]

@@LearnCodeByGaming Could you please tell me how to give input to it.Thanks in advance

[@Magistrmate]

Спасибо за видео!

[@AntonVasilev-q3t]

KOOOOOOOOOOOL!!!!!!!! THANKS!!!!!!!!!!!!!!!!!!

[@airidasrom583]

it opens for me like 4 windows and in the last it only shows all the images im searching tho after i close same image with same marked sports appear.. any fix ? or its suppose to be like that ?

[@jcorrify]

Using python 3.12 and opencv python 4.8.1 and the result format look different in my debugger, I can't scroll through results

[@MultiBelz]

very cool thank you

[@locutusvonborg2k3]

ok, i do have the problem, that a result, which is false, has a higher confidence value, than what i am looking for, so i cant set the threshold higher, cause then i dont get what i want. however, my object im looking for isnt static, its kinda a moving gif xD still very nice videos so far, learned a lot

[@ismaelRR]

Your are a boss!!!!!

[@whxmpy56]

Matchtemplate thing does not work comes up with this (-215:Assertion failed) _img.size().height

[@wolfpackgaming2571]

what settings do you use in vs code?

[@nguyenhungiot]

Hello, I,m using openCV to play game Mir4 automate by detect quests in game, please help me some idea! Thank you very much

[@kuhicop]

For some reason, sometimes it will return a full black image when calling the WinCap.get_screenshot() method. I have to keep restarting the game and visual studio until it works again. Someone figured out this?

[@siman211]

I have Pycharm so can't run and debug i tried to get visual code but can't make it to install opencv on it don't know why I am new to coding i try a lot of site codeacadamy, pirple but i just can't still learn you somehow manage to explain a little better then others i hope i manage to learn something from you

[@diegogutierrez5989]

Thanks 👍👍👍

[@Egeevis]

thanks a lot

[@matheuskjn]

Conteúdo muito bom

[@onggiachoigame4929]

I use your code: threshold = 0.85 locations = np.where(result >= threshold) print(locations) # the np.where() return value will look like this: # (array([1, 5], dtype=int32), array([1, 15], dtype=int32)) and this is my result: (array([483], dtype=int32), array([514], dtype=int32)) please help me.

[@Alex_Aly]

Does anybody know a scale-invariant Template Matching method ?

[@vitustiberius2645]

try out the python plugin and use the '# %%' code block to run python interactive cell

[@hamburgerhamburger986]

My code draws 5-10 lines for every object for some reason

[@sobrelaroca2701]

Emm bro. I love you

[@EranM]

9:20 I think theres a thing called "Transpose" Which is much more intuitive and readable

[@aminghasembeigi7078]

commenting for the algorithm

[@pastuh]

And peoples think Diablo Resurrected will not have bots :X

[@dedpihto680]

it couldn't even select all the required objects without garbage

[@CyberZyro]

its way more easier with cascade files

[@leingod7779]

trying to make a BOT for a game :D