OpenCV Object Detection in Games Python Tutorial #1

Learn how to use OpenCV for object detection in video games. This intro tutorial will show you how to install OpenCV for Python and get started with simple image template matching. This will serve as our foundation as we explore many different computer vision techniques in future videos.
Full tutorial playlist: • OpenCV Object Detectio...
GitHub Repo: github.com/learncodebygaming/...
OpenCV documentation: docs.opencv.org/4.2.0/
Official template matching tutorial: docs.opencv.org/4.2.0/d4/dc6/...
0:24 How to install OpenCV
1:45 How OpenCV relates to PyAutoGUI
2:52 How to navigate the OpenCV documentation
4:53 Comparison methods visualized
5:39 Writing code for cv.matchTemplate()
OpenCV is an open source computer vision library with hundreds of functions for processing and understanding images. In this tutorial, I'm going to show you how to get started with OpenCV in Python by using it to find an image inside another image. This simple form of object detection will be a good starting point before we move on to more advanced image recognition techniques.
The quickest way to get started with OpenCV is: pip install opencv-python
Once installed, you can use the library by importing cv2. Numpy is used extensively when working with OpenCV data, so the top of your Python files will look like this:
import cv2 as cv
import numpy as np
That's all there is for setup. Now let's grab an image we want to process. I'm going to be using this screenshot from Albion Online, but any screenshot will do.
What we're going to do is crop out a small section from our screenshot, save that as a separate image file, and then we're going to use OpenCV to find the position of the smaller image inside our entire screenshot. From my screenshot, I'll crop out one of the cabbages.
The OpenCV function we'll be focusing on is called matchTemplate(). In the documentation, we can see we're going to give this function an image to search over, an image to search for, and a method type for doing the comparison. And we'll end up with a result array. You'll want to experiment with the different comparison methods to see what works best for your use-case.
Alright, let's write some code. The first thing we want to do is load our image files.
haystack_img = cv.imread('albion_farm.jpg', cv.IMREAD_UNCHANGED)
needle_img = cv.imread('albion_cabbage.jpg', cv.IMREAD_UNCHANGED)
The "haystack" image is our screenshot, and we'll be search that for the "needle" image we cropped out. In imread() the first parameter is the image file path, and the second parameter is a flag that allows us to do some pre-processing when loading the images. In this case, we're loading them in unchanged.
Now that we have our images loaded, we can go ahead and call matchTemplate(). I've had good luck using the TM_CCOEFF_NORMED comparison algorithm.
result = cv.matchTemplate(haystack_img, needle_img, cv.TM_CCOEFF_NORMED)
We can quickly see the results from matchTemplate() by displaying that data with imshow().
cv.imshow('Result', result)
cv.waitKey()
In imshow(), the first parameter is the window name and the second is the image we want to show. I've also called waitKey() to pause our script while we review the image. Without this, our script would quickly close before we could see the image. Pressing any key on the keyboard will trigger waitKey() to stop waiting, thus ending our script.
In this result image, the bright white pixels represent the positions that best match the cropped image. The black pixels are the worst matches. Note that these best match positions correspond with the upper left corner of where you'd place the needle image.
Now that we've visualized the results of matchTemplate(), let's get those best match coordinates. We can do that using minMaxLoc().
The minMaxLoc() function returns four values. First are the confidence values for the worst and best matches, on a scale from 0 to 1. These are how black or how white the darkest/brightest pixels are in our result image, where 0 would be perfect black and 1 would be perfect white. The last two values minMaxLoc() returns are the positions of those worst/best match pixels in the form of an (X,Y) tuple.
For every needle image that we give matchTemplate(), we will always get back some values from minMaxLoc(), even if that cropped image appears nowhere in the haystack. We can tell when we didn't find a good match because the max confidence value will be low. How low is too low depends on the images you're working with and what you're trying to achieve.
Now that we've found a good match, let's outline where we found it in the haystack image. We can do that using OpenCV's rectangle() function.
Continue with the written tutorial here: learncodebygaming.com/blog/op...
Join me on Discord: / discord

[@ilkayatil]

I just found out your video and even as a senior python and opencv developer I enjoyed watching all of it. Learning image processing through games is an excellent idea. Your presentation is clear and precise, I disagree with comments which states it is a bit boring. Your explanation level is just right for the beginners, not overexplaining things and exploding beginner's minds :D I wanted to take my time and display my support with a comment because you clearly dedicate your time and enthusiasm to create a helpful content for the community. You just earned a subscriber, keep up the good work my friend. p.s. About the not so perfect matching score, it is probably due to using jpeg format's lossy compression. After you saved those images as jpeg, they will have very small differences invisible to eye but clearly visible to code as in this example. You might prefer using PNG for lossless image saving or even BMP to get faster image write/read times (high fps).

[@josh9761]

second this by far the best explanation i have came across, zoomed in code, well explained step by step

[@E30Andy]

I went through this tutorial but deviated by using png files instead. ultimately this makes the tutorial not work as written because the data was no longer in the expected format of CV_8U or CV_32F for the matchTemplate function. Just a warning for those that like to experiment lol. My understanding is that you can easily convert to the proper structure, but me, being a noob, did not find success with this. I saved my files to .jpg and moved on with the tutorial instead of getting hung up and everything worked.

[@wingchangwow]

Honestly, the quality of your videos and tutorials is simply astounding. I'm constantly amazed that you don't have hundreds of thousands of subscribers....yet!

[@brednbudder]

I love your videos. they make it more fun to learn to code. thank you very much for taking the time to make these, especially into easily digestible chunks in a playlist. youre awesome!

[@Joshua-kr5fq]

Nice to see you taking it to the next level

[@deusexpersona95]

This channel is God sent. Keep going and you'll grow very fast

[@mangojango3745]

I was JUST looking up info on this. Who knew it would be from the guy I subbed to from a few days ago. Keep up the good work your putting out unique and great content

[@LearnCodeByGaming]

Thanks means a lot!

[@diegorivera_cr]

Thanks alot for this video! Do you think you can cover how to find an object and also detect when it is gone in a video game?

[@jamiestwrt]

Trueeee

[@mangojango3745]

@Carroll Horsely stop scamming people

[@Tradie1]

Exact same here! What an underrated channel

[@Charvin]

This was something I was wondering about, Thank you. I don't think there are much interactive content like this out there. Glad to have found this channel. Thank you once again.

[@LearnCodeByGaming]

Glad you like it!

[@eliotbicak8296]

It took two years but I am very happy that I managed to find your channel! Thank you for the great tutorials and videos

[@raakapena]

Your OpenCV videos inspired me so much, been learning constantly new things and doing some low profile botting too. Best channel I've stumbled upon in a long time :).

[@darkferiousity]

Wow dude been watching a lot of videos on open cv and yours is far the best I have seen. Just the way you teach is awesome.

[@warpcode]

I've just started getting into object detection in opencv along with tensorflow, yolov4 etc. Whilst I'm not using it for gaming, these tutorials have been great for real dynamic and practical applications and for that i want to say thank you :)

[@saint-jiub]

I'm on part #3 of your series so far but wanted to revisit this to say - you've been helping me get out of tutorial hell. Thank you so much Ben! You have an amazing teaching style; The documentation and comments along the way in both video & blog posts provides more clarity than other tutorials. Your channel is one of the best out there, please don't stop. Peace.

[@LearnCodeByGaming]

Thanks man!

[@setokibah4476]

Here I was wondering how i was going to pull of object detection, then my man presents the answer to me just like that! Always appreciate you, never fail to expedite my thought process!

[@fierce1340]

Just stumbled upon your channel! Love the style and format! Can't wait to check out more videos!

[@teslar735]

this channel is so much underrated.Very good content bro,keep it up

[@redegs_]

You did a great job at making this easy to understand, I followed it all the way through and it worked, Thank you

[@johannes-euquerofalaralema4374]

Hi there, this is the best channel about this content. I have watched your Videos about Python at least 20 times each! I am a beginner and was able to follow your explanation and even to costumize the code for my needs! Really amazing!

[@user-oc8on5gz5s]

I have started learning computer vision not long ago and basically you are the best. Thank you very much for your tutorials

[@dazealex]

Brilliant channel. Enjoyed every bit of it. Love the short length. Subbed!

[@hansenliu2000]

God I love you and this series, the quality content we all need but do not deserve

[@MrBru96]

Thank you so much. Your explanations are so clear. I learn a lot from you. You are the best!

[@devpatel5597]

Underrated KZbinr. Great content!

[@WhyNotProgram]

Thank you, I love how you guide us through the documentation! This will help me start creating AI's to play games and automate tasks.

[@onion9377]

Wicked video! thank you for the tutorial. After fumbling around with a few other guides, it was yours that really got me into it.

[@LearnCodeByGaming]

Awesome!

[@tower1990]

Thanks Ben :). I really like your teaching style. Please do more if you’re available!

[@MrFox-uf7kv]

why this guy don't have above 100k subs.. he explained it so good. +1

[@EvolMe]

hey i JUST found this channel today, MapleStory is my childhood favorite game so this has motivated me to learn to code and use maplestory as my game, my younger self is screaming right now lol love your content

[@cavaleirosemcavalo69]

Thanks for the video, I'm from Brazil and here we don't have many good videos about OpenCv and Python , the simple and objective way you speak is perfect for me who doesn't is a native and thanks for not using so many slangs

[@SquallHart05]

You're an EXCELLENT teacher. Thank you!

[@OsmioGonzalez]

(Español) Muy buen video, conciso pero detallado. Atiende a problemas pequeños de forma que permite aprender poco a poco y construir el código lentamente y de una forma muy natural. Mis felicitaciones sinceras, espero continúes con este trabajo de divulgación tan pulido. Solo me queda felicitarte de nuevo y seguir tu canal y aportes. Mucha suerte para el futuro. (English by google translate) Very good video, concise but detailed. It attends to small problems in a way that allows you to learn little by little and build the code slowly and in a very natural way. My sincere congratulations, I hope you continue with this polished disclosure work. I can only congratulate you again and follow your channel and contributions. Good luck for the future.

[@shadowhuntzgaming3055]

YOUR TUTORIALS ARE THE BEST IVE SEEN, THENK YOU SO MUUUCH!!!!!!!

[@topkek239]

This is my new favorite channel, thanks for good content ^_^

[@iamrockstar2101]

GOLD CONTENT thanks , i just needed something like this to complete my project you earned a new subscriber :D

[@LearnCodeByGaming]

Glad you like it! Good luck on your project.

[@oswaldogerardino]

I'm from Latin America, and I understand your tutorials better than the ones in my language. You're great, thanks

[@GoeieWoofer]

Yooooo Ben i hope you are still Alive and coding! I honestly cant thank you enough for all your content! It has finally gotten me started with programming and coding, im studying to become a python developer and ive truly found my passion and calling in life now. The way you teach and explain things is amazing, wish i would have gotten help like this sooner in life! Much love for you

[@LordXavier79]

Thx, People like you make the world a better place

[@pollos4157]

Whoever disliked this needs their head checking, this was one of the best programming tutorials I have seen yet.

[@sanyamchaurasia1542]

LOVED IT! Please, please, please! I would love more tutorials like this! Can we also make basic games like tower defense, basic shooter, or something similar?

[@JordhanRDZ]

YES !! This is going to be very interesting ! =D

[@ramborambo5677]

Thank you for this. Keep em coming :)

[@alchercananea5975]

Hi there Ben! I stumbled upon your channel because I wanna learn how to code. I watched this video to see if it interests me though I don't have any knowledge yet. But boy! You explain things so well that I get the idea of what you are doing. I am glad I've found your channel. +1 sub!

[@LearnCodeByGaming]

Awesome, always happy to see new people giving code a try! Thanks for subscribing.

[@jaski8143]

absolutely amazing content !

[@thedeveloper643]

this guy should get more subscribers! thank you for the great contents

[@LearnCodeByGaming]

Thanks!

[@shdwrzr07]

Hey man just want to let you know that I'm so happy right now as a gamer and python programmer :) You just earned a new subscriber!

[@LearnCodeByGaming]

Glad other people like this stuff too!

[@DamafiakingzDMK]

You are great bruh I'd literally crown u as king.

[@GoldmediaSubscribeme]

This me first time watching your videos and loved it!

[@angelsdemons8520]

Open CV is great no denying but the library is just huge , it's full of functions to do all sorts of thing , this is a great advantage(keeps ur code short and neat) and disadvantage ( I really find it difficult to remember these lol) like think about it for a sec, there is function for like literally everything and parameters which makes no sense for newbies (like me myself), but nevertheless great video mate , really liked how u walked through the documentation (encouraging viewers to explore) and explained as much as possible while writing the code and made the video as short as it possibly could have been u certainly deserve more subs and recognition mate :)

[@javagenbu7085]

Thanks for the video my man, very useful and well explained.

[@ae171092]

great stuff. no idea why the views are not in millions. kudos

[@daltonrinck3617]

Thank u very much for this video. It was Informative and engaging. Makes me want to learn more about OpenCV!

[@VersusBR]

print("Hello from Brazil") Thx bro

[@redmenace5536]

Thank you for your tutorials!Really appreciate it :)

[@LearnCodeByGaming]

You're welcome, glad you like them!

[@kaioshin9143]

Thanks for the video, I now learned what I needed. Cheers!

[@vromel4331]

Bruh ty so much! You save so much time for as !

[@LearnCodeByGaming]

You're welcome!

[@mcleber100]

Thank you for sharing your knowledge. Greetings from Brazil.

[@PythonHarry]

love you so much, I love this course, you just got a Chinese Subscriber!!!

[@dzdeparsio4676]

thanks i will watch ur vids and learn this library with u keep up the good work

[@n1guild]

Great content as always!

[@LearnCodeByGaming]

Thanks Shane!

[@SolusAU]

Your videos are super clear. Thankyou :)

[@indirectthought8150]

Thank you so much for these videos, this is gold

[@AskAKill99]

Amazing videos!thankyou so much sir!! Much praise for your work

[@shi-woonyi2605]

Thanks for the tutorials, I like all of them, it's much more fun to learn coding/new libraries if the examples are interesting topics too in this case, games! keep up the good work, looking forward to future tutorials, OpenCV looks awesome.

[@LearnCodeByGaming]

Yes, that's the idea of my channel! Thanks!

[@FollyOx]

Thanks dude, I want to make a very simple bot to automate a PIA element for a really old game and this is a good excuse to learn Python a bit more

[@Lucasbc]

Amazing explanation!

[@spadedaman4958]

good video and a writeup! You are doing a great job!

[@lamvandinh789]

Thank you for the great tutorial!

[@nolancao2878]

thanks for these 9 masterpiece videos

[@kevinflynn4867]

Thank you Ben , very interesting content !

[@imperatoreTomas]

Keep making videos. This is awesome!

[@renaud6904]

thank you so much Ben ! I viewed your video, red your web site, and i have done my new project in few minutes !

[@Googlrr]

hey this was insanely helpful, thank you very much

[@Joshua-kr5fq]

I just did the sentdex opencv tutorial so this is perfect

[@LearnCodeByGaming]

Awesome hope it's helpful for you

[@Joshua-kr5fq]

@@LearnCodeByGaming Stardew valley is about to get more interesting.

[@RK-de3uu]

thanks for the clear explanation!

[@situwang1831]

Thank you for your videos, you are very good, I have received a lot of inspiration

[@AhmedKamel-ol6qv]

Great playlist

[@lordraphi]

Nice video! Good for starting programming a mmorpg bot!

[@zarakikenpachi1238]

Thank you for the video :) That s very interesting

[@rcpinto]

Wonderful video!

[@AlejoCuartas]

Great tutorial, thanks!

[@billsultani8967]

Your a legend mate thank you so much!!!!!!!

[@uzzifuzzi9884]

Can't wait to see where this goes

[@LearnCodeByGaming]

Same! I need to slow down on coding and catch up with making the videos.

[@bimamrth]

Nice this verry useful for me 👍

[@jesusyanez100]

ur the goat @code by gaming

[@amirhosseinfalahati4388]

just found good stuff on yo channel, Thanks!

[@user-ei9dv9nr2p]

ความรู้ทั้งนั้นขอบคุณมากครับ

[@lightningcode7603]

Great video!

[@krocodilnaohote1412]

Very good, thx for your videos

[@shiny_apricot]

Nice explanation

[@slttcoding4530]

Great Video!

[@szymonduranowski4510]

great explanation !!!

[@ibnelaiq]

Nice Need More Like This

[@CallingTheReaper]

Love the videos dude! Looking forward to the next one! I was getting an error with this code, Had to convert both variables to greyscale?

[@LearnCodeByGaming]

That's interesting. Maybe something to do with the format of the image files you used? Glad you like the videos!

[@MultsElMesco]

I had the same error. I was using PNGs but it worked well when I switched to JPGs

[@ron_jak]

Hi, love the vids! Hope you're doing well. Keep up the content. Cheers.

[@LearnCodeByGaming]

Thanks Ron!

[@vampirefox9981]

Thank you!

[@S25plus]

🙏 Thank you sensei!

[@mikeygoby6154]

good explanation, ty

[@user-mh1ki8ke9p]

It has been two years already, where is the continuation? Why did you abandon your channel? You explain better than all my teachers! You're the best! It has been 2 years already, perhaps you have improved object detection, bot performance, and their movement between points, or maybe there's something else interesting to see!

[@xdeymin1612]

Very nice Video!

[@spaska1184]

great tutorial

[@bezdna7366]

LOVE LOVE LOVE!!!!!