Fast Window Capture - OpenCV Object Detection in Games #4

Learn how to capture window data in real-time as a video stream for processing with OpenCV. We try several different methods searching for the fastest way possible. In this tutorial I also discuss the importance of good Google search skills as a programmer, and we revisit some basic object-oriented programming concepts.
Full tutorial playlist: • OpenCV Object Detectio...
Grab the code on GitHub: github.com/learncodebygaming/...
Research discussed:
OpenCV getting started with videos: docs.opencv.org/4.2.0/dd/d43/...
Fastest way to take a screenshot with Python on Windows: stackoverflow.com/a/3586280/1...
Convert PyAutoGUI image to OpenCV: stackoverflow.com/q/32918978/...
Convert SaveBitmapFile to an OpenCV image instead: stackoverflow.com/q/41785831/...
Best Numpy reference: numpy.org/doc/1.18/user/quick...
List all your windows: stackoverflow.com/q/55547940/...
0:47 Main capture loop
3:16 Using PyAutoGUI screenshot()
6:24 Measure FPS
8:27 Using Pillow ImageGrab
9:31 Using Pywin32 for screenshots
13:48 Converting win32ui.CreateBitmap() for OpenCV
17:23 Confining screenshots to a specific window
18:42 Creating a WindowCapture class
25:05 Trimming the window capture
28:15 Image to screen position conversion
29:35 Wrap up
Read the full written tutorial with code samples here: learncodebygaming.com/blog/fa...
Up to this point, we've been using OpenCV to detect objects in static images. Now we're ready to apply those same techniques to video games in real time.
Remember that video is just a series of still images shown in rapid succession. In this tutorial our goal is to capture screenshots as fast as possible and display them in an OpenCV window, so that we get a real time video stream of the game we're interested in. This will set us up to begin processing image data with OpenCV in real-time.
OpenCV has a tutorial on "Getting Started with Videos" that will serve as the basis for our code. Our starting point differs from the official tutorial only in that we are preparing to work with screenshot data instead of frames from a camera.
When defining get_screenshot() you could simply use pyautogui.screenshot() from the PyAutoGUI library, or ImageGrab.grab() from the Python Image Library.
And this would work, but there are several benefits to calling the Windows API directly instead. Firstly, we approach the theoretical limit for how fast we can take these screenshots by dealing right with the operating system itself. Secondly, the Windows API has methods that will allow us to grab the screen data for only the window we're interested in, even when it's minimized or off screen.
To do this, we must first pip install pywin32 to get access to the Win32 API in Python.
Let's start with some code to capture a single screenshot of our entire desktop and save that to a file. This will confirm for us that the Windows API calls are working. By calling this function, you should end up with a debug.bmp screenshot file.
The next step is to modify this function so that instead of saving an image file, it instead returns the image data, formatted to work with OpenCV.
Now we can call this function from our original infinite loop and get a real-time stream of our desktop.
To improve upon this, we can use win32gui.FindWindow(None, window_name) to capture just the window we're interested in. Replace the window_name with a string that contains the name found in the title bar of the window you want to capture. Doing so will allow you to capture the frames from that window even when it's hidden behind other windows.
If you're having trouble figuring out the name of the window you want, you can use this code to list the names of all your existing windows:
We can improve our code further by trimming off the excess around the window we're interested in. When you run the above code, you will notice black space to the right and below the window image, as well as the window borders and title bar. Removing these will not only clean things up, it will also improve our frame rate. We can also get improvements by not calling win32gui.FindWindow() on every call to get_screenshot(), so let's turn this into a class.
Finally, we'll need a way to convert positions we detect in our screenshots back to pixel positions on our actual monitor. In the WindowCapture class constructor, I've already included code to calculate the window offset using the window position data from win32gui.GetWindowRect(). Let's add a method to our class that uses this offset to return that converted screen position.
Continue with the written tutorial here: learncodebygaming.com/blog/fa...

[@LearnCodeByGaming]

If you're having trouble with a black screen when you try to do the window capture, continue on to video #5 for a fix.

[@surf7863]

I have been trying to capture a game to make a notification system to notify a player on discord via a bot when they find a match. i am able to do it when the game is open in windowed or borderless windowed, when using fullscreen it returns a blackscreen.

[@trevorsatori]

search skill increased to 70

[@simona4294]

Nice video but I don't get the point, it's not faster than pyautogui at all, same 20 fps so why even bother?

[@trevorsatori]

@@simona4294 Try MSS for screen capture. You should see noticeable fps increase. Also If you watch his last video in the series where he implements threading, fps increases to over 100+ and no longer is an issue

[@deckard6052]

Which VScode color theme are you using in this video, I love it! Awsome video btw helped me a lot.

[@navidchaichi71]

The most honest programmer I have seen in KZbin, keep continuing what you are doing. I rarely put comment for someone... Thanks!

[@LearnCodeByGaming]

Thanks appreciate it!

[@s1ack3r07]

This is the first playlist on KZbin I've been waiting on the next episode, learning tons thanks for this resource!

[@TWBIG65662]

This is one of the finest tutorial I've searched. It goes step by step and everything was explained. I'm glad I didn't miss this one.

[@psiborger7274]

You are quite possibly the best instructor I have ever had...in any subject. Fantastic!

[@VonchkynProduction]

Thank you for making this video. It's really helpful to see the full process of not just building the code, but also researching reference codes.

[@Newton-vi4vs]

this is by far the best episode of the series. I always love the idea of the author walking through the mental process on how solve the problem. Appreciate your effort Ben. Take care

[@TabishTabby]

I have no words to describe how much I like your tutorials. All I can think of is Thank you.

[@Al_Gonzo]

Underrated channel. I'd be happy to see even more videos from you, no pressure.

[@luddesson]

This was exactly what i needed for my project, great timing

[@methodiconion8523]

I appreciate your emphasis on problem-solving and research. It's almost like the specific purpose of your video is second to the more general process of research/coding/bug-fixing.

[@y.m.6988]

It’s quite nice you included your thought process when you were googling stuff, for beginners like me that could be quite inspiring and i haven’t seen any other tutorial videos which have done this

[@FromIVtoV]

It’s awesome that you go into how you research. This probably helped so many people with just research generally - great work! Awesome post

[@PureForloko]

I’ve only seen one of your videos so far, and skimmed through this one but I can definitely say that your unique content has gotten me very motivated to dedicate even more time for programming.

[@LearnCodeByGaming]

Awesome man, keep at it.

[@daddyofalltrades]

You are probably one of the most underrated channels on KZbin for this type of content. You deserve more subs.

[@PainfulDeath]

Man, I seriously enjoy your style of explaining things. It's the small stuff like that remark at 6:10 for example. Kudos and keep up the good work! :)

[@LearnCodeByGaming]

Thanks!

[@mischavandenburg]

I got to hand it to you, you are an amazing teacher. I have worked through a couple of Python Fundamentals books / courses which all included a chapter or two on OOP. But it wasn't until I listened to your game character class analogy that things finally clicked for me! I still have a lot to learn, but I finally understand why the code I am trying to grasp contains all of these "self" references, haha! Thank you very much for these great tutorials and I look forward to watching the rest of the series.

[@thanhhaphan5183]

Thanks a lots for the great video! The way you demonstrate the research to get the solutions is really helpful. Looking forward to your next video!

[@Nathan2904]

i was about to say how good teacher you are but ppl actually just did. Wow, in seconds you were able to explain concepts that ppl take hours to doo. so simple and easy to understand. GRATS

[@lucasfontesgaspareto]

Woow, thank you so much, i saw this video more than one year ago and today i was with an issue related with shape of screenshots in cv2 and i remembered that you used in the tutorial

[@user-cp1vh2qm1m]

Man, thank you so much this really helped a lot. I liked that instead of just saying the answer and keep going, you search Google in front of us about each problem you encounter and then analyze the written code and try to make us understand more. This is the best openCV tutorial I ever saw on KZbin❤

[@Ehxx]

I'm hoping the next video is out today/tmrw, loving these! Surprised they don't have 100x more views.

[@kitsab8415]

Forever grateful for your videos. I haven't started diving into Python but watching you explain a lot of stuff that are not often explained to beginners is sooo helpful. Keep it up!

[@LearnCodeByGaming]

Thanks for your comment. Good to know I'm on the right track!

[@andykim1614]

I can't stress enough how your style of explanation helps us understand both the concept and the practicality of OpenCV. Huge huge thanks!!

[@LearnCodeByGaming]

Glad to help!

[@andrewgreen2626]

Thank you for your calm style of teaching; you alone have helped me learn more than anybody else on KZbin.

[@LearnCodeByGaming]

Awesome! I wasn't sure when I started, but hoped my style would resonate with some people.

[@scoringdigitsson.5194]

This is crazy man, thanks for sharing. Now i have some ideas for my project.

[@NateSpring]

Wow, I learned a bit about Python classes. You present very well and to the point. Nice work!

[@jt_mmxx]

I love the comment at the beginning "Talking about the importance of good googling skills" Hahaha

[@preethamdbz2023]

Cool tuts man. I'm currently developing minimilitia game bot using adb_shell, pynput,numpy. I never thought of using opencv to detects it's characters location - x, y axis. Thanks man.

[@akanuwolf]

I have never been so motivated to learn, I love this video

[@thedeveloper643]

this series keeps getting more interesting!!! aw man thank you so much

[@LearnCodeByGaming]

Glad you like it!

[@garic4]

I had to stop by and personally thank you for this amazing tutorial video. This is top quality material. I have just subscribed and hope to see more of your content. Ty so much for sharing

[@Yes.Im.Mr.Anderson]

This is the first time im gonna turn on the notifications on youtube.

[@kieudung]

Not only got what I need, but also more lesson about googling. Very good tutorial :)

[@riskant8263]

that class explanation was a very creative and gave me a new, more fun perspective regarding classes. Thank you

[@zanesimmons1482]

I've been smacking my head into a keyboard all day trying to understand Classes. I've had it explained to me 4 times (usually with a Class Person analogy), but your explanation was the first time it clicked. Thank you!

[@S1lenc31991]

If i ever would have to teach someone basics about classes, i definately use your explanation. That was awesome, which i would have learned it that way :D

[@davidjmstewart]

Great content, great channel idea. I'm not new to programming but I am new to OpenCV, this is a really engaging way for me to get into the content. Hope you continue on with it. Subscribed with notifications, can't wait for the next one!

[@LearnCodeByGaming]

Thanks David!

[@muhammadasyraf1830]

This is the best free software Ive seen. Respect.

[@abdelalielhammadi6404]

thank you alot for this video , i like the way you explain things especially learning how to search instead of giving us the solution directly

[@Themusicbiz]

Dude I'm so glad I found this channel!

[@uglypole13]

Can I just mention that at 21:05 you explaining why self.h needs to happen as it's no longer within the function is the first time I actually got why that happens. Great material and teacher!

[@ngoantranthanh5492]

you are great. I don't speak English so I have to use google translate to comment. Very detailed instructions easy to understand. For those who don't know anything about programming like me, but also understand. Thank you very much.

[@n1guild]

Capturing just the window is a game changer, looking forward to your next video. Cheers

[@LearnCodeByGaming]

Yeah it definitely is!

[@fernandoaguate5509]

Great videos! Easy to understand and follow. Thank you

[@mariusschubert4882]

Exactly what i was looking for, well explained, thanks A LOT :)

[@kalopseeia1617]

This is the best explaining while coding, Thanks

[@bobbyjoe4012]

Absolutely brilliant work

[@soundofbeauty]

Installed, everything works, thanks!

[@MRSMITHENTOP]

You are a true master at opencv xD bravo 👏

[@CrazyFanaticMan]

I would honestly put you in the top 1% of coders on KZbin who are actually good teachers as well. Wow.. just wow.. I'm amazed at how clear and concise you made everything I'm subscribing for sure

[@ilia9872]

Top 0.1% I would say

[@CrazyFanaticMan]

@@ilia9872 Tbh yeah haha he's just that good

[@ElTebe]

This is awesome! Thank you!

[@JoshCA2009]

Love your videos, very helpful and easy to understand.

[@LearnCodeByGaming]

Thanks!

[@knightUmbra]

"This video is getting kind of long" What are you talking about? We had been here for 5 min... whoa, time does fly when you are enjoying something! Absolutely fantastic, thank you for your work!

[@leonardobarros9800]

The best videos I have ever seen, I look forward to the next part

[@LearnCodeByGaming]

Thanks!

[@4767039]

Easy to follow as you explain.

[@hakanates1188]

insane tutorial thanks for the content!

[@vtgaming9204]

You have taught me so much, such a beast thank you.

[@notu483]

Thank you for helping us beginners out.

[@jimlo]

Great tutorial, thanks for posting!

[@sa-hq8jk]

thanks for showing me some useful tips for google searching solutions to my problems but looks like i didn’t need to! this video was so helpful! thanks so much

[@victorrodrigues9634]

Really nice man! Keep up the good work!!

[@saint-jiub]

Epic video Ben! I've been enjoying the series. This is better than paid udemy courses btw. Thanks!

[@RogerCh888]

This was very helpful, thank you for the content.

[@thiagobrazileiro6690]

Amazing, thank you so much for this video. I'm from Brazil and I dont have a really good english level, but I achieved to understand what you said. Thank you again, see you um next videos

[@rodpadev]

This is a very good tutorial. Subscirbed! I'm learning this so I can build a tool that waits for a specific S class ships to land in No Man's Sky and pause the game. Proably not going to be easy but this already putting me in the right path

[@topeque6993]

DUDEEE I UNDERSTOOD CLASSES NOW SO MUCH MORE THNX TO YOUUUUUUUUUUUU THANK YOU SO MUCHHHHHHHHHHH IVE BEEN STRUGGLING PYTHON OOP AND WHEN I WATCED THIS VIDEO IT MADE IT SOOO CLEAR

[@lougvar]

Thank you very much! I like your class analogy.

[@alokpanchal2953]

exactly the same I was looking for, subscribed and liked too.

[@about1082]

Your Videos Helped Me SO MUCH

[@alexkowalczyk8618]

Wow, this is great. Keep it up!

[@VictorHenrique-mc1gc]

You're amazing! Thank you so much for this masterpiece of a class

[@ramborambo5677]

The way you teach is awesome. Very precise and to the point. Mark my words. You will easily be one of the most followed teachers for programming in the future. Looking forward to your videos. Please do consider integrating ML into game automation for your next series.

[@LearnCodeByGaming]

Thanks for the kind words. Yeah I definitely want to get to machine learning stuff before too long.

[@0day386]

Good instructor. Enjoyed the video.

[@rayth9860]

You are a life saver! Kudos to you! You rock!!

[@zououoz3588]

You literally king

[@chrstnobrmyr9315]

Thx! Great work! I looked exactly for solving this problem...

[@shadikhalil7030]

I learn lots of new things in every single video from you. Although I'm not a beginner Tank you very much ❤️❤️❤️

[@LearnCodeByGaming]

Thanks for watching!

[@rebe1Yell]

Instead of removing alpha by looping through the image array you can run cv.cvtColor(screenshot, cv.COLOR_RGBA2RGB) before cv.imshow('Computer Vision', screenshot). That gives back ~10 FPS.

[@ariyudopertama2371]

noted (by me 😊)

[@brunotatagiba6588]

did you mean this drop inside get_screenshot function? -> img = img[...,:3] the cvtColor can be inputed in the window capture return in get_screenshot function -> return cv.cvtColor(img, cv.COLOR_RGBA2RGB), doesn't necesseraly need to be inputed in main, but i get your point, gonna test it, thanks

[@jairtontf]

Dude! This is perfect! I'm doing a bot using opencv and your video help me a lot!!! Thank you so much!!!

[@LearnCodeByGaming]

Nice!

[@shamelessvideoeditor3839]

thanks for the great video Ben.

[@paulobras4473]

brother you are insane in a very good way! genius thanks for sharing knowledge

[@josh-ryan]

Great video. Keep it up!

[@imperatoreTomas]

Thank you for making this!

[@proxy8918]

You're a legend, thank you so much.

[@AdarshGupta-dk5py]

I love this guy's video

[@michael4353]

Thanks for the videos. I am really enjoying them. Little bit slower progress because I have to translate everything into C++. :) So far so good though!

[@kosmonautofficial296]

Great video! This is very helpful!

[@1kamikaze1]

Aside from the importance of good googling skills, it's wise to also give back to the community, a single upvote could greatly help. I feel sad for the StackOverflow link who deserves an upvote whom you got the useful info from, so I gave both of them one :) Great content along with humor!

[@the7ak3]

Strong work comrade!

[@doctorglock1691]

I really like the way you teach. You are not a "know-it-all" guy. I appreciate that you showed how real programmers work. Sometimes the most basic concepts are forgotten and just a simple google search for an example function is what needed. You got a subscriber here, keep it up! :)

[@LearnCodeByGaming]

Thanks!

[@Alex_Aly]

love your work keep up the good work man :D

[@fofinhoc137]

Você é um gênio kkkk um ano procurando isso, muito legal parabéns

[@federicocanete2905]

Thank you very much! This is an awesome tutorial, helped me a lot :)

[@aarjey5236]

brooo u are the best your videos are just what i wanted to learn thanks man

[@jntb3000]

Very helpful . Thank you so much !

[@shajed99]

I really love this. Thank you.

[@TheDerpsFinest]

best tutorials ever!! keep it up

[@ascenderx]

I'm playing Albion Online and learning to work with open CV How perfect is this!!

[@PierreLR]

The penny has finally dropped after the way you explained classes and the self variable. I can't believe I've never understood classes before. Thanks!

[@LearnCodeByGaming]

That's awesome to hear!