How To Send Inputs to Multiple Windows / Minimized Windows with Python. Or Die Trying

Let's explore using SendMessage to send automated inputs to multiple windows at once, or to windows that are minimized or in the background. I'll share my progress and discuss the suitability of using this method for botting.
View this code on GitHub: github.com/learncodebygaming/...
Join us on Discord: / discord
Today I want to answer the most asked question on my channel:
- How do I send mouse and keyboard inputs to multiple windows at once
- How do I send inputs to a minimized window or background window
- How can I keep using my mouse and keyboard while also sending automated inputs?
I'm guessing most of you want to know how to do this because you've written a bot, and you want to run multiple bots at once, or you want your bot to run in the background while you continue using your computer.
And my initial reaction when I started getting this question was: just use virtual machines. You could have multiple VM's running in the background and those would be isolated from whatever you're doing on your main desktop. I've confirmed that this does work, but the problem is the performance of virtual machines is typically not very good. At least with Hyper-V, even basic games show considerable lag. You might be able to find better performance by setting up hardware graphics acceleration for your VM, but that's not the route I decided to explore today.
So what other options do we have?
About a year ago now, as I was working on my OpenCV series, I got a tip from a viewer telling me about this SendMessage() function he was using to send inputs to a window in the background. It's a Windows API function he was calling via pywin32, and it allows you to send messages, including mouse and keyboard inputs, to a specific window. And there's a couple StackOverflow discussions about SendMessage(), but other than that, there really isn't a whole lot I could find about working with this. So let's try it out and see what we can do with it.
I first tested a simple example with Notepad, and this gave me more problems than I was expecting.
The Notepad window has an inner window called 'Edit' that you need to target if you want to send inputs to the main text area. Even after I solved that, I wasn't able to get any results using WM_KEYDOWN and WM_KEYUP. Only by using WM_CHAR with ord() was I able to get text to appear in Notepad. Both SendMessage() and PostMessage() work this way.
Next I decided to take this to a simple browser game. I found that WM_KEYDOWN and WM_KEYUP did work in Chrome, and there weren't any inner windows to worry about either.
The first challenge I faced here was how to press down multiple keys at the same time. If you sleep() between WM_KEYDOWN and WM_KEYUP that's blocking, so other code can't execute at the same time. You could solve this by using threading, but this time I decided to use sched to schedule each command in queue, and then run that queue once it's been built. This worked really well for one game, so now I wanted to scale it up to play games in multiple windows.
Unfortunately in Chrome, I couldn't get SendMessage to work when the window wasn't focused. So I ended up using SetForegroundWindow() to quickly bring focus to the window that I'm sending the next command to. This does work, but because setting the foreground window takes a little bit of time you have to be careful to not have two commands run at the same time. I wasn't very careful about how I avoided those collisions here, so that's something you'll want to develop if you plan on using this method of quickly swapping between windows. The more copies of the game you're running, the more likely you are to run into these conflicts.
As a side note, if you're going to be swapping window focus anyway, you might find better performance using SendInput(), or even just PyAutoGUI or PyDirectInput as you normally would. The only benefit that SendMessage() or PostMessage() give us is that we can target a specific window handle with them.
So now I kinda have a solution for botting in multiple windows at once, but we still have the issue of wanting to send inputs to minimized or unfocused windows so that we can run our bots in the background.
I found that how SendMessage and PostMessage behave are very dependent on what program you're trying to automate. For example with Firefox, I was able to send inputs to Mario even when the browser was minimized or in the background. But this wasn't quite a perfect solution because Firefox was laggy compared to Chrome, and I couldn't get SendMessage to work with multiple tabs or multiple windows in Firefox unless I resorted back to our SetForegroundWindow method.
Continue reading here: learncodebygaming.com/blog/ho...
#programming #python

[@umapessoa6051]

Hi, you don't upload videos in a while, hope you're fine, your videos are pretty helpful, please if you're reading this, we need more content like yours, make more videos when you can.

[@pabloortega1569]

I came here to say almost the same. I hope you're doing fine. Your videos have been helping me a lot. You're amazing!!! Thanks a lot for all the videos you've made.

[@Kanjas19]

I'll leave another comment to support this. This channel is way too useful to learn anything. I hope someday you comeback man.

[@cesarpadilla8358]

cant let this comment die

[@SapienciaEstoica]

just notice he didn't upload anything in one year. I really hope hes okay, his channel is a hidden gem. He can explain everything so simple yet so full of useful information.

[@DanielXavierDosReis]

Came here too to show support to Ben! Please update us about you, hope you're doing fine

[@NobungaGames]

Please Don’t stop uploading!!! You have probably the most helpful AI videos I’ve found so far! Please keep going

[@jinnie7882]

I found your channel just recently and gotta say your little series on writing a bot got me back into coding and I'm having a lot of fun! :) So thank you kind sir

[@marceloviana7622]

hey bro, please dont quit, the most pleasure content of yours is how to make bots, so you should focus on that. Your audience will increase easily, focus on various types of games.

[@compinerd732]

the way you teach and show how you go about solving a problem and give us tools at hand and show us how you figure out stuff is exceptional. i thank you for all the work you've put in your videos so far.

[@henrypark2472]

Don’t know if you stopped making videos but, just found your channel and learnt so much useful stuff that I’ve always wanted to figure out! Thanks for the great info and I hope to see new videos from you in the future.

[@Keklmao]

I just wanted to come to your newest video to let you know i love the way you teach , and have been learning alot! thank you!

[@vitordomingos8569]

yoooooo, felt actually that we were " Learning" and exploring together XD amazing man,

[@kentalontaga6850]

Thank you!! ive been waiting for your uploads

[@ktc50222]

Bro your content is the best python content I have ever had the pleasure to watch. I have learned so much from you in like 4 videos. I know you haven't uploaded in a while but I hope you're doing well. All the best my man, cheers!

[@user-wo2sx3gi5z]

Hey, just want to say thank you for making these videos! They are so much helpful :)

[@LavonRich]

Subscribed to you some time ago. Never regretted it. Hope you are well.

[@359w]

Thanks a lot for all your videos. You have helped me a lot with deeper understanding of some of the subjects. THANKS A LOT!

[@Alexandurs]

This was very enlightening, thanks!

[@LearnCodeByGaming]

Thanks Alex! 😀

[@aarongill1075]

Subscribed when you said "I don't know lets figure it out." Thanks for these videos

[@juliopeguero5665]

lol same, really cool to see

[@nazalaMV]

Hi, I'm grateful for your video. It explained well the limitation usage using python that I'm also have the same issue as yours. I can't get around to make my python program to run through a minimized window. But anyway, thanks a lot for your explanation. I'm hoping to get new updates either from this channel or this comment section for how to using python and doing automated on a minimized apps.

[@Currychicken5]

Oh my god. This is what I needed THANKS 😀

[@dariusz4434]

This was really helpful, great work!! Hope you are doing well!:)

[@CommentFrom]

You got any plans for more videos? Really loved everything you've uploaded so far

[@jtvalente]

Great video, this helped me automate key presses in a ps4 game (using remote play linked with my computer)

[@espvolt4833]

abandoned looking for something like this for months this is sick

[@ByteBeacon9660]

I was trying to do this for a browser game that I was automating. In the middle of trying this I found a solution specific to browsers. You can use playwright or selenium to automate a browser. This includes sending mouse and keyboard inputs on specific elements and/or coordinates of canvases. These libraries include a screenshot function which you can pair up with opencv and do image recognition. This can all work in the background. The browser doesn't have to be maximized.

[@brandonesqueda4469]

Do you have an example? I just need to have 1 button be pressed in 2 browser's

[@Kanjas19]

this is really helpfully.thank you so much.

[@baquant]

I hate how I can recall those stack overflow pages.

[@LearnCodeByGaming]

lol

[@mzmahid1212]

is it possible to input mouse clicks too while the window is minimized or inactive.

[@VaticansAssasian]

@@mzmahid1212 yes there is an example at object detecting playlist

[@simonsmaster12]

Thanks for the informations. Now i can start my project.

[@secretarybailey770]

love your work bro haircut looking wicked too

[@thangdinh4838]

You have the most interesting coding instruction channel

[@lotto5883]

This is way too specific but its exactly what I'm looking for... Thanks for uploading

[@kex0]

Specific? The amount of people asking how to do this would suggest the opposite.

[@ethanstephanis4034]

Man I’m glad you made this video I would have been going down the rabbit hole trying to do this for hours lol. I instead spent the entire day trying to get modded minecraft to run in a VM and finally got it to run at around 10 FPS which is good enough for my code to do it’s thing

[@kan8602]

2b2t?

[@ethanstephanis4034]

@@kan8602 nah fishing in a pixelmon server. Ended up getting banned lol

[@OlofMemester]

Hey I know you probably won't see this but I am trying to write a bot for Minecraft as well and for some reason when I try to input the keys it doesn't release them. Did you encounter this issue and do you know how to fix it?

[@ethanstephanis4034]

@@OlofMemester dang I replied but for some reason it didn't go through. To sum up what I said in my previous reply that I didn't acutally post There's multiple ways to simulate key presses and I think I used pydirectinput for my script. You should be able to pip install pydirect input, import it, and put pydirectinput.press('esc') I can't check that I used pydriectinput or that that works so if for some reason it gives the same bug you could try pydirectinput.keydown('esc') pydirectinput.keyup('esc') kzbin.info/www/bejne/gneneJp8h9aJf9U&ab_channel=LearnCodeByGaming video on pydirectinput

[@gm4984]

I think that writeprocessmemory might just be the solution. In theory there's a 1byte or 4byte value that determines if the window is active or not. This value is then used to allow/disallow inputs, if the value is flipped to a 1 then it will start reading keyboard inputs. I worked a lot with assembly and many applications prevent custom inputs, but they do still allow modified Xinput inputs, and that can also be an alternative.

[@gerooqelvis3466]

could you elaborate more on this?

[@SkuldVi]

best code channel

[@cesarpadilla8358]

hey bro, hope you are ok pretty thankful with the vids you uploaded

[@gbitencourt]

Man, the level of this content is so high.

[@Linda-mj2cz]

Love the new hairstyle :$ Great video!

[@WholesomeContentForYou]

love the videos keep it up!

[@canal_conectando]

Where are you? Show up again! I love your videos! Your channel is one of the best around here.

[@HolyShitILovePie]

i want to soak up all of your knowledge!

[@Sockheadrps]

keep it up, im subbing!

[@zosyn320]

Hey man, hope you're doing well, love your videos, hope you come back and post some more ! :)

[@theonecallednick]

Where were you when I was struggling to do all of these? XD I guess i wasn't looking hard enough so that was my fault. Love the video though and learn a lot of things!

[@theonecallednick]

Well I'm still struggling to do it xD. I just did some workarounds accepting slme comprises lol

[@erykadolphson9491]

Thank you!!

[@snowmanstan3933]

You should do a series about ML playing a video game. That would be pretty dang cool

[@kyubi2uzumaki]

Personally I use both post/send message for my bot but I also have find out that Autoit is also really powerful and can do things that send and post message cannot. In the end I settled on using a mix of both of them for my bot by using the pyAutoit module. This results in my bot being blazing fast and consistent for multiple windows even when unfocused.

[@LearnCodeByGaming]

PyAutoIt looks cool

[@DaKrucial]

Yup I just pyAutoit. It was the quick and easiest way to solve this problem.

[@_claud]

Would you mind uploading it to Github or anywhere else to see how you did it? Would be interesting to see. I just ran my bots in VMs. Running them natively would be much better. xd

[@Hemoplaguer]

I would love some more information on how you accomplish this. I, as well as probably many others who came to this video, are still searching for a way to send input to inactive windows. Please get back to me if you care to elaborate. Discord: Chai#5765

[@nicholashammond6638]

@@DaKrucial It looks like autoit (and thus any bindings for it) uses PostMessage internally

[@guimauve522]

I would look into creating virtual keyboard and mouses. I know that there used to be one app on windows that was able to do just that but apparently it isn't working as good since some windows updates.. It was created so people would be able to connect multiple mouses to a pc so they could all work from the same machine; this would let you do most of what you want I believe

[@piorprogramadorbr9481]

Ppjoykey?

[@alamar11]

Thank you so much!

[@emiliomota2459]

dang you are a genius bro

[@dc22199x]

You need coroutine to do that rather than blocking each window, or create a json file where it contains the lists of PIDs used so you can run the main.py multiple times, thus you dont need to have that forloop

[@Vylerr]

i have ran into this problem too and have been on all of those stack overflow pages. damn

[@kosmonautofficial296]

hey man hope you are doing well I miss this amazing videos!

[@messto531]

he is dying Rest In peace i guess

[@nafisarkar3587]

Man you are awesome.

[@abdallahazme3737]

Excuse me Ben, is this mean you cannot use the bot you have build for Alibion online on more than window?

[@ebadaniba0]

Sir, do you have some content about identify a image in window and move to it using WASD? i saw you videos about opencv, but i cant find nothing about WASD moving to detected object.

[@s1dev]

man, you are just awesome. I am having trouble with pyautogui when importing from a different .py file and it just didn't work. also, I wanted to know if I could resize a window(resize x,y) and position it on my screen with code?

[@dharmatejasingampalli6480]

Is it possible to send multiple inputs of same to single application...like example I want to mimic ctrl+c on multiple text box in a desktop application at once or having tab key work on two or more text boxes at once....is there any approach or idea on this

[@Diduty]

really helpfull tks u alot

[@farfalleus8568]

Do you think you could use multithreading to press two keys at once?

[@CoderKid12]

Nice Content

[@pizza88]

Hi I used AutoHotkey which actually has send keys function, you may try that with python associated?

[@mzmahid1212]

is it possible to input mouse clicks too while the window is minimized or inactive.

[@iapeiro4077]

Is there a way to find a new opened window? For example, I want my bot to open 2 windows at a time for 30 minutes then open another 2, and windows will have the name (window1,window2,window3,etc)

[@schindlerjahudisi9837]

Thank youu

[@ineverchangemyplayericon3016]

This video reminded me of the pain in win32. I even made a SO post about windows messages and it was down voted by an ex MS employee because I was asking something that is not possible. Idk if Linux has a better window message architecture for user inputs. The answer is probably just use a better OS. But it may be possible to rewrite windows message architecture, but probably not worth it.

[@dharmatejasingampalli6480]

Hai..is there any possible ways to record keyboard strocks and repeat it when we call a method...

[@funktionone821]

We didn not see you since long time, I have a great video suggestion for you, make a video about how to connect a custom trained YoloV8 model to any game window. The current version is V8 which has so many options, for Detection ,Tracking, Segmentation and so on, now it is even easier deploying implementing for custom projects. Reply

[@7up340]

I'm not a python guy but this was interesting and entertaining for me

[@tonyr0xx]

do you now how to set workwindow by mouse position ? i need to send keys like 1, 2, 3, to a game.exe in minimized window, this window must be captured by a mouse positon, by PID and set PID to workwindow.

[@promisedid4571]

que daora, irei ver todos vídeos.

[@lyricsms5448]

I love this but I wanted to know something. So how easy is it for Anti-cheats to detect that your mouse clicks or mouse movement comes from a program or hardware? Like if I send the mouse movement command via program would that be easy for them to detect? Would it be harder to detect if its hardware?

[@wellhellothere1785]

I want to know the answer of this

[@XXnlcodXX]

They are just gonna see what inputs you did/do, not the source that is sending the inputs. So if you make the movement consistent , they will detect it fast. No way a human is following exact same pixels/pathing every time, or move crazy fast etc etc. The trick is getting it to randomise and act "human like".

[@tetteykn110]

im nob in Python and this stuff, can you create me 4 keyboard keys to send to game? for example i want to press "E then Z then Shift then Space" i want to create script like when i press F8 it sends that inputs when im playing a game.

[@richardking6980]

Out of interest, why can we not use pyautogui and just select the windows when broadcast the inputs?

[@poepew417]

i am a developer who uses MacOs. Which libraries should I choose for handling windows?

[@piorprogramadorbr9481]

Am trying to create a bot for a game, but I wanna do mouse clicks in a minimize window

[@taoufikidrissi835]

Would it be possible to make a series about Man In The Middle bots MITM in Python of course, seeing how to read the signals. hooking, sockets all that good stuff . Also I made a first bot using your series for another video game and IT WORKS WONDERS !

[@jjang_won]

Thank you for the tutorials! This is exactly what I looking for. However, I struggle with one more problem. Could you help me if you don’t mind? I want to assign hotkeys to start/pause/reload/exit macro in Python. The macro is infinite loop, like AutoHokey code below. F1:: Loop { DoSomething() } Return F2::Pause F3::Reload F4::ExitApp

[@bmfs348]

Not sure if you figured it out, but the easiest way is probably to have two different threads

[@w0lm7b97]

hope you are doing well wherever you are!

[@s_e_t_z3248]

Where have you been man? I love your videos, they got me into coding and I cannot thank you enough.

[@frazz2k4]

Bro when you coming back with more videos, we miss you!

[@alexsalgado-developeradvoc2886]

congrats!! Does it works for MacOS?

[@eternablue730]

you left eye didn't notice threading was the best option.

[@user-oz3oe6px8j]

any way to send mouse click? the video does explain for key and it works, but idk what for mouse click

[@sak.4001]

Ahk lib?

[@saw6053]

Hey man, how are you? Have you stopped making youtube content? Hope all is good with you. Cheers and thanks for the great content.

[@beeasy247]

Any solution for sending a mouse click to a specific part of the window (without taking the mouse or focus away from whatever the user is doing)

[@oma6393]

win32gui.PostMessage() will work,for keyboard you need send to pids

[@beeasy247]

Am I the only one that is seeing 2 replies for this comment but only seeing 1 when you click the drop down?

[@erfansn869]

Nice

[@lethanhphong7812]

8 months now still :( i hope you okay

[@foxpro9524]

Your fan from Vietnam. Hello, could you make a video about "Detect custom object using HOG cascade training". Thanks a lot.

[@Hemoplaguer]

Ah, so still no solution... I was really hopeful. I got excited when I read your title.

[@SixtySixVideo]

same

[@Erotukas]

Scripting program called uopilot as well as autoit can do it easily. Maybe Python is limited in that sense, or libraries needs to be created to make it work with unfocused windows

[@ryanthompson1326]

You could put the bot in it's own vm and have it view an rdp window from the host.

[@kumboi]

King shit ❤️

[@IARENOBODY]

Can you do a video on detecting sounds from a game?

[@00Jannis00]

This is exactly what I am looking for :) is this also possible for macOs?

[@Processador_DE_BaixaQualidade]

Hey i'm new here,i actually can't understand nothing that he's saying in the video,what i need to know to do the exactly thing that he did in the video?

[@fghtyfght2348]

Does anyone know the virtual key code of hash(#) ? (Ctrl + Alt + 3)

[@lordraphi]

Can you come back and show us some more cool stuff? Thank you :)

[@fyzbit]

I can finally make cane/wart script thanks!

[@pedrobizzotto556]

Does anyone know how I can simulate a mouse click drag and drop in a minimizer window?

[@oma6393]

you cannot. but try hack dirct3D lib

[@hakankosebas2085]

hey can you make serie with opencv self driving car project? but roads shouldn't have clear view, like mud roads in foxhole game.

[@alexanderher7692]

I used both sendmessage and postmessage... nothing happened lol. also tried your method. nothing... does windows version matter

[@ashersmith3466]

hope you're good

[@messto531]

he's dead i guess rest in peace

[@messto531]

Reast In peace

[@messto531]

you see he is depressing and the end of title is "Die Trying"

[@ashersmith3466]

@@messto531source?

[@Dannybeasted]

hey, can you do more javascript stuff? great content!

[@LearnCodeByGaming]

Thanks! Yeah I plan on doing more javascript stuff. Anything in particular you'd like to see?

[@Dannybeasted]

Learn Code By Gaming maybe how to work with APIs from game companies such as riot games and display the data somewhere

[@Dannybeasted]

Learn Code By Gaming or making a browser based game, spin the wheel or blackjack/ chess

[@AhmedMusrin]

@@LearnCodeByGaming I loved the runescape scripting series. Maybe something similiar:D

[@sciencekhuong]

if you can put bot and game run in raspberry pi or server, run 24/7 it will be very cool