The most important Python script I ever wrote

[@martyc5674]

Not Python- but I automated a task using VBA in the first few weeks of my current role that I have no doubt was the reason they couldn’t keep someone in my role for more than a few weeks before they ran for the hills. Still using it most days and it literally saves me 20 hours a week.

[@sultown4343]

What was the task?

[@martyc5674]

Taking MRP downloads from our ERP system - organizing them by part and creating nice legible worksheets for each part. It also pulls in relevant master data/specific order comments from the network into each sheet that are relevant for each part.

[@daves4026]

Yep VBA has served me like this for 25 years. Now starting to learn python but within the use cases for excel it’s hard to beat

[@Perovidence]

@@daves4026I think I heard that Microsoft was going to allow python within excel, like how VBA is now. I’m pretty sure office libre already allows it

[@Trevmasterfunk]

@@Perovidence I've done something similar at a job before. What was also nice is creating queries under the data tab. They make it so I could make automated worksheets for others and not have to worry about the macros messing up.

[@mgaugy]

"Automate Everything": 1. Automate task A 2. Automate task B 3. Automate execution for task automation of A and B 4. Find a way to automate the automation in step 3 5. Keep building the automation tree

[@wit4u]

No, not actually. Automate comes after having streamlined the process: kzbin.info/www/bejne/nJe2lWiNa5yZm68

[@CHURCHISAWESUM]

6. Accidentally develop artificial general intelligence

[@quintonrichards4805]

You may have just accidentally built the most important ChatGPT prompt in existence for IT. 😂

[@jeyemdeesee]

@@CHURCHISAWESUM😂

[@Gigusx]

That's how most clicker/incremental games work! First you click manually and get credits, then you buy something for those credits to click for you (at a slower pace), then you improve it to click a bit faster or "better", then you buy something that automatically upgrades the thing that clicks, then you buy something that automatically upgrades that thing, and so on until things go too crazy to read without scientific notation 😁

[@ankushgaur9367]

Same story 🙂 Python automation has also helmed me make new friends in every department in the company. Your channel is great. Thank you.

[@rollinOnCode]

how does it helm you? what does helm mean?

[@ComputersAndLife]

Likely meant helped and not helmed​@rollinOnCode

[@GrimChu69]

Yay, new friends to do work for haha

[@BreetaiZentradi]

At a job I had, one of the more important clerical staff lost 4 hours a week, comparing Folder A to Folder B, and Folder B to Folder C, and noting what had changed with what folders were added, or moved. I wrote a script that would daily create a directory listing, DIFF with yesterdays listing, produce a report of ADDs, MOVEs and DELETEs and email it to them daily, then make today's listing "yesterdays listing" for the next run. This saved them 4 wasted work weeks a year. In addition they had the changes daily now instead of weekly. Once that was available it turned out there were 7 or 8 other people who benefited from this information. I just changed the program to email them all. It is amazing how one little script can save so much time.

[@valknut9648]

Never give away that script "for free" at any workplace.

[@gh05tparkourfreerunning31]

@@valknut9648 Any programs made or used on company time or on company devices is technically owned by the company. At least, most employer contracts include that detail.

[@andercorujao]

@@gh05tparkourfreerunning31 thats why its better to do it for yourself and not tell to anyone, use the gained free time to learn more or do personal projects only if you have personal friendship with your boss you can gain something from letting this kind of thing benefit your work, otherwise you'll only get pressured to work more

[@Vampirat3]

Uh huh , and what business did you do that for are they taking care of you now? Mmmmhmmm. And how much did they pay you ,and , are you helping other people learn ? If your answers feel off , consider doing business somewhere else.

[@BreetaiZentradi]

@@Vampirat3 I had a job that was "convenient" for me. Almost totally unsupervised. I was able to set my own schedule. Own I lived 3 minutes from work. Lived in a very very small community and there was not much for IT opportunities in the community. I was happy there. The pay was ok, and not great. Yes. I did mentor someone else, taught them how to code. Taught them a bunch of best practices. I even donated my lunch hour to anyone interested in learning anything IT releated.

[@alishagrace]

Python is awesome. I automated a task with Python that was repetitive, that took about an hour every time. Now instead of me creating multiple CSV files from one large file with Excel manually (Excel struggles with a lot of data), Python does it with the run of a command in just a second.

[@capability-snob]

input() is a nice alternative to sleep you may like - it waits until you press enter in the shell, so you can wait as little or as much as you like.

[@MustafaAli-ve1vm]

Same experience, I used to work as a Data Entry Officer at an e-commerce company (nothing fancy here) but I can't stress this enough, 60% to 70% even till 80% of my tasks can be automated from changing product price, disabling it, adding promoted price for certain duration, changing its status to in/out of stock, changing product's description with a pre-defined description based on a given google sheet. Imagine doing this manually for 100+ of products, no thanks! I even helped our brand managers by fetching 100+ of production information from other vendors and competitors.

[@BanAaron]

During COVID my usual work wasn't required any more but my job didn't want to just let go on furlough (grrrr) so I was giving the task of RDPing onto every server we had to copy files from a folder onto the main server for processing. We had many many servers that were incredibly slow so this would take 4 hours a day of mind numbing boringness I completely automated this task with Python and it took 10ish minutes to run. I even went further and used multiprocessing to kick off the moves on each server concurrently which took it down to 2~3 minutes

[@deeperblue77]

Nice one and fascinating. While I was working on a tedious, repetitive task, I started looking into Python. I had a little experience with C#, but it just did not feel right, and Python was simple to adapt. I've never been regretting my Python journey since.

[@JohnWatsonRooney]

It really is great for automation tasks

[@macgyverswissarmykni]

Step 1: Automate your job Step 2: Tell NOBODY, EVER When someone knows that your work can be so easily scripted and automated away, your value only goes down and not up.

[@88onage]

Same here 👏 In my case I had to copy-paste social media data into a clunky spreadsheet. I started with some simple Python scripts to automate whatever I could, ended up in love with programming and am currently building large-scale analytics apps for my company. Your channel has been huge help throughout the years! Thank you 🙏

[@JohnWatsonRooney]

Thank you for watching your very kind

[@sharkysharkerson]

Companies underestimate the power of tools and scripts when they look at productivity. Features and bugs are given priority whereas they tend to kick the can when you want to schedule time for tools, and the only time they get written is when someone carves out their own time to work on them while also hitting their other milestones. At the same time, tools are the things that can realistically achieve 10x or higher magnitudes of performance improvements, if you consider the amount of time everyone has to spend doing things manually all the time across the company.

[@ff0x]

Well, that was such a cool and informative video that I had to leave a comment to help your KZbin algorithm. Well done :)

[@JohnWatsonRooney]

Thanks for that!

[@BrownStain_Silver]

I'm just finishing up Automate the Boring Stuff now. I've been writing some programs for work as well. I'm a few months in and it's a lot of fun. So far i've automated a few work processes. The most meaningful one is reduce a potentially 30+ hour task to hitting run on a program, providing 3 user response, and a 2-5 minute wait.

[@adrianowallin]

beyond incredible. I hope they give you a huge raise haha...

[@MichaelWyattMDW]

I DIDN'T KNOW YOU COULD DO THIS!!! I mean... now that I think about it, I knew there were -programs- that did this... like Q&A testing type packages. This looks relatively simple, though!!! I had no idea! This is great! I also have a weekly thing I have to do for work that is tedious, and I've been thinking it could be scripted... but just assumed that would be way over my head. This will definitely help me get started! Thank you so much for posting this! (So glad it shows up on my feed!)

[@thomaspetermygind8022]

Came for the hope of automating parts of my work. Stayed for the CS talk!

[@immrsv]

Laughing hard at CloudFlare's "verifying you are human" passing the script through 🤣🤣

[@JohnWatsonRooney]

Haha

[@ohokcool]

I did the same thing but with Selenium for python, ultimately I wasn’t allowed to use it departmentally because selenium was not in the approved software for the company but it sure made my job a lot easier

[@JohnWatsonRooney]

I was fortunate enough that I was allowed to and trusted too I know not everyone will be though

[@shaftymaze]

You changed my life. Only your videos. No joke.

[@shaftymaze]

I mean there was a lot of grind. Except these videos made it possible.

[@divyanshugogna6152]

Thank you so much for sharing your experiences and story John. In my case main python use in my life was also webscrapping from Amazon to find deals. I have improved a lot in my code and generally over time with lots of learning sources Would appreciate some content about automation of these python scripts please ? And may be dynamic content too (where one code runs other code, or passes data to other code at a dynamic time)

[@eziola]

Incredible! Would you consider keep making Playwright videos? I think there's a lot of demand for scraping AI web search (particularly Google Gemini responses).

[@JohnWatsonRooney]

Sure, I can

[@Ronypls2555]

Nice, video. I need to start looking into automating more things before I get automated.

[@JohnWatsonRooney]

Always be one step ahead hah

[@dmcf1975]

Great example and helps inspire ideas for me.

[@keagankemp6275]

Aah once again you have outdone yourself, thanks I was struggling with tho for a while now, didn’t know how to get rid of the second pop up thanks a mil now off to code whoop whoop

[@jasonjohnson3424]

I want to know how you utilize the time efficiencies gained? 1) Do you run that task and extend your coffee break +X hrs? 2) Do you work remotely and bill the client for X hrs when you worked Y mins? Or 3) do you let your manager know how efficient you've become with the task to get more $$ or promotion? 😄 I've def done a lot of option 1&2. Option 3 rarely seems to work out the way I'd like, in my experience.

[@JohnWatsonRooney]

Learn how to automate more and more!

[@kyron5574]

When worlds collide lol nice to see a Prime nugget mentioned. Great video man!

[@nmsv1]

Been watching your videos for a while. Very informative and easy to understand for a newbie like me. Self taught programmer?

[@JohnWatsonRooney]

Hey - thanks for watching, appreciate it! Yes I am self taught!

[@nmsv1]

@@JohnWatsonRooney great 🙂 background in computer science or something else totally different?

[@dosomething6975]

Web scrapping is such a powerful tool

[@alexkaspi]

The CS references really hit home for me. 🤣🤣🤣🤣 great video!

[@saboo480]

Same story for me too :) well done staying with it!

[@andreacota2098]

Nice video as always. It would be cool to also see a video about solving captcha!

[@SaikiranJetti]

You're moving through the steps so fast. It will help newbies follow if you pause at the right moments.

[@Cra1gst]

I did dispatch for breakdowns and I would have to edit all live jobs to our format and our codes from companies codes , I used script and read text and edit all the job sheet in 1 click 5 seconds, 1 year tho but 5 min per job editing a job 30-50 a day while having to do so much , even today I'm now using AI chat bot to do all my paperwork :) so far AI can take any job sheet and put in to a sheet as a data base and no AI can see all jobs I've ever done , love automation

[@larz4437]

Rooney's a CS enthusiast too! LFG!

[@ryanrhind5789]

Can you do a video on how you decide what selectors you use in playwright - get_by_placeholder? get_by_role? query_selector? xpath?

[@divyanshugogna6152]

Yea I had a lot of problems in them too, where like one goes wrong and code then needs to be changed again. I know in selenium we could just put literally all of them inside of a try: except Exception as E: pass Method and that way we can have it try it all in selenium, by xpath, by ID etc. Didn't really know how to do similarly with playwright but I think there might be something similar and that way we can put them all in some sort of try, except blocks ?? Although I don't really know if it works with playwright or not (try, except)

[@ea3gug]

Superb, wouldn't be even better if you start with "playwright codegen" and tune it afterwards?

[@JohnWatsonRooney]

Yes that does help - i did that in my other automation video but in this I wanted to build it up myself and show it that way

[@polestarpilatestrainingcou1892]

Love playwright codegen!! Only limitation..... I want to use a Chrome Extension and have playwright activate buttons within the extension..... Unfortunately playwright uses Incognito, for which you cannot use extensions.... UNLESS SOMEONE OUT THERE HAS THE ANSWER TO THIS?? 😱

[@Sfeclicel]

Almost every data from a website comes from a server/db if this is the case you can use the endpoints your browser calls when you click on things (dev tools, network tab) and like this you dont have to worry when ui changes

[@donsurlylyte]

that is awesome. filed for ref.

[@adventurefilms]

What resource did you come across at 1:03? Apologies, the title and author all ran together.

[@varaprasadreddy-j6z]

Automate the boring stuff with python - Al Sweigart, there is a udemy course on the same content from the author as well.

[@markuswierer1755]

As a person who doesn't understand much about coding: is having your username and password openly in a script the norm? It kinda... "feels" wrong... I assume you could make those two things an interactive prompt to type in everytime you start the program instead of having it written in the code permanently. But that won't help much if it is scheduled to run automatically at certain times without user interaction... Again, I don't understand much about programming, so I'm curious if that is a security concern in the "real world"?

[@JohnWatsonRooney]

You’re right, it isn’t. It’s a security risk. But I wanted to keep it as simple as possible here. I would store the user and pass in environment variables or similar. But as I designed this script to only run locally with me there I didn’t worry about it

[@markuswierer1755]

@@JohnWatsonRooney Thank you very much for your reply! Good to know! Then this is something I have to look further into, should it come to a practical application for me. But at least my "feeling" wasn't wrong. That's something, I guess. Keep up the good work, I enjoyed the video. :)

[@alexdnaz]

@@JohnWatsonRooneyhow do I save the user name and password and call on it when you need it I.e when u store it in its own environment etc….

[@harshnj]

Did this stuff a lot, literally spent 1 day of making this kind of automation for a task that I could easily do in 5 hours. But doing that was boring, and I enjoy writing code. The another time I was doing that task, I was much faster than anyone 😂

[@gamalalejandroabdulsalam904]

Helix editor master race

[@SajjadHaider1]

Which window manager/desktop environment/distro are you using? I find toggling and managing windows a major hassle. I want what you got there! It's super smooth!

[@JohnWatsonRooney]

I use Linux and i3wm to manage it, I’ll never go back to a standard desktop!

[@Cluni-80s]

Why can't I shake the feeling I have played witrh you on a UK server many years ago 1.6 TAT...... Great video, gg)

[@JohnWatsonRooney]

i played a load of 1.6 in the early 2000s

[@craiggazimbi]

Hey man thank you so much. So I would love to actually be able to automate stuff as you do without necessarily following tutorials ,could you share your thought process or perhaps make a video on how one can become good as you. I do understand Pythona and coding in general. I am interested in expanding my knowledge within automation scripts. Any pointers?

[@drewsepeczi]

Read the book “Automate the Boring Stuff with Python” or look into the course “Automate everything with Python”, do not get caught up in watching short tutorials like this, although this video was very informative, you need to learn how to learn yourself. Cheers mate

[@bryce3851]

Dude ngl i immediately click on your videos it has help me learning programs that help me with work thank you i appreciate your videos

[@JohnWatsonRooney]

Thanks that’s very kind

[@buzzdev]

Python is great, and we also use opencart. However, if you obviously have access to OC admin, why wouldn’t you write that whole stuff in PHP, directly accessing the OC classes and functions and read that stuff and much more from the database?

[@sgntsids]

Because php sucks and he also does scraping of external resources?

[@guitarchitectural]

I had chatgpt write me a python script that interfaces with Google's groups and sheets API, saving me countless hours and headaches. I don't know the first thing about code or API work so it actually feels like magic 😂

[@KyleLanmon]

Love these quick and dirty scripts!

[@dj0men666]

Does the system not have an export or an api? Would be much easier than hacking around the front end. If there was no export then i can see its use - or even running things in the UI - ive used test software to automate things for me before :) Its great seeing the computer do what you woudl be doing normally while you get on with something else more productive.

[@arcticslay6012]

May I ask why you chose this method over the playwright codegen method you used in your past vid?

[@JohnWatsonRooney]

Sometimes codegen doesn’t pick the best selectors so I often do it myself

[@karthikb.s.k.4486]

Nice . Is there any tutorial on Installtion of neovim and Installtion of python libraries please let me know

[@lauriran]

sudo apt install neovim

[@Betttcpp]

0:33 hooked

[@Mousoukyou]

Very good info, thanks!

[@xXxRK0xXx]

Even better? An exposed api endpoint to grab the order ids. Simple, tiny amount of code, 100% data integrity and instant.

[@johanlarsson9805]

In 2009 I automated half the workday for 24 people, since it was mindnumbing and I couldnt bare doing it anymore. The company fired me.... so i figured I try to start a company doing what I had just done, automate office work conducted by buisiness. I tried to get some government aid and sponsor money. During the evalation I did not get any money since they did not believe in the ide of automating office work was possible. Yeah, I am always about 20 years too early.

[@RafaelConceicao-wz5ko]

I've just been doing the same thing, and thinking about starting a company doing just that. Would you say now is a better time to do such a thing?

[@johanlarsson9805]

@@RafaelConceicao-wz5ko Now people would not doubt that it could be done, atleast. There are many players on the market, all the big consulting companies are pushing to sell their automation services too.

[@101excelhacks8]

You can give it a shot now: call yourself an Automation Specialist or something and tell companies you can save them thousands of dollars in operating costs by automating their workflows. Report Creation workflows are ripe for this kind of stuff

[@johanlarsson9805]

@@101excelhacks8 Yeah, I am actually working for a large tech company for the last 6 years and it has become a popular thing to sell to other buisines now. Reports, workflow automation, chatbots etc... everything I wanted to try 15 years ago.

[@joshuadias2468]

Fantastic video

[@SolidBuildersInc]

1st off, I guess you didn't realize you could have done all of what's in your demonstration by just saying, "Abracadabra, use Playwright and duplicate what you see me do", and Shazam !!! it would have been done. Wait, sorry..... That would be called magic and your presentation is real. My Bag, 🤣🤣🤣🤣🤣🤣🤣🤣 Congratulations, you have made my subscribe list and I seriously must thank you for sharing. I have tried Selenium before and it was way too quirky for me, and maybe Playwright is too, but you have managed to get my attention to give it try with the setup you are using less the VSCode ide. Thanks Again.......

[@StormWolf01]

The video was pleasant, but the title didn't give any info on what the video was about.

[@diego750]

You appear to have reading comprehension issues. Seek help

[@MichaKuczma]

Hi John, what do you use for such a smooth windows management?

[@dfcastro]

I do some automations of some activities that are repetitive and boring. On my case I used a bash script.

[@njl1282]

I did just this and now work remotely making six figures at the same company

[@JoshPeterson]

What colorscheme are you using? It's gorgeous.

[@JohnWatsonRooney]

It’s kaolin-dark, my new favourite

[@JoshPeterson]

@@JohnWatsonRooney cool, thank you. I'm excited to try it out. Right now, I'm using Nord for everything, and I really like it. But it's time to switch things up

[@tastycles1062]

Very good!

[@joachimcronquist]

Great tutorial, thanks!

[@Mrcaffinebean]

Really funny the company I work for uses the exact same angular admin theme. I think it’s called Color Admin. The only thing to note is that they might have an API that you can access that make your automation more stable by taking the UI out of it.

[@thecozyplace1206]

Unfortunately, what I understand from this is that it only works for browsers... The company I worked at had some sort of CRM, very old, very laggy at every button press. With tons of pages and small fields to fill up. I REALLY wish I could find a way to automate the data entry of that one... It would replace all backoffices data entry people of italy xD

[@JohnWatsonRooney]

You could try pyautogui- not sure how reliable it would be for you but might help in some way?

[@cchgray]

Where / how does original orders - “for order in orders:” get populated before you populate via csv reader?

[@lucaskellerlive]

Would you be available to meet on a call via Zoom at your hourly rate for a period of time you'd be available. I have a question regarding scraping Python API and I've watched all your videos that gave me an excellent overview I just have a few specific questions and would greatly appreciate your time helping me solve these couple of issues. Thank you!

[@bokaroid1946]

I'm a newbie data entry, I want to automate data processing and I need to start from the last page of the data and then export it to excel. Can I do that using this technique? thank you!

[@Sam-lj9vj]

Might be a silly question, but in this particular example you selected the particular "Orders", e.g. 3000, 3001, 3001. Now imagine it starts at 3000, however it keeps on going and you don't know when it stops. What would you write in the code?

[@fxhp1]

that's called a while loop with a variable incrementing until an error is reached

[@billwilkin2996]

Thanks for your video. I'm following along and getting stuck at 8:52 with `page.get_by_role("button", name="Login").click()`. Has this happened for anyone else? Any thoughts on how to troubleshoot this issue? Thanks for any assistance you can provide.

[@babaidebnath1]

I think to have much more control if I would have used browser extension (self coded) to do same automation, instead of selenium or any other 3rd party

[@egonkirchof]

I know I don´t know quantum mechanics. I know I don´t know how to play tennis.

[@moloxiao]

if you do this, all the difficult part is about cloudflare

[@deeneyugn4824]

All my scripts are in bash employing mostly awk, grep, sed, wc, find, expect, xargs, psql or sqlplus,

[@paladine4life]

Hey John, why you are never at your computer? Because scripts! Ok, here is more work then.

[@mcdinkysgarage4514]

The answer to "why aren't you ever at your desk" is always just one word: bowels.

[@patrickharilantoraherinjat2994]

How do you manage captcha ?

[@doleo_metal]

Can you help me write a python script that will make a ppt that describes all the steps and processes for the SDLC of a team of 100 IT professionals? Thanks

[@FireFly969]

Yes its something great, and feels like you are so developed 😂😂, when you see someone do something that takes like hours while you can automate it, and let computer do it, while you are saving your energy and time to important things

[@technicaltrialsandtribulat4574]

looks good / useful, but does this shopping cart software have a python API available, that would be easier to pull data from rather than resort to screen scraping.

[@HumanityAsCode]

This is awesome but if you have access to APIs rather than UIs it's a lot more efficient to do it without any UI involved at all. Ideally software developers would be involved in helping eliminate so much busy work in the first place. I try to be helpful to other departments in that way where some small changes to a tool someone uses every day can make all the difference.

[@christcombiccombichrist2651]

hello, I have watch several of your webscraping videos. those demo was straight foward. I dable a bit in python, nothing really to talk about on that. but could you just give a demo on: lets say I wanted to scrape a header and the associated images of that header. the headers by using some keyword from the website that has this as their headline : 'samsung new galaxy (some fancy name ) is a market hit says CEO. and the image have some text that is attatched to it that states: source getty images or somthing like that. to get those header and images returned. Can I just give all the header and image text as a keyword so beautiful soup could parse and ger the url for the image and assiciated header or header with assiciated image url then return the valueof the image and header. why I ask this?. because I don't want to be looking for those element manually (google inspect tool) I just need to add keywords and hit run then the code automatically find those and return them.

[@jpsl5281]

why 20 minutes ?

[@_Pymander]

Subd 🤙🏻

[@Jisamaniac]

I understand code. But I can't code for crap. The power Chat GPT and Claude AI, I can get by with the basics to tm do the basics.

[@exkalybur_dev]

nice

[@DavidLee-jd5ot]

Quick question, dose opencart allow for these scraping activities in their policy or they can band the IP, i know we are using playwrite with sleep time to mimic human but good to know if they anti-scrape bot or not

[@kevinfoendoe6987]

UiPath would be better fot this task

[@DominicFlynn]

Try it again with Puppeteer

[@bakasenpaidesu]

Nande?

[@satishm5260]

1st like

[@JohnWatsonRooney]

🎉

[@bryce3851]

Beat me to it .

[@bakasenpaidesu]

6th 😂

[@illyloko]

Thanks this really gives a great outline to pull and analyze more efficiently. Exactly the type of thing I needed to see to get me back on the keys 🫡