STOP Using Plain Python Scripts! Do this instead (5 reasons)

Sign up for the totally free tier of Prefect Cloud here: prefec.tv/doug-mercer
-----------------------------------------
Sign up for 1-on-1 coaching at dougmercer.dev
-----------------------------------------
One of the most frustrating parts of the workday is doing something that you know could be automated, but just… isn’t yet.
In this video, we use Prefect to schedule a Python script to run every week. After, we find that scheduling was really only the first of five problems, and we use Prefect to solve all of them.
You'll learn how just one Python library, Prefect, and a few minor changes to your code can allow you to:
- Schedule your Python Script to run on local or remote systems
- Handle Errors with Retries
- Monitor your workflows in an incredibly powerful web-based UI
- Parallelize and scale your code's using on-prem compute clusters or Cloud platforms
- Persist Python results and Markdown reports
- and more...
Prefect changed the way that our data science and machine learning focused research team approached scripting.
Chapters
---------------
00:00 Intro
01:38 I want to take Monday off
03:28 I don’t want to babysit code
06:35 No one knows what’s going on
10:33 My code is embarrassingly slow
12:41 I keep throwing away our data
13:47 Wrap-Up
Special thanks to Siara M. for making artwork used in the intro sequence.
#python #prefect #scheduling #scripting #mlops

[@dougmercer]

Sign up for the totally free tier of Prefect Cloud here: prefec.tv/doug-mercer

[@LeveragedAlpha]

Could I say this video changed my life? I have gone on a prefect binge. And wow. Already put it to use, and wished I had it all these years instead of cron jobs.

[@dougmercer]

That's so awesome! It was a game changer for us to. Thanks so much for sharing-- your comment made my morning =]

[@LeveragedAlpha]

@@dougmercer No - thank you, Doug! So glad I had the privilege of discovering your youtube channel. It's already my favorite technical channels (and among my favorites, overall). You go deep into providing actionable advice for solving problems I face, with incredibly high-quality content, but without the puffery and cringe thumbnails etc. It has already helped me improve as a developer, and I look forward to continuing by watching (and implementing) more videos.

[@khawarshehzad487]

Appreciate the effort you put in your videos. Thanks for sharing. keep it up!

[@dougmercer]

Thanks so much! Will do =]

[@adamgkruger]

Such a good video! TY for the content. I would 100% take a more indepth course on using Prefect.

[@dougmercer]

Thanks Adam! I may do another Prefect video in the future, but in the meantime you can maybe check out prefect.io for their blog/docs or their KZbin channel @PrefectIO. They have a ton of cool integrations with other software that I barely scratched the surface of

[@MaxShapira2real]

Thank you for the sublime content! I'm curious about the font in your text editor (1:57).

[@dougmercer]

Hey Max! That's Anonymous Pro. It's available for free on Google Fonts, fonts.google.com/specimen/Anonymous+Pro

[@LucasVianaCodes]

Your videos are incredible! Would you share your editing stack? Cheers!

[@dougmercer]

Hey Lucas! Thanks so much =] I currently edit on... Davinci Resolve 18.1 Krokodove plugin for Fusion (which is preventing me from updating to latest Davinci Resolve, because BlackMagic is dragging their feet on releasing an updated SDK 😡) A `pygments` formatter I wrote for converting code text to formatted text plus node for my code, which I then animate in Fusion in Resolve. (Which is not automatic, and a total pain in the butt) Epidemic sound for music Storyblocks/envato for other stock media In the future, I'd like to better automate my code animations. Previously I have dumped a ton of hours into trying to make a `manim` (python animation library) extension for animating the difference in code, but manim's Code mobject is really poorly implemented and requires too many workarounds to deal with. I've considered using reveal.js for code animations, but I think the result would be worse for what I do.

[@dinoscheidt]

Looks like a python only alternative to Kestra - and still no sight of the software engineering practice to maintain build and ETL pipelines next to code in a CICD fashion… 😮‍💨

[@maurolimaok]

Nice channel. Hope it grows.

[@dougmercer]

Thanks Mauro! I hope so too🤞

[@glaucojunior]

I used to run an Apache Airflow installation on an EC2 from AWS, to me its look a lot of an Airflow alternative, is that correct? Can you tell about advantages (and disadvantages) between they, if you know of course?

[@dougmercer]

Sure! Quick caveat-- I've only used Airflow a bit, but I was so frustrated with the on-boarding and user experience that I gave up on it and have used Prefect ever since. My take is that Prefect is a more Python-native library. Writing Prefect flows/tasks feels a lot more like just writing Python than Airflow, which feels like I'm writing config files with a thin Python wrapper. There's actually quite a few breakdowns comparing the two on the web. - www.prefect.io/prefect-vs-airflow - neptune.ai/blog/argo-vs-airflow-vs-prefect-differences Beyond that, Prefect's blog has an interesting deep dive point out some frustrating "Airflow-isms" and explaining how Airflow can break down at scale-- www.prefect.io/blog/implications-of-scaling-airflow Some pros/cons that were most relevant to me... Prefect Pros: - more modern, user friendly API - really easy to get started (pip install one library, add some decorators, run a terminal command to get a server running for monitoring) - supports dynamic workflows (rather just static DAGs) Prefect Cons: - Some really cool features are Prefect Cloud exclusive. Airflow Pros: - Oldest workflow orchestration out there, so wide user community/documentation (i.e. Mature). Airflow Cons: - I hate their API. - Static DAGs

[@glaucojunior]

@@dougmercer wow, thank so much for the information, just knwoed your channel and loved it, cheers from Brazil!

[@dougmercer]

Cheers!

[@fffklan3986]

pros of airflow: - Mature, tried and tested, runs on 100% open source tech. - multiple ways of writing DAGS (they have a newer taskflow api) - If you're ever looking into using it in a production environment, entry level config of AWS managed airflow can be as little as $400/mo. - do cool stuff, learn data engineering, fun fun fun cons: - if you don't know linux/docker well, good luck. - xcom can be a painpoint (limits the amount of data that can be passed between tasks, meaning that large amounts of data need to be stored between tasks). - you have to use your brainpower and thoughts to accomplish things. pros of prefect: - easy cons: - paid version starts at $1,850. - can't really put it on a resume as a skill (if ur trying to get a job) - sponsoring youtubers for publicity in conclusion: if u want something done quick, use plain old cron. if you want a trusted, robust orchestration system, use airflow.

[@martimlobao]

I’ve used both Airflow and Prefect for work, and can say that setting up and running Airflow is a million times more painful that Prefect. OTOH, Prefect can become extremely expensive with even just a few users, whereas Airflow is dirt cheap, which is why we ended up switching from Prefect to Airflow 😢

[@dirtdart81]

This looks really exciting, doesnt seem to do much new, bit really easy to use and all in one place. I can think of some places to use it at work already and am definitely keeping prefect in mind. Ignore the trolls! 😂

[@dougmercer]

Hah! thanks 😂

[@MrApresmoiledeluge]

Fantastic stuff!

[@dougmercer]

Thanks! I appreciate it =]

[@alexandreramos8484]

The combo prefect+dbt is amazing

[@robosergTV]

meh, there are alternatives like Flyte, Vertex AI, Kubeflow, etc. Or one can use Kedro to write the pipeline and export it to run in the cloud

[@dougmercer]

Oh neat, I'll have to look into them a bit more. I've seen kubeflow but not the other three

[@BenHammond]

…tech people are weird in their level of reflex dismissiveness (I say this as a tech person).

[@FadkinsDiet]

How is this different from Apache Airflow?

[@dougmercer]

I left a pretty detailed reply on @glaucojunior's comment, so maybe check that out High level gist is-- Prefect is a bit more modern, feels like you're writing Python, and has more dynamic features. Airflow is, uh, mature, but feels like you're basically just writing config files laundered. Here's a third party comparison of the two neptune.ai/blog/argo-vs-airflow-vs-prefect-differences

[@famed3777]

This is how promotional videos should look like!

[@dougmercer]

Thanks! It helps that I had already used their library for years, so I knew what I wanted to make before they reached out.

[@adamgkruger]

Thanks!

[@dougmercer]

Wow! Thanks Adam, you made my whole week =]

[@doofyguck6186]

Airflow user here 😂

[@dougmercer]

Airflow is totally fine 😂! It's been the standard for a while for a reason... That said, when I tried to use it, I felt like I was basically creating a config file through Python kwargs instead of writing Python code. Wasn't the right choice for me

[@doofyguck6186]

@@dougmercer the way I tend to use Airflow (due to my company's architecture) is composing and extending the operators to fit our needs. So in our main airflow DAG file it's simply a config file but it uses our custom made extended operators. That's where we write most of our logic.

[@dearheart2]

Nice video, just wish the music was mute/gone.

[@dougmercer]

Sorry about that. I saw you posted a similar comment on several videos. I agree that it'd be awesome if KZbin supported multichannel audio that allowed the viewer to entirely mute music. In my past videos, I've definitely mixed the audio too loud. I really tried to keep it quiet in the 1brc, and will continue to be more mindful about future videos. However, I probably will never have a video without music. In my personal opinion, music has the potential to make videos more engaging. Admittedly, sometimes my mixing and song selection misses the mark. So, thanks for at least attempting to watch my videos, but sorry about the VO/music mix

[@rafaeldbx]

how do I make my application more reliable? rely on some dumb cloud ai startup. these all feel like already solved problems, this is just a new set of tooling the developer has to learn, except this time there's probably next to no documentation

[@dougmercer]

Prefect's open source library is useful in and of itself-- no cloud required. Also, their documentation is good ¯\_(ツ)_/¯

[@mohak9102]

How is this better than airflow?

[@dougmercer]

"better" is probably a matter of taste. I prefer it because it feels more like writing Python and less like writing a config file. There are several comparisons out there that dive into the differences. Here's a third party link comparing them + another approach (Argo) neptune.ai/blog/argo-vs-airflow-vs-prefect-differences

[@ardenthebibliophile]

Novice to automation workflows: why not jenkins?

[@dougmercer]

Typically Jenkins is used for managing CI/CD pipelines (automating build, test, deployment). Though, I suppose you could use it for data pipelining tasks 🤔 Prefect is a bit more specifically tailored for data engineering, ML ops, or just generally writing pipelines with Python Code (and would not be well suited for administering CI/CD pipelines out of the box). To my mind, similar technologies to Jenkins include GitHub Actions, Gitlab CI, or bamboo. Similar technologies to Prefect include Airflow, Argo, Luigi, Dagster.

[@ardenthebibliophile]

@@dougmercer appreciate it. The company I work for, while very large, is relatively new to the data engineering space. We've had to learn a lot as regular engineers and scientists as we had a bit of huxterism a couple years ago where worlds were promised and delivery was thin (at best). Add in our technology groups are limited in what is approved for use and it's a complicated scenario that Jenkins, somehow, made it through. Will check out airflow and Argo too!

[@dougmercer]

My day job sounds really similar to yours. Hopefully you can find something that's approved!

[@ardenthebibliophile]

@@dougmercer pharmaceutical industry is what I work in

[@dougmercer]

I work with the government. Getting software approved is a whole process...

[@marlan__]

Why not just use Cron?

[@dougmercer]

Good question! Basically-- scheduling is only the first of five problems (and is the only problem that CRON would solve). Later in the video, you'll see why. Our code had flaky upstream processes that needed retries, our team had no idea what was going on mid-script run (where the real script took multiple days), we weren't scaling/distributing work effectively, and we didn't have a good system for keeping track of our pipeline's results. CRON alone wouldn't be enough without writing our own... well.. workflow orchestration system. Prefect gave me a ton of features out of the box, and I was able to turn around a refactored script in an afternoon that blew my team's minds.

[@dougmercer]

@ordinarygg As someone who has written their own general purpose pipelining library (because I wanted to) and multiple web dashboards (using Flask, FastAPI, Streamlit, Panel, etc.), I can promise you that it's way easier to pip install a dependency, add a few decorators, and run a few terminal commands than it is to replicate all the useful functionality that Prefect provides. Check out my comment I left above in response to @marlan__ to get a better idea where I'm coming from.

[@Redditard]

Me: This all can be fixed using cron and internal error handling.

[@AniaKovas]

Because it's about more than that, Cron is a crude tool anyway, but also you use the tools you know and are willing to learn. He's using Cron anyhow, and there's, just like I said, more going on. That's the point. I teach this stuff, this is a good lesson. Didn't know about prefect, and this is a good place for my students to look.

[@danielot]

Talented but definitely need a senior engineer or a tech lead. There are a lot of ways to solve all of these problems (khm airflow, mlflow😂) Was this a real situation or just made up to help illustrate why this is in fact a great library?

[@dougmercer]

Yeah, there are a lot of ways to solve these sort of problems. We chose this and it worked well for us ¯\_(ツ)_/¯

[@damonguzman]

This feels like a 15 minute ad. I dislike this video greatly.

[@dougmercer]

Sorry you feel that way... Although the video is sponsored, this is a true story from my day job, and the script doesn't make any claims I don't stand by. That said, feedback taken. Thanks for commenting.

[@kc12394]

Dude he literally said it's sponsored in the first 90 seconds of this video and you're still complaining it "feels" like an ad. No one is forcing you to watch a video. You can turn off the video as soon as you hear that it's sponsored you know...