How do software systems achieve zero downtime on new deploys?

📘 T3 Stack Tutorial: 1017897100294....
🤖 SaaS I'm Building: www.icongenera...
✂️ Background Cutter: www.background...
💬 Discord: / discord
🔔 Newsletter: newsletter.web...
📁 GitHub: github.com/web...
📺 Twitch: / webdevcody
🤖 Website: webdevcody.com
🐦 Twitter: / webdevcody

[@Flash136]

I'm really liking these topics that are more focused on software architecture. I see very few videos on these on KZbin.

[@catbroh]

Me too, this stuff is invaluable

[@xxri552]

same

[@MrShaheer]

the channel we didnt deserve but desperately needed.

[@reubendrummond]

I've been loving these software architecture videos lately. I would love to see something about database migrations and how they fit into the different git branching approaches.

[@mcdonaldslover52]

im not a dev but a business analyst, these videos are really helping me better understand some of the ways my devs work! thanks

[@warlock5624]

These systems design and architect videos are good. I don't see others doing this. Easy to understand.

[@alazar7685]

Another Banger! these videos are so great for juniors like myself. Thanks

[@jadeedstoresupport8916]

I find this approach of graphically showing the process very helpful in grasping the concept. I find other videos of Web Dev Cody equally helpful. God bless for doing the good work.

[@Sky-yy]

This is dope as always .most underrated youtuber 🔥

[@SeibertSwirl]

Omg maybe you were first today!!

[@Sky-yy]

@@SeibertSwirl yep 😉

[@wasifzaki]

These architecture videos are the most valuable ones in your KZbin Channel. Keep making these on every development architecture topic. Extremely Valuable! I subscried due to this only!

[@oussamajilaliarbaoui1981]

These became my fav series to watch first thing in the morning, Keep up!

[@andredasilva6807]

really intresting video. one of the most informative channels on yt. especially for real work environments. and not just simple topics. thanks and please keep on doing these videos :)

[@nova2wl]

nice vid and subject matter. I worked at a fortune 500 ecommerce company (not amazon lol) in IT operations and our group handled the switch over from Blue to Green. not only did we have B/G set up, we had a B/G environment set up for each cloud environment. we used GCP, Azure , AWS and a DIY vsphere for redundancy. Scaling gets kind of wild at large scale operations. can only image what the cloud bill was.

[@WebDevCody]

I bet it took months to figure out how to setup, especially when it comes to db migrations

[@MrEfrain15]

Your videos are getting better and better dude, keep it up 👍🏼

[@amirmalul1562]

I really enjoyed this video. I started to code recently and I found this video really helped me understand. Really well explained! Thank you

[@unusedTV]

"Let's pretend that you have users" Mate, I didn't come here to get burnt.

[@WebDevCody]

🤣 that was an unintentional comment, you do have users, yourself.

[@SeniorIntern]

Hey Cody. I really loved this video. And I hope to see more videos like this

[@mayankmaheshwari2544]

again G.O.A.T., keep making more such videos

[@Anuuurag]

great vids! ppl need to see the engineering side of software and not just the grindy leetcode stuff on youtube

[@emee__]

I have always thought about this, thanks alot

[@artsyleadership]

I love your videos, you've gained a new subscriber :) keep it up!

[@adriandeveraaa]

Thank you for explaining. Very clear and concise. Do you have examples for the loadbalancer and redirection script?

[@WebDevCody]

personally I use aws at work so it would just be as simple are pointing api gateway to another version of lambda. If you're using nginx, you'd just update your reverse proxy configuration and reload nginx.

[@adriandeveraaa]

@@WebDevCody We use nginx for our dev deployments internally but aws outside. The latter explanation was what I was looking for but poorly phrased. Thanks for the info!

[@abdjanshvamdjsj]

Very good channel. Great useful content.

[@tacopito]

Keep up with these system design videos

[@berwillem7416]

What would be really cool is sharing the shcemas made for each video

[@berwillem7416]

Wonderful explanation love your videos man thanks

[@Ali-tm7ly]

Good topics sir…keep up!

[@williamx0]

Can you talk about how to change your database schema in production? How do you move old data to the new schema in production? Also how would you switch DB providers like from supabase to aws in the future at scale in production?

[@WebDevCody]

So in regards to blue / green, typically you'd have to have 2 production databases. One is live, one is passive. You need a system behind the scenes to synchronize database changes from the live database to the passive one. You'd do your new database migration on the passive database, then you'd just switch your traffic to the passive environment and you'd be good. This is what we do at work and it's complex to setup. Another approach is to read up on expand and contract. Often if you need a migration, you should avoid deleting or doing alters on columns. If for some reason you DO need to change a column name or datatype, you'd need to first run a migration which adds a new column, then you need to deploy code which reads from the old column and writes to both the old and new columns, then you need to run a script which fills in any empty "new column" values with the old data, then you need to deploy code which ONLY uses the new column, then you run a migration script to delete the old column. It's crazy, so if you can just turn the system off for a couple of minutes and run your migration, just do that 🤣 It'll save you and your team months of implementing a solution that actually works.

[@williamx0]

@@WebDevCody thanks! Thought it'd be a good video idea. This is also very helpful. Thank you!

[@Wolkenkeller]

Thank you, sir.

[@hakanaki]

I love your videos, can you please do a system design tutorial 🥺🥺🥺🥺

[@mdomerarafat4611]

boy i am impressed!

[@stelioslagaras5334]

New subscriber here :), congrats for the excellent content. Let me add my thought on the db issue on "00:05:10", I remember working with db migration tools like alembic in Python. I guess there are similar tools for large scale applications?

[@Magns-AI-m]

Can you teach us how you plan out a real world application?

[@eshw23]

Is system design important to know for Junior devs? Seems like a lot to manage with regular algorithms, react, and javascript interview prep. But if its important for juniors ill add it in!

[@WebDevCody]

no, system design is more advanced

[@embiem_]

"Let's pretend like you have users"... oh Cody, that's all I do :'P

[@andrewoikonomidis8021]

Hey Cody, very nice explanation of this topic. One question, though, instead of vms, could these 2 boxes(blue, green) be containers inside a vm?

[@WebDevCody]

Yeah, but honestly you wouldn’t want to do anything of this yourself. Something like kubernetes would handle all of this for you when you deploy a new container version

[@ruud1071]

Great video! This might be a stupid question, but could it be that during switch-over the end-users' sessions need to be moved over too?

[@WebDevCody]

most load balancers are smart enough to just let old sessions finish and only new sessions are routed to the new color. But yes, if you are using websockets or something, you'll need to figure out a way to switch those connections to the new color, probably broadcasting some event to the UI so that users know to try re-connecting. Idk if loadbalancers would be smart enough to disconnect existing WS connections.

[@ruud1071]

@@WebDevCody Thanks a lot for the swift reply! Really appreciated.

[@iJuce]

Please make more of these

[@tomengelmann7272]

Hey Cody, thank you so much for your videos! They offer a lot of value to junior devs like me. I wanted to ask if you could make a video about pricing of hosting in general? I currently have the problem that i don‘t know exactly where to host my project without losing track of the costs. I built a react app with a go backend and think it‘s pretty hard to read through the jungle of hosting providers. Best regards from Germany :)

[@WebDevCody]

it depends on your budget. The cheapest option I know would be using sst to deploy the front end, and maybe railway or digital ocean; any type of container host for your Go api. I'd avoid services that require a base $20 a month fee; just find a service which is pay per usage.

[@StefaNoneD]

What if the dabtase scheme alters in V2?

[@MK-xl5yo]

Problem with blue/green deployments when there're significant migration changes in DB schema itself, and in this case DB cant be compatible with both blue and green versions same time. and therefore it's impossible to move forward without downtime. Either system have to have 2 DBs as well but that's costly and prone to data loss and errors, or green version needs to supplied with backward compatible code version, which makes code maintenance more challenging

[@WebDevCody]

yeah that's the biggest and hardest problem. There are strategies such as expand and contract, but ultimately it's not as simple as just switching a toggle and you're done. We spend a long time on our project setting up a blue / green data migration using dynamodb so that when we need to modify the existing data, we spin up a new database, migrate the data over, setup a stream to allow the live system to sync new changes from the old database to the new database, then orchestrate all of this using a complex CI/CD pipeline. After having to help work on all of that complexity, I personally would rather just turn the system off, do the update, turn the system back on.

[@MK-xl5yo]

@@WebDevCody exactly sometimes, it just needs to be some maintenance window usually during weekends/midnights when things needs to be done to avoid such complexity in infra and code, just to make sure some 0.001% of user traffic didnt experience downtime.. I saw such complications in multiple companies I was working previously and Im very doubt on net benefit of such no-downtime approaches..

[@WebDevCody]

@@MK-xl5yo I think there are benefits for huge companies like aws, google, etc. but for a smaller company that doesn’t have a global reach, idk if it’s worth it. If you can inspect your traffic at midnight and you only have a handful of users clicking around, it’s probably best to not worry about this complexity

[@lolcommunity4987]

Nice video :) Isn't it an heavy overhead to have 2 VMs just for achieve zero downtime on new deploy? (assuming I don't need load balancer atm)

[@WebDevCody]

yes, that's why I'd only try to do this if zero downtime is important to you, your company, or project. If you can do deployments at 12:00am and have no issue, I'd do that.

[@dharshans9954]

Can you provide more context on scripts?

[@adomicarts]

Thanks

[@oSpam]

I'm confused, cause if you're deploying a new version, technically V1 wouldn't work anymore. Let's say you changed a field in the database, or renamed an old API endpoint. I don't get how it would be able to work on both V1 and V2? I use django and using django this wouldn't work, just wondered if this is a thing that would work on nodejs but not django?

[@WebDevCody]

I think I forgot to mention the database side of things. If all you’re doing to your database is adding fields, then the old version doesn’t care about those fields and will work fine. If you rename a field or delete a table, that’s when you actually need to do a more complex setup or do expand and contract database migration. I have a video about expand and contract. You can actually get pretty far without using destructive database changes.

[@vinividipepe]

“Let’s pretend you have users…” 😭😭😂

[@xpressivemusic4578]

is there a book with these techniques

[@rakhisingh9797]

Bro but in AWS we can make a group in which we can add many VMs. And then put a load balancer on that group so if you change any VM in that group your downtime would be zero. But How we can do that zero downtime process with lamda func and amplify,pls make a video on it 😊

[@WebDevCody]

yes, that's why I said many PaaS provide some type of no-downtime deploy out of the box. If you're using lambdas, you can setup lambda versions and use a lambda alias in front of the versions. Then you'd point api gateway to that alias. When you need to deploy a new version, just update your alias to point to the new version and everyone will start using the new code. If you need to rollback, just update your alias back to the old version. I will say that deploying a new lambda without versions already results in zero down time, but users will instantly be switched over to the new code. Typically the blue-green setup is if you want to run a bunch of automated or manual tests against the new environment before switching.

[@rakhisingh9797]

@@WebDevCody 👍

[@terry.chootiyaa]

*We need more theoretical videos like this* ☺👍

[@shadowspyes]

do you have any examples of the load balancer interaction to send all new requests to v2, while allowing currently executing v1 connections to finish? preferably with actual code / live / demo examples.

[@WebDevCody]

That is often a feature built into the load balancer. For example, nginx will keep all the old connections alive and only route new traffic to the new color. You'd just go into nginx configuration, change it to the point the new color, reload nginx (this is a no-down-time process), and new users will hit the new colors.

[@shadowspyes]

@@WebDevCody that is great to know, I would assume other reverse proxies have similar functionality when it comes to editing the functionality to point a route to a different host. does this functionality have a name (the no-down-time reload of configuration) that I can use to figure out if it's available in other reverse proxies/load balancers like traefik?

[@WebDevCody]

@@shadowspyes it’s called hot reloading I think

[@SeibertSwirl]

Good job love!!!!! 👸🏿

[@Sky-yy]

I'm first Tina 🥲

[@SeibertSwirl]

@@Sky-yy lol not today, today I was first. Sorry 👀

[@worldwide6626]

Queen you are so wholesome

[@last.journey]

What about railway

[@WebDevCody]

I haven't noticed downtime with railway, so I'm assuming they have some type of behind the scenes routing once the new docker container is online.

[@simonshkilevich3032]

Or just use k8s and rollout versions in pods...

[@WebDevCody]

Sure, but the underlying approach is similar that k8s has abstracted away

[@LuisEDITS_KLK]

where is the credibility? do you have a linkedin?

[@WebDevCody]

I've been coding for 2 months, trust me, I'm an expert

[@LuisEDITS_KLK]

@WebDevCody bro, seriously I like your a few of the videos I watched but I want to know if I'm learning from sb who's got real work experience