Mastering Argo CD Sync Waves: A Deep Dive into Effective GitOps Synchronization Strategies

Discover the ultimate control for your Kubernetes GitOps deployments with Argo CD's Sync Waves, Statuses, and Dry-Run features! 🌊⚙️
#argocd #kubernetes #gitops
Consider joining the channel: kzbin.infojoin
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Gist with the commands: gist.github.com/vfarcic/64a2d3ed69aa159b6578d0e1e2d0b225
🎬 Argo CD - Applying GitOps Principles To Manage A Production Environment In Kubernetes: kzbin.info/www/bejne/rKG6gpills6Fg5Y
🎬 How To Apply GitOps To Everything - Combining Argo CD And Crossplane: kzbin.info/www/bejne/r6PNZZ-jiMh-frM
🎬 Crossplane - GitOps-based Infrastructure as Code through Kubernetes API: kzbin.info/www/bejne/pGmum4mjqq2gbaM
🎬 How To Shift Left Infrastructure Management Using Crossplane Compositions: kzbin.info/www/bejne/d6XFhGSrZ89qptE
🎬 SchemaHero - Database Schema Migrations Inside Kubernetes: kzbin.info/www/bejne/iaDJgquYaah3h7M
🎬 Is Timoni With CUE a Helm Replacement?: kzbin.info/www/bejne/mJOoYnV8eNhoapo
🎬 GitHub CLI - How to manage repositories more efficiently: kzbin.info/www/bejne/eHqsZ42PZ7ehosU
▬▬▬▬▬▬ 💰 Sponsoships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please use calendly.com/vfarcic/meet to book a timeslot that suits you, and we'll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: vfarcic
➡ LinkedIn: www.linkedin.com/in/viktorfarcic/
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: kzbin.info
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 Introduction to Argo CD dry-run, sync order, and statuses
03:26 Stateless Application With a Database
05:37 Create The Application, The Database Server, and The Schema
07:42 Argo CD Phases and Sync Waves
10:00 Argo CD Resource Statuses
12:13 Skip Argo CD Dry-Run
14:05 Resource Ordering With Argo CD

[@DevOpsToolkit]

How do you orchestrate your orchestrators? 🤔

[@DennisHaney]

Add a couple of retries in your argocd application and most of these issues disappear. Then you only need it for critical things where order really matters

[@DevOpsToolkit]

You're right. More often than not, sync waves and LUA status scripts and not needed and dry-runs can keep being enabled. But there are cases when that's not true. Dry-run fails miserably if you have CRDs and CRs in the same app, sync waves are a must with databases, and status scripts are often necessary when working with CRs.

[@DennisHaney]

@@DevOpsToolkit No? I haven't had any problems with that since we switched to ArgoCD. It's gonna fail to sync the cr, but the retries will pick it up in a few seconds usually after the crd has synced. Perhaps it is because i use micro helm Chart.yaml to specify dependency instead of directly in Application CR.

[@DevOpsToolkit]

Yeah. Some of those features are available in helm so you might not have experienced those issues.

[@DennisHaney]

@@DevOpsToolkit Helm alone absolutely sucks with cr and crd in same chart/subcharts. But argocd dry-run+serverside apply+retry and the problem goes away since it does sync for each resource on its own is my experience.

[@lukas-pastva]

YEAS! Finally I was given explanation how to make the sync waves are working. I've been dealing with it for half a year. Chicken/Egg issue solved ;-)

[@microst99]

Great stuff as always ! Also: Yes, we need the schemahero replacement vid too ! PS: Whenever i see the Timoni thumbnail i read REPLAY CEMENT xD

[@DevOpsToolkit]

Yeah. It's not one of the best thumbnails.

[@nzalex1]

Man, you content is awesome. I am Devops engineer for years, and still find tons of very useful things in your videos. Thank you!

[@IvanRizzante]

Thanks for another great video 🎉 yes I am curious to know about the new upcoming misterious tool for the database schemas

[@Fayaz-Rehman]

Thank you for sharing your experience - how would you address cache memory after changing ID and Passwords in an application through ArgoCD?

[@franpereyra5459]

hey @DevOpsToolkit , thank you very much for your content. Here's a small suggestion/request for future videos: with so many new technologies coming up periodically, could you create a video or maybe a series of videos that helps Kubernetes Administrators and/or architects to choose a set of recommended toolkit and tools for creating and managing a K8S cluster from SCRATCH? This could be based on specific use cases or company/team sizes, etc. Thanks for advance!

[@DevOpsToolkit]

I do something like that once a year. It's not limited to kubernetes but it does end up being about kubernetes in most cases. kzbin.info/www/bejne/Y6uuiodofsSKnqc is the last one.

[@SiCrip09]

Interesting to see using hooks with timoni in ArgoCD which is like helm hooks 🎉

[@hugolopes5604]

these are edge cases... imo eventual consistency is the #1 solution. regarding the schema changes in particular, if one wanted to do them without downtime, aren't backwards compatible schema changes methodologies needed? ... and fully agree that schemahero should return a state... amazed they failed such basic stuff!

[@aditya-iq8cp]

firstly, thank you for the awesome tutorials. Being said that I have a question, if a resource creation during the sync wave fails, would the other resource of sync wave rollback too along with the failed resource? if not then how we can achieve this so that there isn't a version drift in between the components of application upon failed upgrade.

[@DevOpsToolkit]

As far as I know, there is not auto rollback in that scenario. The question is whether there should be rollback (and i would say that if you want HA, the answer is no).

[@aditya-iq8cp]

@@DevOpsToolkit Makes sense, can you please suggest a standard way to tackle this situation since if the one component is at a different version of application, than the other components due to failure upon upgradation that would result in incompatibility in between the components. I thought of reverting back to the previous commit (healthy state) in the repository that argocd is monitoring although am not sure if that's the best solution.

[@DevOpsToolkit]

@aditya-iq8cp it's very simple. You cannot release a new version that is not compatible with the previous one. Otherwise, it is impossible to accomplish high availability and zero downtime.

[@CestMoi-rw8zx]

I am new to GitOps, kubernetes, containers, etc and have been consuming yours and other's videos in an effort to get preview environments established for our developers. I seem to be having this very problem with ordering, but I'm using flux. Could you please cover how you would order and skip dry runs as related to flux?

[@DevOpsToolkit]

Adding it to my to-do list... 🙂

[@CestMoi-rw8zx]

Thank you! I am so new to this I wasn't sure whether to choose flux or argo, but I went with flux because of the bootstrapping. Always keeping argo in mind though because it is included in most vcluster mentions. When trying to envision how we might test our containers prior to merging to master, I saw your videos concerning vclusters and preview environments, and then I could finally see a path forward. Even so, finding the particulars on how to set this vcluster preview env up has me combing through the internet for details with little success so far. Any instruction you can throw this way will be greatly appreciated, I assure you. Thank you!

[@DevOpsToolkit]

I'll do my best to make a video on that subject.

[@hugolopes5604]

for the schema change, can't a readiness probe check if the required schema change was applied? just an idea... like that, one could deploy the app container before the schema change no?

[@DevOpsToolkit]

Readyness probe could and should do that but schema hero does not report the status so there nothing to hook the probe.

[@hugolopes5604]

maybe the readyness probe code would connect directly to the db or a specialized api of the app that would return the db schema or an schema compatibility value... ugly, but it would work... and if one forgot to update the probe, it would be caught some CI test. just an idea... not sure if this would be a better workaround than other options...

[@DevOpsToolkit]

That would work, but only if there is no better alternative (alternative tool) that would do what is expected from Kubernetes resources to do.

[@geowalrus]

What about orchestrating using k8s native workflow tools like argo workflows or kubevela workflows?

[@DevOpsToolkit]

That's a different type of orchestration. I put those into pipelines category that is useful for one-shot actions like building images and running tests, but not for those that should be continuous like drift detection and reconciliation (e.g. gitops).

[@joebowbeer]

I don't think you mentioned ArgoCD's ApplicationSet in this video, but if you did, it would be worth mentioning that ApplicationSet doesn't currently obey sync waves on updates.

[@DevOpsToolkit]

You're right. I forgot to mention that one.

[@Luther_Luffeigh]

ApplicationSet has Progressive-Sync which currently experimental feature

[@cacojr5381]

Why does applicationset need syncwave? Does this refer to the update order of the app the appset renders?

[@DevOpsToolkit]

I don't think syncwave is mandatory@@cacojr5381 . It's there if you need it.

[@cacojr5381]

Sorry for the delayed response. I may have not expressed myself accurately earlier. As I understand it, sync-wave pertains to the synchronize order of resources within an app. However, I'm not sure what applicationset's sync-wave refers to. Is there an official wiki that provides guidance on this?"@@DevOpsToolkit

[@RonakKogta]

Good video! Can we do inter application ordering as well. Cause from way i understand it pre-sync and post-sync are good annotations for objects belonging to same argocd application. But i was hoping to see waves to co-ordinate one argocd application dependency on another one.

[@DevOpsToolkit]

If they are manifests if the same argo CD application, the answer is yes. If they are different argo CD apps, then no. They are likely not even the same commit.

[@pavelpikat8950]

A better way of managing schema - is it Atlas? :D

[@DevOpsToolkit]

We'll see in 4 weeks approx 🙂

[@invalidred]

What is the best way to manage schema k8s style? Suck a cliffhanger!

[@julianomoraisbarbosa]

# til

[@kasvith]

for somehow if a deployment failed, would previous version be deployed? so users can access it anyways

[@DevOpsToolkit]

Assuming that you're using the default rollout strategy in a Deployment, changing a Deployment creates a new ReplicaSet and starts the process by creating a new Pod, waiting until it is healthy, then shutting down the old Pod, and so on. So, as long as you have healthchecks, it should be fine. As a side note, that is completely unrelated to Argo CD.

[@kasvith]

@@DevOpsToolkit thanks

[@kasvith]

@@DevOpsToolkit for example let's say a PreSyncHook failed, it runs the db migrations(usually application ships migration schema with tools to deploy it[like knex in js world]). In platforms like Heroku, Dokku when this happens whole release rollout and previous app keeps working until the new version fixes issue and pass release phase. how can we achive something like this in argo

[@DevOpsToolkit]

Now I (think I) understand what you mean. You're asking what happens if you're applying multiple workloads, and one of them fails. Is that correct? If that's the case, Argo CD will move to the next sync wave only if the workloads in the previous one were successful. So, if the first wave is a change to the DB schema and that fails, the second with, let's say, the application will not be applied. Now, if you have 3 waves, and the second one fails (by fail, I mean that the health checks failed), then the 3rd will not be applied, but the 1st will stay as it is (with whatever was applied to the first wave).

[@DevOpsToolkit]

That you have more than two waves, and the last scenario is unacceptable, all you have to do is revert the commit or re-commit it.