Automation of Everything - How To Combine Argo Events, Workflows & Pipelines, CD, and Rollouts

What do we get if we combine events, workflows, GitOps, progressive delivery, and secrets management? The short answer is that we get automation of everything in Kubernetes in a way that we should be operating in 2021.
We'll combine Argo Events, Workflows & Pipelines, CD, and Rollouts and sprinkle all that with SealedSecrets, Kaniko, and a few other tools.
#argo #kubernetes #automation
Timecodes ⏱:
00:00 Intro
01:08 Setup
04:07 GitOps deployments
12:43 Events and workflows
22:19 GitOps upgrades
23:50 Canary deployments
26:00 Final thoughts
➡ Gist with the commands: N/A (the Gist is not working anymore since quite a few things changed since I created the video).
🔗 Argo Workflows: argoproj.github.io/projects/argo
🎬 Argo CD: • Argo CD - Applying Git...
🎬 Argo Events: • Argo Events - Event-Ba...
🎬 Argo Rollouts: • Argo Rollouts - Canary...
🎬 Kaniko: • Kaniko - Building Cont...
🎬 Sealed Secrets: • Bitnami Sealed Secrets...
🎬 Kustomize: • Kustomize - How to Sim...
📚 DevOps Catalog, Patterns, And Blueprints: www.devopstoolkitseries.com/p...
📚 Books and courses: www.devopstoolkitseries.com
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox
➡ Follow me on Twitter: / vfarcic
➡ Follow me on LinkedIn: / viktorfarcic
Corrections:
04:10 The Gist is not working anymore since quite a few things changed since I created the video. Nevertheless, the logic behind the video is as valid as it ever was. The only change is in the manifests.

[@valour.se47]

This is one of the most valuable video for argo stack, thank you für making it, everything sits well in its place, now let me build my own pipelines (^_-)

[@DevOpsToolkit]

Glad it was helpful!

[@dmitriinadein4733]

Best video on KZbin about Argo stack. Very high quality. Thanks a lot!

[@JayCeeVids]

Man your videos on Argo have been so good, great examples and explanations! Keep up the good work!

[@DevOpsToolkit]

Glad you like them!

[@angelmanuelcamarero9813]

Amazing video, a temple of wisdom. Now it is time to read carefully all repo and try to digest all this information. Thank you very much, Master.

[@mangeshtekale5199]

This is absolutely fantastic, very well explained in an intuitive manner as always. Thank you so much!

[@mark94213]

Love it - thanks so much for the great summary and keeping it fun. Appreciate all your work!

[@MohammedNoureldin]

The content of your channel is very amazing. Thanks a lot!

[@bewilderedlearningevolving]

Absolutely awesome! Far cry from delivering code (in a car) to the server via disk in the original toolkit book. We've come full circle! Thanks, Viktor!

[@DevOpsToolkit]

When I wrote the first toolkit book, I thought it was innovative. Today, I am almost ashamed of it. A lot has changed.

[@j0Nt4Mbi]

Amazing demonstration thanks for sharing.

[@kulinskyvs]

Awesome video! I was looking for something that can finally connect all the dots in my head about using argo projects... and I found it!

[@DevOpsToolkit]

Glad it was helpful!

[@localho]

love the effort you put in, argo is really nice and this is only the beginning!

[@DevOpsToolkit]

Thanks a lot!

[@HaraldKubota]

This is a great video to see all those puzzle pieces put together to form a coherent system.

[@DevOpsToolkit]

Oh yeah. Combining all Argo projects is potentially a very powerful solution. It might take a while to piece it all together but it is well worth the effort.

[@sunnyxd5000]

Wow. Amazing. You helped me so much! Thanks :)

[@Clemens2022]

Hi Viktor, this is an amzing overview on how everything fits together and a great pointer to get into all the seperate topics in detail. It would be amazing to see your approach of leveraging a GitOps approach to a statefull application with a backing database and database migrations running as part of the CD Pipeline when changing the app version.

[@DevOpsToolkit]

Adding it to my TODO list... Until I do a video you might want to check out schemahero (you'll find it on this channel). If fits nicely into the scenario you mentioned.

[@karimnaufal9792]

Thank you, Victor! My mind just exploded to a thousand pieces lol

[@stanrock8015]

Amazing. This is where I want to be

[@silverfish27]

Thank you for the video, great help1

[@wollginator]

I've told you, I'm gonna watch it right away after seeing the other argo videos, thanks, BIG TIME, keep it up and running :0)

[@DevOpsToolkit]

Any suggestion on a topic I should explore in one of the upcoming videos?

[@wollginator]

@@DevOpsToolkit There are currently three things on my list I'd really like to know more about and see you exploring: 1) Yugabyte which is a distributed, ACID-compliant SQL database also offering a Cassandra-like API, 2) OPA Gatekeeper for policy management and 3) Linkerd as an easy to use Istio alternative, what do you think?

[@DevOpsToolkit]

Great suggestions. Added all three to my TODO list.

[@DevOpsToolkit]

I'm finishing a video about OPA Gatekeeper. It will be published next week.

[@wollginator]

@@DevOpsToolkit That's awesome, thank you very much, I'm super curious!

[@stepan-kuksenko]

thank you a lot, still this is the most informative video and example (in github) how to build complete ci/cd with argo

[@bules12]

I followed this guide and created my own CI/CD Pipeline using argo-combined-app and argo-combined-demo repositories I will give you a flow schema I did after some days struggling to setup my CI/CD using this logic: git push main branch -> github webhook triggered -> argo-events triggered -> argo-workflows triggered -> workflow build your images and push to registry -> workflow perform git push to production branch updating image name -> argoCd triggered due to that push

[@bules12]

Thank you so much Victor for this

[@AntonGorbachevDev]

I think we can skip workflow with git push and use Argo image updater instead

[@DevOpsToolkit]

@lastbreath2437 even better, use Kargo From Akuity.

[@bules12]

@@DevOpsToolkit Kargo demo vídeo when?? 🤣

[@DevOpsToolkit]

@bules12 probably 6 weeks from now. It is in the "started working on it" section of my to-do list.

[@cnuk1515]

Thanks for the wonderful videos on Argo and other CI/CD tools. Wondering if there’s any other video for setting up Argo in an enterprise with proper RBAC, groups, Projects and other features. Thanks.

[@DevOpsToolkit]

I haven't made such a video (yet). It would be an interesting subject. Adding it to my TODO list...

[@RafaelDurelli]

Nice video as always. Would be nice a video about argocd notifications :D

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@mohammedragab4966]

one of the most best videos thanks a lot , what the problem I am always facing on these kind of integration is that if I have the K8s cluster and argo stuff on a private network while github or azureDevops is public and K8s stuff accessible only via VPN, I think as a work around we can have a private runner on the private network attached to the github or azureDevOps and create a pipeline that trigger the argo event webhook so the pipeline will use the private runner to curl the webhook

[@artemkosenko9551]

Really grate video! Looks like a real life. Thanks for showing how these tools work together. Maybe it it time to get rid from CI part on jenkins. Wow! Really cool automation. How to find a time to make all this things done on the project now? ))

[@gustavorivero8395]

excellent

[@stepan-kuksenko]

Thanks!

[@DevOpsToolkit]

Thanks a ton, Stepan.

[@ranrubin7377]

Hi Viktor, thank you so much for this video. You present the subject in a very clear manner, it was a pleasure watching. I also like the App Of Apps Pattern that you described. I've noticed (correct me if I'm wrong :) ) that instead of using a Helm chart to gather all the applications (as shown in ArgoCD documentation), you used a simple directory with the Application files (argo-combined-demo/production/). When working with similar environments, wouldn't that cause a potential code duplication? while with an helm chart you just have to create 3 different value files, and have the application files reside under "templates ".

[@DevOpsToolkit]

To be honest, I don't remember what exactly is in the docs anymore. I've spent a lot of time in docs, code, and other places that I'm not sure anymore what comes from where, and what are my own practices. It is true that using Argo CD apps for each application/environment might result in additional lines in YAML. Still, I find that easier that working with Helm charts with Argo CD apps. Given that I'm trying to automate as much as possible, it is easier for pipelines to add/edit/remove files than to deal with Helm values. Also, I believe that "my way" is more flexible. I might have apps that are in one environment but not in others. I might be "experimenting" with something in staging that should not be in production (at least not yet). Also, I believe it is easier to update a file that is dedicated to an application (e.g., change a tag whenever there is a new release) then to update Helm values.yaml. That being said, everything can be done both through charts and directly through Argo CD apps. Think of them as being the same as Helm values, with a few additional lines of YAML. Ultimately, I'm more interested in automation than in avoiding duplication. Duplication is problematic mostly when we do things manually. That does not mean that we should not avoid duplication, but, rather, that it is of lesser importance than automation.

[@BoatsForALiving]

Quite the compilation.

[@DevOpsToolkit]

Thanks

[@ChiliT01]

Great video and overall series! One question I have is you seemed to start favoring Flux v2 over ArgoCD in a previous video. Did you choose ArgoCD here because of the Argo components or do you think it may still be superior (slightly) to Flux?

[@DevOpsToolkit]

I think that the differences between Argo CD and Flux are small. The reason I tend to be oriented more towards Argo CD is because it combines very well with the rest of Argo projects. If you would like to accomplish something similar with Flux, you would need to combine it with, let's say Tekton (equivalent to Argo Workflows), some kind of eventing (like Argo Events), and Flagger. In other words, Flux might be a bit better (or a bit worse) than Argo CD when looked at in isolation, but when combined with other Argo projects, Argo CD is a clear winner (at least for now).

[@maverickme22]

Simply wow…. Very well explained.. can you please make a video on securely connecting to ArgoCD dashboard using VPN in AWS for team.. the only missing piece…

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@matthiaslarre1269]

First I would like to thank you for the video, it is very well explained and easy to follow. There is only one thing I do not get is what is the URL you use in Github webhook definition and in the host in the corresponding ingress ?

[@DevOpsToolkit]

It's nip.io service that I use in almost all of my demos. It allows us to have valid domains by redirecting requests to the IP specified as a subdomain. For example, if the IP of you LB is 1.2.3.4 and you send a request to `something.1.2.3.4.nip.io`, that request will be forwarded to 1.2.3.4. It's great for demos and testing when you do need a "real" domain.

[@yts3355]

Got a question on deploying Sealed Secrets using ArgoCD. If I am redeploying the Kubernetes cluster (along with all the applications on it), is it possible to also restore the backed up encryption keys for Sealed Secret keys using GitOps/ArgoCD, such that the SealedSecret objects committed in Git for other applications are reusable (without having to reencrypt)? Or should we be manually restoring the encryption keys after ArgoCD has deployed Sealed Secrets?

[@DevOpsToolkit]

You should backup and restore the keys manually or through some automation like pipelines. You cannot do that with ArgoCD simply because it uses Git as the source and that's the last place you'd want to put your keys.

[@artiston1]

¡Gracias!

[@DevOpsToolkit]

Gracias a ti. Lo agradezco.

[@witekprzeciechowski7695]

Great video. One thing I'm trying to figure out is how would you delay integration tests to run after the deployment. It seems to me that in the video tests executed before the deployment to staging was picked up by Argo CD. Do you have any advice?

[@DevOpsToolkit]

I think that integration tests should run before, during, and after releases are fully rolled out. But, if you want to run them only after a deployment is fully rolled out, you should probably execute `argocd app sync` command to ensure that the changes you made to Argo CD manifests were synced inside the cluster.

[@witekprzeciechowski7695]

@@DevOpsToolkit thank you for the quick reply! I'd love to see a video about such integration testing strategy.

[@DevOpsToolkit]

@@witekprzeciechowski7695adding it to my TODO list :)

[@andreidascalu3079]

I loved this tutorial, though I have to ask. I noticed in the devopsparadox repo for this workshop, the kustomize for argocd also includes an ingress definition. So, aside from Argo, it's assumed there was an ingress deployed that's technically not managed by Argo, right ?

[@DevOpsToolkit]

There should be a link to the gist with everything, including the setup I did before I started recording. If I remember correctly (it's been a while), ingress is included in the gist, together with all the other tools I used. I can double check...

[@bogdan_angh]

Hello, Viktor! I watched this video again and I understood the concepts much better now! I will have to watch again in the future, because it is a gold mine. I don't understand one thing though: you have a lot of Ingress definitions, and I don't really understand how that works. In my head, an Ingress definition creates a new Ingress => more costs. Shouldn't one Ingress definition be enough to handle all the routes and the hosts? Thank you for your effort!!

[@DevOpsToolkit]

There is Ingress controller and Ingress resource. You get Ingress controller when, for example, you install NGINX Ingress. That is one instance (it could have multiple replicas though). Now, whenever you create an Ingress resource (not a controller), the controller reconfigures the Ingress instance. So, if you create 10 Ingress resources (like those I have in that video), that only means that the Ingress instance (e.g. NGINX) was reconfigured 10 times. There is still only one physical instance of NGINX. In other words, there are many Ingress definitions, but only one physical Ingress instance which, among other things, contains a combination of the configs from all those definitions. I'm not 100% sure whether I understood your question and whether my answer makes sense. Please let me know if it doesn't and I'll do my best to go into more detail.

[@bogdan_angh]

@@DevOpsToolkit Thanks for the detailed response! I understand it now, that one Ingress instance was reconfigured multiple times in order to statisfy everyone's needs.

[@KanshiTV-INT]

Hi Viktor, brilliant video's learnt a lot from them. I'm trying to get Traefik ingress working with workflows and not having much luck, can you give me some pointers please.

[@DevOpsToolkit]

Generally speaking, it should work with traefik ingress without any extra steps. Can you make a gist so that I reproduce your issue?

[@KanshiTV-INT]

@@DevOpsToolkit Hey Viktor, I managed to get it working buy adding the following - server - --secure=false and letting Traefik do the certificates

[@DevOpsToolkit]

@@KanshiTV-INT I thought I already had those args set for Argo CD in my example. I probably missed it in that Gist.

[@sidkris7757]

Really good video. In fact the only good one I have seen. Got it to almost work with EKS with a registered domain but trying to figure out how to use with a private ECR registry. Any hints would be appreciated. Thanks.

[@DevOpsToolkit]

You need to store ECR creds as a secret and, more importantly, your EKS cluster itself needs to have permissions to use ecr.

[@sidkris7757]

@@DevOpsToolkit Thanks

[@mrnobody5763]

How can I channel who produce such great content have only 2k subscribers? It's insane.

[@DevOpsToolkit]

There are multiple reasons for that. To begin with, I started uploading videos only 6 months ago. I had a channel longer, but almost nothing was happening in it until September 2020. On top of that, I am incapable of doing marketing and being present in social media. I do not promote my stuff, I do not Tweet, and I do not even have a Facebook account. The only thing I do is post a link to my videos on Twitter and LinkedIn without engaging in other social activities. To be clear, I am not saying there is anything wrong in doing that, but rather that I am not good at marketing. I just like sharing. Now, that does not mean that I would reject help. If you or anyone else can help promote, I would be eternally grateful. Unfortunately, the goal of the channel is not to earn money, so gratitude is the only compensation I can give :)

[@rodrigo41087]

Hello, Is it possible to integrate keel with argoCD to raise an argo rollout event? Or add a tag to the Kind: Rollout so that it is restarted by the keel tags?

[@DevOpsToolkit]

I'm not sure whether it would make sense to integrate Argo CD with keel since they are accomplishing similar objectives but with very different approaches. Argo CD wants to keep the desired state in Git while keel updates manifests directly. If you're using Argo CD, you might want to explore Argo CD Image Updater instead of keel, or keep keel and ditch Argo CD. I suggest the former. There's nothing to do with Rollout. The moment you change the image tag in a manifest applied to a cluster (Argo CD does that for you), Argo Rollouts kicks off automatically.

[@dakaugu]

No luck running Argo-cd on an empty kubernetes cluster with nginx ingress and sealed secrets GKE. argocd-dex-server and argocd-server deployments always fail showing “Does not have minimum availability” . Checked logs and describe pod says failed volume mount or unknown command -staticassets 🤷‍♂️

[@DevOpsToolkit]

Not sure what's going on without a bit more info. Can you share the output of the logs and `kubectl describe`? You can PM me on Twitter (@vfarcic) or LinkedIn (www.linkedin.com/in/viktorfarcic/).

[@sbj7452]

how to use kube configmap across preview/active services. For us we have 1 configmap right now and after we setup argo rollout we might want a seperate configmap while testing preview service. How can we achieve that.

[@DevOpsToolkit]

Is that config map part of the definition of the app? If it is, you shouldn't have a problem. Each application instance will have its config map. I'm guessing I misunderstood the question. If I did, please give a bit more info about the issue.

[@snehotoshbanerjee1938]

Does it makes sense to use github action and argo workflow together? We use github action and argoCD. Since, github action is a complete platform with action, workflow, events etc. So trying to understand the pros and cons of using both as complimentary and substitute product. Thanks

[@DevOpsToolkit]

Argo workflows and GitHub actions serve the same purpose so I would not use both in parallel. GitHub actions is easier, more user friendly, and it's SaaS. Those age huge pros.

[@amabamo5769]

do you know how can you include argo in github/gitlab as a bot to show us workflow related to commit / pull request?

[@DevOpsToolkit]

Good question. I haven't (yet) tried it myself. Let me check it out...

[@maylingtan3716]

Hi Viktor, thank you for your teaching. Need some advice from you. After linking an ingress to the event source service, i encountered an issue "Some backend services are in UNHEALTHY state" (namely the event source service is unhealthy). Ingress is working fine as I tested with other backend services. On the argo event UI side, the event source, sensor and trigger are visible and connected. All pods are running fine, as I tested via port forwarding instead of ingress at first. Do you have any advice on this issue? Thank you.

[@DevOpsToolkit]

Did you create ingress for the event backend as well? Can you simulate the event by sending a curl request and check that you can indeed reach events backend from outside the cluster?

[@maylingtan3716]

@@DevOpsToolkit thanks for you reply. Yes I did create a Nodeport service and curl it from outside the cluster and it works. Then link the Nodeport service to the ingress, then got the issue above.

[@DevOpsToolkit]

@@maylingtan3716 I'm not sure what you mean when you said that you linked "the NodePort service to the Ingress". That NodePort service was created by the Ingress controller or something else?

[@spenhouet]

Great presentation! I could really use a step-by-step version of this video (or an extensive blog post + video). Probably a very long video but it would be something no other resource is currently offering. A full on "how to setup full CI / CD solution using Argo". You could make it a playlist with multipart videos. But it would need more focus on the details and less skipping / glossing over. Also less marketing and more fact talk. Would you be up for that? You already made the video. You already have the basic setup. I see that it would deviate from your usual content but it would be incredibly valuable for the community. Hope you are considering it. You could also put it on Udemy ;)

[@DevOpsToolkit]

That's a great idea. Adding it to my TODO list... I can't guarantee when I'll be able to do it, but that I certainly will whenever time permits.

[@spenhouet]

@@DevOpsToolkit You are the best! I'm very much looking forward to this. Keep it up :) We are a small dev team currently working with codefresh but looking for alternatives. We have no dedicated DevOps person and are struggling to set us up a system like this and there aren't any good resources on this. Really surprised that no one else wrote a guide like this yet.

[@DevOpsToolkit]

In the meantime... Did you check out the gist referenced in the video description? It does not provide a detailed explanation, but it does allow you to reproduce everything I did.

[@spenhouet]

@@DevOpsToolkit Not yet. We reevaluated Jenkins X again (did already a year ago). Could still not get it running on-prem. Next up is Argo. Will check out your Gist then!

[@tairitzhak2608]

Amazing and super helpful video! :) Do you have a github with all of the components and examples you used in this video?

[@DevOpsToolkit]

I had a gist with all the commands but, in the mean time, some things changed and the gist was not working so i had to remove it. That's probably the only video without a gist. I'll do my best to recreate it or make a new and up to date video on that subject.

[@tairitzhak2608]

@@DevOpsToolkit Ok thank you!

[@komalmisra9842]

Error during creaing a image using kaniko and your given code . error is " task 'build-container-image.build' errored: when using the emissary executor you must either explicitly specify the command, or list the image's command in the index "

[@DevOpsToolkit]

Recent releases of Argo Workflows, when using the default setup, require to define `command` explicitly. I just updated the repo to reflect that. Can you please pull the latest version of the repos or add github.com/vfarcic/argo-combined-demo/blob/master/argo-workflows/base/templates.yaml#L22 to your manifests? Please let me know if that worked or you're still experiencing issues.

[@gauravagnihotri4912]

where I can find out template for the container image creation using kaniko? and where do I get the workflow full file ??

[@DevOpsToolkit]

There is a link to the gist with all the commands in the description of the video. The gist includes the referenced to the repos with the manifests.

[@marwan.alrihawi]

How can i use argo to provision infra out of k8s by using teraform or crossplan for example ?

[@DevOpsToolkit]

If you use crossplane, your infra definitions are Kubernetes resources. As such, you can manage them through Argo CD just as any other Kubernetes resource. Terraform, on the other hand, cannot be managed by Argo CD, but you can create Argo Workflows that would be triggered whenever you change terraform manifests in Git. Such workflows can simply execute "terraform apply". Based on that, Crossplane is a more elegant solution. That does not mean that it is better than terraform, but that it fits better into the k8s ecosystem.

[@DevOpsToolkit]

The video I published about crossplane even has the example that combined it with Argo. You'll find all the commands in the gist in the description.

[@sarthak8055]

Hello Viktor, I had one query...! Can we use ApplicationSet with ArgoRollouts? Currently in our project all our applications are deployed via ApplicationSet using ListGenerator. Now we want to Implement argo rollouts to achieve canary feature. Rollout is achieved via Kind: Rollout so all our templates are changing please suggest

[@DevOpsToolkit]

Argo in general, including ApplicationSets, is all about synchronizing Kubernetes resources stored in Git with those in clusters. There is no difference from Argo CD perspective whether those are Deployment, Knative, or Argo Rollouts resources. So yes. It'll work with ApplicationSets.

[@sarthak8055]

@@DevOpsToolkit got it thank you

[@mohsin-ashraf]

How do I pass aws credentials to argo workflow to pull custom image from ECR?

[@DevOpsToolkit]

You can set imagepullsecrets. Something like github.com/argoproj/argo-workflows/blob/master/examples/image-pull-secrets.yaml#L11

[@n4870s]

So stuff like jenkins, Dockerfile, VMs, etc are rendered obsolete, right? Do you use ancible/puppet/chef/teraform?

[@DevOpsToolkit]

Not necessarily. Dockerfile is used no matter which container image building tool we prefer. Jenkins is the most versatile pipeline tool and especially useful in huge orgs with everything from 30 years old legacy to today's apps. VMs are servers inside k8s, even through we do not deploy to them directly.

[@n4870s]

@@DevOpsToolkit yes, well, with java/spring boot we use buildpacks to build docker images, not Dockerfile, so i thought there are better ways nowardays. And for jenkins - i guess we can skip it if it's a new project/company.

[@DevOpsToolkit]

@@n4870s What I wanted to say is that Dockerfile is not dead. It's useful even without using Docker to build images. That does not mean that it is the best format to describe images. Buildpacks is, indeed, a great way to build images without it.

[@mangeshtekale5199]

The videos are awesome as always. Just one thing, if you could share the working Gist, it will help us follow along with you

[@DevOpsToolkit]

I have a gist for each video, including this one. However, a few things changed in Argo projects and, as a result, the gist stopped working. I removed it from this video only to avoid confusion. I will put it back once I get a bit of time to go through it and fix the issues.

[@DevOpsToolkit]

I'll make a new one. It'll be better, and it will include a bigger scope of automation. It'll probably take me a few months to make it though.

[@mangeshtekale5199]

Thank you so much, I highly appreciate your efforts. Looking forward to the new gist :)

[@DevOpsToolkit]

@@mangeshtekale5199 It will probably not be a new Gist but a whole new video (with a Gist). Many things changed since I created this video, and I feel I can do better and with a bigger scope :)

[@CelesteOnYoutube]

@@DevOpsToolkit Now I'm dying to see the new video about this. Any idea when it will come ?

[@zenmaster24]

does this mean when argo workflows runs, it changes to the new image tag in the argo-combined-app manifest and commits/pushes it? for argocd to do gitops, this has value has to be in git somewhere, doesnt it?

[@DevOpsToolkit]

That's correct. Among other things, Argo Workflows modifies and pushes changes to the manifests which are, later on, picked by Argo CD. Instead making direct changes to the state, the job of the pipelines is to ensure that the desired state (Git) is up-to-date.

[@zenmaster24]

@@DevOpsToolkit cool - i havent watched it, but is it covered in your workflows video?

[@DevOpsToolkit]

@@zenmaster24 The whole flow is covered in this video (the one where this comment is). On top of that, there is a video for each of Argo projects separately.

[@zenmaster24]

@@DevOpsToolkit the link in the description? that seems to be to the argo project is this the one? kzbin.info/www/bejne/i37Emqmtp6asiqM

[@DevOpsToolkit]

@@zenmaster24 I meant that kzbin.info/www/bejne/jn-7e6euoLp-m7s (the video we are commenting on) is doing what you said (commits and pushes manifests that are later picked by Argo CD). Or. more likely, I misunderstood the question.

[@hackerops1407]

Please can you make video on argocd image updater how argocd image updater will pull latest image from our docker respositry

[@DevOpsToolkit]

That's quite a conincidence. I just finished a video about Argo CD Image Updater. It'll go live in less than two hours.

[@hackerops1407]

@@DevOpsToolkit eagerly waiting 😊

[@DevOpsToolkit]

I went live a few minutes ago. You can find it at kzbin.info/www/bejne/l6ezhoSfo56esLc.

[@amitgelber2060]

Hey! is there a pattern to automacally commit to git repo in order to add a new instance of existing app with a new config: something like -> kafka> lambda (commit to repo new config of existing app) ->argo cd-> new app instance Thanks!

[@DevOpsToolkit]

You can trigger an Argo event from Kafka and that could commit something to git. Is that what you meant?

[@amitgelber2060]

@@DevOpsToolkit yes, sounds good, I'll try that

[@amitgelber2060]

@@DevOpsToolkit Hi! Is there an existing trigger for git commit? should i create a custom one? Thanks!

[@DevOpsToolkit]

In case of Argo, you do that by defining Argo Events.

[@amitgelber2060]

​@@DevOpsToolkit not sure i understand, there is no git trigger for argo events. ( there is a sensor per my understanding) argo events will listen to kafka and use which trigger? thanks again

[@idoyogev2604]

Hi Viktor, First of all, thanks for all the video content and the amount of investment to create them. Keep the hard work, much appreciated !!! Can you please add a video on how to set up Prometheus and Grafan (or any other tool you prefer) in a K8s cluster configuration? Also, is it possible to save the Prometheus ingest data and Grafana local DB outside of K8s to not lose history data? If yes, what is the recommended approach to achieve this? Kindly Regards and keep safe!!!

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@idoyogev2604]

@@DevOpsToolkit Thanks, that would be really wonderful !!!

[@harrydam_]

​@@idoyogev2604 ​ @DevOps Toolkit Sorry but do we clear this question from TODO list :D if yes, could you pls send me the video for it? I significantly appreciated it.

[@DevOpsToolkit]

Something like kzbin.info/www/bejne/joPCqoqihKp6n6M

[@harrydam_]

@@DevOpsToolkit this is cool

[@olegk3930]

But how ArgoCD in ArgoCD on 11-15 minute? 0_0

[@artiston1]

Good Day Viktor! Why after appling apps.yaml I can not longer see argo-cd UI. It returs a 404. FYI before applying apps.yam I was able to see argo-cd UI. Thanks for the videos! I´m a new learner!

[@DevOpsToolkit]

I can't say why that is happening. Kubernetes might have kicked out the pod of the UI due to insufficient resources. Or it could be any other among an infinite number of reasons. I suggest you check out the statuses of the pods running in the cluster.

[@artiston1]

@@DevOpsToolkit Thank you for the fast reply! I really appreciate that! All pods on argocd namesapce are up and running. No errors. Does istio chage something that could affect the ingres url as defined on overlay->production-> ingress? I'm using minikube tunnel, not sure if that could be the issue

[@DevOpsToolkit]

Did you, by any chance, install istio gateway? That might be using the same port as ingress.

[@artiston1]

@@DevOpsToolkit Yes when running apps.yaml it installs istio gateway.

[@DevOpsToolkit]

That might be the cause of the problem. Typically, each would create a separate external LB but that's not the case on minikube. Check the services and see which ports they expose

[@lemuelnabong3517]

What is the minimum CPU and memory do I need to run this on my laptop?

[@DevOpsToolkit]

To be honest, I never measured what the minimum reqs are. I know that all those tool use very little memory and CPU and I can't tell you the exact reqs. Now, that applies to those tools at rest. When you start using them, Events, CD, and Rollouts will not increase their footprint but Workflows is different. It spins up a Pod for every step in your pipeline, for every build. How much many resources such a pod will consume depends on the images you use in the steps, as well as how many builds you run in parallel. If your cluster is elastic (e.g., with cluster-autoscaler), that should not be a problem either, unless you do start using images in your builds that are unreasonably big.

[@lemuelnabong3517]

@@DevOpsToolkit yup, just wanted to know that i can test it with the smallest app to deploy possible :) Great demo and resources man!

[@lemuelnabong3517]

30G of free memory at least (estimate). I will try it with linode and update you I can successfully deploy up to argocd w/out projects with my 16G macbook pro, it shuts down when i try to deploy apps in argo :D

[@DevOpsToolkit]

Did you allocate all 16gb to the local Kubernetes when trying it out? Normally, Docker Desktop, Rancher Desktop, or whatever you're using to run kid locally has only a fraction of your laptop memory allocated to it.

[@DevOpsToolkit]

When you deploy Argo projects, execute `kubectl top pods --all-namespaces` to see how much CPU and memory they're using. I bet it's way below 16 GB at rest (without Workflow build Pods and other apps).

[@ziaurrehman4738]

We deployed some applications on one cluster using argo cd applications but after some time one of team member deleted the cluster but didn't delete the application from ArgoCD. Now those applications are showing in unknown state. How to delete them

[@DevOpsToolkit]

I think that removing the finalizer annotation should fix that.

[@ziaurrehman4738]

We have already removed those manifest from Github as well. Then where from can we remove those finalize annotation

[@DevOpsToolkit]

Edit the Argo CD application manifest from inside the cluster where it's running. Since you can see the app in Argo CD UI, it must have been applied in a cluster other than the one that was removed.

[@ziaurrehman4738]

On the cluster where those applications was deployed that's cluster is deleted and manifest of argocd applications from Github as well removed. Right now on the argocd UI those applications are showing unknown state

[@DevOpsToolkit]

If it's showing in the Argo CD UI, it means that the Argo CD application manifest is in that cluster and not the one that was deleted. Execute 'kubectl get applications -A' in the cluster where Argo CD is running and you should see the app you're looking for.

[@lextr3110]

What is your take on argocd autopilot? is it legit?

[@DevOpsToolkit]

I was involved in early stages of the project (while I was in Codefresh), but I haven't used it since then so I cannot comment on it.

[@bogdan_angh]

After a lot of trying, I am unable to configure argocd-server on GKE. I am doing everything in the video, and, when trying to access argo-cd.$INGRESS_NGINX_ADDRESS.xip.io the connection halts and I get no result. While taking a look through the logs I saw that there was this error "Translation failed: invalid ingress spec: service "argocd/argocd-server" is type "ClusterIP", expected "NodePort" or "LoadBalancer" " so I patcher the argocd-server Service from type ClusterIP to type LoadBalancer and changed the Ingress host with the new LoadBalancer's External IP Address, but the result is the same; no connection. If anyone managed to deploy argo-cd on a GKE cluster, I would appreciate any help! UPDATE: This update comes a minute after writing the above comment. It's weird, but if I access just the new LoadBalancer's IP address, it redirects me to the ARGO-CD server login page. If, instead, I try to access argo-cd.$LOAD_BALANCER_IP.xip.io, it freezes. Viktor, do you have any idea why is this happening?

[@DevOpsToolkit]

Since a day or two ago, I noticed that there is something wrong with `xip.io`. Can you try the same, but with `nip.io` (it's almost the same service)?

[@DevOpsToolkit]

I just updated that and other Gists to use nip.io. Please let me know if it worked in your case as well.

[@bogdan_angh]

@@DevOpsToolkit will try this weekend and let you know! But what are your thoughts about needing to patch the argocd-server to LoadBalancer? Which one of the IP addresses should I use? The NGINX ingress controller or the argocd-server LoadBalancer's ?

[@bogdan_angh]

@@DevOpsToolkit and btw, the fact that you're replying and helping people so much is awesome and you rock 🤘

[@DevOpsToolkit]

You should not need to patch the server service. Ingress service should already be lb type. I run it myself in GKE and it worked. If you run into an issue after changing to nip.io, we can take a look at it together.

[@bala9823]

Link to the gist please

[@DevOpsToolkit]

Unfortinately, It stopped working due to some updates to the apps involved so i removed it until i redo it all.

[@jirityr]

Nice video but it's not "everything". You are completely missing the automation of the cloud resources (VPC, Kube cluster, buckets, databases, ...) which are then used by the application running on top of Kubernetes. For example how do you automate passing IP/domain name of a database to the application? How do you manage pod permissions (Workload Identity in GCP, IAM role for service account in AWS)? Unless you can run all that as a pipeline, it's only a partial solution.

[@DevOpsToolkit]

You are right. I did not present pipelines with examples to automate everything. Rather, I showed a mechanism how you can automate everything and now it is up to you to add the specific steps/commands you need to execute as part of your pipelines. In other words, you can automate everything through pipelines including the examples you mentioned, but I could not go through all the permutations in a short video. For your use cases, you probably want to define those resources as terraform manifests and execute the steps through a pipeline whenever you push changes to the repo where those manifests reside. On top of that, next Thursday will be a video specific to infra for which you might not even need pipelines.

[@DevOpsToolkit]

... actually, your examples are a combination of terraform and Kubernetes manifests since part of the work is done inside Kubernetes, and the other part outside. For the things that are inside, you do not need pipelines. Argo CD should do.

[@jirityr]

@@DevOpsToolkit Thanks for your reply. It would be great to see how you can marry the two worlds via ArgoCD peipelines. Especially how to use Terraform outputs in Helm values.

[@DevOpsToolkit]

@@jirityr Do you mean something like `helm upgrade --install --set something=$(terraform output something)`? You can do that, but you probably shouldn't. Infra values rarely change. Most of those that are used by apps tend to be the same since the moment a resource is created. For example, if you create a cluster, you will access it through a DNS or a fixed IP of an LB, so that will not change over time. You might increase the number of nodes, spread it across zones or event regions, and that address will still be the same. If that's the case, I would rather store Terraform output in a file than "discover" it over and over again. In your examples, that file would be Helm `values.yaml`. As a side note, Argo CD doesn't have pipelines. That's Argo Workflows.

[@jirityr]

​@@DevOpsToolkit It's not only about using domain names. It's about plenty of other information I need to pass from Terraform to Helm. For example, if I want to use Workload Identity on GCP, I need to create a new GCP Service Account and then pass the name of the GCP Service Account to the Helm chart to create the Kubernetes Service Account with the right Annotation. How do you orchestrate that? The last thing I want to do is to create yet another values file which I have to keep up-to-date.

[@Dongdot123]

I think you should increase the sound volume level of your videos

[@DevOpsToolkit]

Good feedback. I'll do that for the upcoming videos (YT does not allow re-uploads).

[@julianomoraisbarbosa]

😱

[@7912249]

really cool tutorial! thanks a lot! just from my point of view, maybe some tasks can be solved better via jenkins by using groovy scripts. In argocd everything is about configuration of bunch of yaml template files and argo-crds... if a customer want do some mission specific tasks/steps, it requires to build customized images and tools, which is in my opinion little bit more complex than writing a groovy job. maybe the mix of jenkins and argo is a better choice?

[@DevOpsToolkit]

True. A common and, in my case, constant, combination is pipelines with GitOps. I use pipelines for one-shot tasks (tasks that are executed once and only once) and GitOps for tasks that are continuous (e.g., synchronization). Now, whether you use Argo Workflows or Jenkins or GitHub Actions or some other type of pipelines does not make much of a difference. Almost all pipeline tools are, more or less, doing the same; execute tasks.

[@7912249]

​@@DevOpsToolkit yeah absolutely, they are basically doing the same thing. I just wanna to say, with argocd it might require extra tools or yaml files to perform a simple task like "ls -l". But currently there is no better solution than this... As software engineer I prefer to write some code for example with python and pass it somehow to argocd instead of creating a extra tool and test & build & deploy for it.

[@aarnialasaarela5342]

Thanks!

[@DevOpsToolkit]

Thanks a ton Aarni.

[@sidkris7757]

Thanks!

[@DevOpsToolkit]

Oh my. That's very generous of you. Thanks a ton sidkris77.

[@scottwatson4552]

Thanks!

[@scottwatson4552]

Really appreciate your knowledge in helping me get started with everything Argo. Great stuff!

[@DevOpsToolkit]

Thanks a ton, Scott.

[@slavafomin]

Thanks!

[@DevOpsToolkit]

Thanks a ton, Slava.

[@slavafomin]

@@DevOpsToolkit thanks to you, Viktor, your videos help tremendously. I can say that they are much more helpful than some of the paid courses. By the way, could you, please, recommend a strategy for reducing LB costs in GKE? I'm planning to have dozens of microservices in the near future and each LB cost starts from $18, even for internal ones. And I'm afraid that I can't justify such expenses to my clients, especially, when comparing to legacy monolithic solutions :)

[@DevOpsToolkit]

If you manage all your hosts through a single ingress, there will be only one lb.

[@slavafomin]

@@DevOpsToolkit that make sense, however, how do I specify references to multiple services from my Ingress, when the Ingress and services are located in different namespaces? E.g. the Ingress is in the "default" namespace, while ArgoCD is in the "argocd" namespace and something else is in yet another one. According to the Ingress definition all referenced services must be in the same namespace as Ingress itself.

[@DevOpsToolkit]

Not at all. You apply an ingress controller and, from there on, define as many ingress resources as you want, in any namespace. That being said, if you prefix the service name with the namespace, ingress resource and be elsewhere. Still, that is rarely used and not really needed. The best and the easiest way is to keep ingress definitions together with the rest of resources of an application (preferably in the app repo).