Argo Rollouts - Canary Deployments Made Easy In Kubernetes

[@DevOpsToolkit]

Are you practicing progressive delivery (canary, blue-green, etc)? If you are, which tool are you using?

[@internationalvillager1990]

ARGO

[@mu.abdelhamid]

This channel should have 10 times more subscribers. Really amazing content!

[@underlecht]

Amazing that you also showed the failing version which i think is very important, and for which the developers and devops have to be prepared. Great toolset is Argo, I think I am going to implement it to my project one day to try it out... Thank you:)

[@dfhmsd]

Thanks for the video, gonna try out Argo Rollouts.

[@DevOpsToolkit]

Ping me if you run into any issues or you have additional questions.

[@vipuljindal1033]

Thanks for your amazing videos. Would be great to have a canary deployment video with Argo + Flagger.

[@DevOpsToolkit]

I've been thinking about doing that, but I thought that not many would be interested. The majority of the teams I'm in contact with tend to choose Argo CD + Argo Rollouts or Flux + Flagger. I guess that for them it makes sense to keep it in the same "family".

[@vipuljindal1033]

@@DevOpsToolkit In my opinion using Flagger will save some effort as most of the resources are created by flagger while doing Canary deployment which has to be created manually in Argo Rollouts. I would love to hear your opinion on this? PS. I have just started with Gitops.

[@DevOpsToolkit]

That is true. With Flagger you can write fewer lines of yaml but I do not think that matters much. That is not it's primary purpose. That also means that it is less flexible.

[@vemulasrinivasrao6250]

Tq for wonderful videos is crazy and how can i handle the database migration at this moment..

[@DevOpsToolkit]

I would suggest Atlas for that. You'll find a few videos about it on this channel.

[@kousikchatterjee3977]

That's really awesome. Absolutely fantastic! could you please share the gist for this demo? It will be great for me. Thanks!

[@DevOpsToolkit]

Unfortunately, I do not have a gist for that one. Shortly after I published it, I made a rule that every video should have a link to the gist in the description.

[@kousikchatterjee3977]

@@DevOpsToolkit Okay...No problem. I am a big fan of your channel. I never missed any video that you uploaded. I learned lots of Kubernetes things from you. Thank you once again.

[@stanrock8015]

I think I've become a super fan of your videos. Keep them coming. Would like to see some videos on Tekton and how to perhaps integrate what Tekton does well with Argo.

[@DevOpsToolkit]

That's quite a coincidence... I just started working on a video about Tekton. If everything goes as planned, it should be published Thursday next week.

[@stanrock8015]

@@DevOpsToolkit I haven’t seen this and I’ve been looking. Am I missing it or still coming?

[@DevOpsToolkit]

Not yet fully finished... I hope I'll be able to complete it and publish it in a few weeks.

[@DevOpsToolkit]

I just published a video about Tekton in kzbin.info/www/bejne/bZ7Zo6Our8R1nKc. I wanted to do it sooner, but life got in the way.

[@quentinlro]

Thanks for your great videos, books and so on ! Would be awesome to have a canary deployment video with Argo Rollouts + Linkerd

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@rajuodela9299]

Good one.thannks for the video. I just need some more information about spec.strategy.canary.analysis.args.value in rollout definition file. What is the need of passing arguments here and how are you passing them from a SVC file.

[@DevOpsToolkit]

args are variables that you can use to pass values to analysis templates. That way, you can reuse templates without repeating yourself.

[@vincentyin6245]

Super! I see a practical problem, though. Argo Rollouts requires us to eliminate `kind: Deployment`; rather, switch to `kind: Rollout`. That is OK as long as we are writing our own manifests and our own Helm charts. However, in practice, we often use 3rd party Helm charts to deploy all kinds of stuff. And they use `kind: Deployment`. Are we then stuck? In comparison, Flux/Flagger's design is to create a `kind: Canary` that references an existing `kind: Deployment`. That means it can work in the above scenario. It's a better design in this regard, then. Am I understanding it right?

[@DevOpsToolkit]

You're right. Flagger might be a bit more tedious to define but it is more flexible.

[@kalyanram100]

Can you post a video which shows a blue green deployment where traffic switch happens based on successfully smoke test execution in fully automated fashion??

[@DevOpsToolkit]

Adding it to my TODO list...

[@rakeshrao7827]

Nice explanation,

[@rakeshrao7827]

Want to try this, can u share the repo. Will see n give a try.

[@DevOpsToolkit]

Unfortunately, that is one of the earliest (serious) videos I made and, at that time, I did not start adding Gists to the description. Most of those that came afterwards have it, and all the future ones will have a Gist as well. If it's of any help, you can try the course that, among many other things, has detailed instructions for using Argo Rollouts. You can use www.udemy.com/course/devops-catalog/?couponCode=7EA8458864D610209468 to get the lowest price Udemy allows me to give.

[@sbj7452]

how to use kube configmap across preview/active services. Right now my deployment spec has 1 configmap and after setting up argo rollout we might want a seperate configmap for testing preview service. How can we achieve that? For eg: I might want to test a feature in preview service which is controlled by configmap entry. But if i change the configmap entry , active service will also receive that updated flag which i would not want.

[@DevOpsToolkit]

Are you using a single config map for all app instances? Is that the issue?

[@milindchavan007]

This is amazing! Can you share the ymls?

[@DevOpsToolkit]

Unfortunately, I started creating Gists that contain all the commands and a reference to the repo with all the manifests after that video. So, I do not have it for that one, but all those done afterwards do have it.

[@rogeriosilvarocha]

Great men+++

[@mpagouli]

Thank you for your videos! They are great and very useful! In case of an AKS Cluster with Azure Monitoring enabled (more specifically, with Container Insights that also scrape Prometheus metrics), is Prometheus installation still necessary for ArgoRollouts Analysis to work for Canary deployments? As I understand it, if no Prometheus Server is deployed in the Cluster no prometheus provider queries will work in an AnalysisTemplate manifest and I cannot find any examples where KQL can be used for such queries. So, in order to use ArgoRollouts Analysis for Canary deployments in AKS, we have to install Prometheus, even though Container Insights already provide some of its functionality, right?

[@DevOpsToolkit]

It does not have to be Prometheus. Rollouts supports quite a few other metrics stores (visits the Analysis sections of the docs). However, Azure Monitoring is not one of them. Typically, support comes for those that are commonly used or for those vendors contribute to, and Azure is neither.

[@mpagouli]

@@DevOpsToolkit thank you very much for your answer! Of course, ArgoRollouts provides many options. Just wanted to make sure I am not missing any Azure specific guidelines. Again, thanks a lot for your great work!

[@M79L]

hi master, question from padawan of yours :) I'm wondering (maybe completely wrong). Lets imagine, that we have lots of microservices in the cluster and testing one of those(will call it SERVICE). but, this SERVICE is not in the first row from the Ingress point of view - meaning, this SERVICE cannot be directly called by Ingress. only other services call this SERVICE. am I missing something, because we cannot re-configure the other services to use the -canary version (or -green) version because we still want have also request arriving to previous version. in the devops toolkit udemy course, you're using traffic management tools to switch some traffic to old version and some to new version. But how can we achieve this for service, which are only called by other services in the cluster? (besides the scenario with basic k8s service, where we have only one service...) Thank you for helping us learn new interesting stuff

[@DevOpsToolkit]

It's still the same. You redirect part of the traffic to a new release and the rest to the old. You evaluate metrics, and depending on the outcome, you move forward, or you revert. Now, it should not matter whether that traffic is coming from outside (e.g., a browser) or from the inside (other services). The process does not care (or should not care) where the traffic comes from.

[@M79L]

@@DevOpsToolkit Thank you for your answer. what would you do in the scenario, where we do not have istio in place, meaning we dont have all of our services configurable by virtual services. We have just Ingress controller + ingress and then couple of services. but we want to test service, which is not behind ingress, but can be only called by other services.

[@DevOpsToolkit]

@@M79L You would have a hard time doing it without some sort of a service mesh. As a result, you'd spend more time building your own "solution" instead of just installing service mesh (e.g., Istio, LinkerD, etc.). The thing is that Kubernetes service does not have a mechanism to define weights or fine tune what goes where. You need something on top of it and if that something is not a service mesh, you'd need to build it yourself.

[@M79L]

@@DevOpsToolkit Thank you Viktor! now its clear to me.....

[@ioannisgko]

Thanks, great content as always! Do you know if there are problems installing Argo Rollouts with Kubernetes v1.22.2? I am on a new cluster, my ingress is working fine, but when I apply the rollouts install yaml file I get a "Failed to watch *v1.beta1.Ingress: The server could not find the requested resource (get ingresses.extensions)" Sorry if it is the wrong place to ask 🙂

[@ioannisgko]

I just found that this is a problem only when using Kubernetes v1.22.2 due to ingress deprecations, there is a PR 1529 that fixes it

[@DevOpsToolkit]

The manifests you're using are probably referencing the old Ingress API that has been removed from 1.22 (and deprecated a while ago). Can you share the command you used to install it so that I take a closer look?

[@maxmage102]

Could you make a video of Argo Rollouts Canary deployment by headers using kubernetes nginx ingress controller :)

[@DevOpsToolkit]

Adding it to my TODO list...

[@maxmage102]

@@DevOpsToolkit Awesome! Thanks :)

[@barefeg]

Is it possible to use Argo rollouts for non-trivial cases like this? What if you have multiple micro services each with multiple downstream dependencies? How would you manage rollouts in that situation?

[@DevOpsToolkit]

That depends on whether by microservice you mean independently deployable applications or not (e.g., distributed monolith). If it's former, the process is the same. Each is deployed independently and the process decides whether to roll forward or roll back for each microservice independently of others. Now, if those apps need to be deployed together (distributed monolith), the situation is a bit trickier and it depends on many factors (e.g., do you have sticky sessions, whether apps are backwards compatible, etc.). Ultimately, canary deployments might not even be a good fit. If you can explain your situation a bit better, I'll do my best to give an answer that makes sense in your use-case. You can contact me on Twitter (@vfarcic) and we can have a chat if that sounds better.

[@barefeg]

@@DevOpsToolkit thanks a lot for the quick answer. Indeed I’m thinking of independently deployed services, but the problem is a bit more subtle. Imagine you have two services A and B, and A calls B. Then in As configmap you put Bs service name. If we do blue green deployment then the service names differ depending on whether it’s the stable or the preview version. So it’s not clear how you will achieve that As preview talks to Bs preview. The promotion process is even more tricky. Say you promote B, then the stable version of A might not work correctly anymore since it’s pointing to the new version of B now, etc

[@DevOpsToolkit]

If, for example, you're using Istio, you would not have A calling the Service of B. Instead, you would have A calling Istio VirtualService of B. That VirtualService will forward requests to B (through one or more "normal" Services). During progressive rollout like canary deployments, Argo Rollouts (or Flagger) will be changing the weight of the VirtualService so that part of the traffic is going to the new release and the rest to the old. The situation is similar with blug-green deployments. You cannot simply change one app to talk to a Service of a specific release (color) of another. That would assume that changes are instant (they never are). Instead, you should have something like Istio VirtualServices so that all new traffic is going to the new release but that all existing traffic is still handled by the old. All in all, in the example of Istio, one app is communicating with another through VirtualServices, and NOT "normal" Kubernetes Services. Istio is, ofcourse, only an example, and you can accomplish the same through other service meshes (e.g., SMI) or without them (e.g., through Ingress or some other proxy). The important note is that nothing is ever instant and whichever strategy you choose, you need to assume that there is a transition period during which both the old and the new release are handling traffic. That is just as valid with blue-green as with canary deployments. The only tangible difference is in the duration of the process. While blue-green process might take a few seconds, or even a few milliseconds, canary lasts for minutes, hours, or, sometimes, even days. Nevertheless, both are iterative and assume that releases are running in parallel (no matter how short or long that is) and, as a result, that they are backwards compatible.

[@barefeg]

@@DevOpsToolkit thanks for the input. I see that the solution can be achieved with a service mesh, which I have implemented before. But I wanted to find a more lightweight solution to release engineering without a SM. Argo rollouts seemed like a good fit but I think it’s only good for simple flat architectures

[@DevOpsToolkit]

You might want to try it with NGINX Ingress instead of Istio (argoproj.github.io/argo-rollouts/features/traffic-management/nginx/). However, that would probably not solve your problem since it deals with traffic from outside the cluster and not the internal one. If canary deployments are not a good fit and, as you mentioned, blue-green is a better choice for you, Argo Rollouts might still be useful. I demoed canary deployments but it also supports the blue-green strategy (argoproj.github.io/argo-rollouts/features/bluegreen/).

[@julianomoraisbarbosa]

👏

[@DevOpsToolkit]

You're binge watching :)