Flux CD v2 With GitOps Toolkit - Kubernetes Deployment And Sync Mechanism

Flux v2 is a tool for converging the actual state (Kubernetes clusters) into the desired state defined in Git. It is a GitOps-based deployment mechanism often used in continuous delivery (CD) processes.
Timecodes ⏱:
00:00 Intro
01:39 Creating environment repos
06:39 Bootstrapping
11:34 Creating sources
20:08 Deploying the first release
25:39 Deploying new releases
27:18 Promoting to production
30:06 What did I do?
32:16 Conclusion
➡ Gist with the commands: gist.github.com/0431989df4836...
🔗 What Is GitOps And Why Do We Want It?: • What Is GitOps And Why...
🔗 Argo CD: Applying GitOps Principles To Manage Production Environment In Kubernetes: • Argo CD - Applying Git...
📚 DevOps Catalog, Patterns, And Blueprints: www.devopstoolkitseries.com/p...
📚 Books and courses: www.devopstoolkitseries.com
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox

[@kenichishibata2411]

I love how forthright this is, we need more people having intellectual honesty like you :) great job!

[@DevOpsToolkit]

Thanks. I'm glad you liked it and I hope it was useful.

[@n4870s]

@@DevOpsToolkit So basically, flux and argo cd monitor both source code repos and environment repos and trigger a deployment to the cluster. What if the env repo is configured with specific container image tag version and we do a source code change? This should result in new image version being created, but since the env repo is configured with specific image version it shouldn't trigger a deployment, right? Also, who is building the new image version on git push in the source repo? And i suppose it is going to com image registry, then does argo/flux know about this ICR?

[@DevOpsToolkit]

@@n4870s That is correct. Source code of apps does not matter in that context. ArgoCD/Flux will only react when you change manifests they monitor (e.g., image tag). New images are built with whichever tool you use to build images. That part is not really in the domain of ArgoCD/Flux. Normally, you'd do that through pipelines. Now, ArgoCD/Flux can either wait for those pipelines to push changes to the manifests repo, or react to pushing images to a registry. I prefer the former. So, my pipelines would, among other things, build images, push them to a registry, clone the manifests repo, change a manifest of an env., and push the changes back to the repo. From there on, ArgoCD/Flux would be making sure not only that those changes are applied once, but that they are always representing the current state.

[@PahaUsd]

@@DevOpsToolkit yes, I would like to review the Russian analogue system of flux - werf - there is no information about Russian solutions at all, as well as North Korean ones))

[@PahaUsd]

@@DevOpsToolkit oh sorry - there is already about werf on your channel - I will watch

[@sujaydey9160]

Just started exploring this channel and I am already loving it❤

[@JohnMatthew1]

Thanks for the demo and enthusiasm! Also, i was unclear on how to do a helmrelease, so thanks for showing that :)

[@SuperCruick]

Really nice Demo / Tutorial.

[@kazakman7772]

This is ok for beginners hands on Keep in mind in real Corp world it is different when u got 5 clusters with 5 different repository U need helm chart to install them and make sure they are syncing with ssh keys

[@StephenSpencer1972]

Indeed. I didn't see the original version. This one is a very well done introduction! Also, thank you for the (gentle) acknowledgement of the quality of flux's documentation ^relative to the code and function. The kube ecosystem is littered with so many hyperbolic claims of the This-Will-Change-Your-Life variety that worthwhile projects can be overlooked simply because the documentation is poorly written and/or poorly maintained. (and in case any fluxcd contributors are reading, try to limit the use of the word "declarative" to no more than twice on any given doc page :grin:)

[@roguelitedev]

You are amazing!!! Keep doing what you're doing!

[@DevOpsToolkit]

Thank you so much!!

[@bsops]

Thanks Viktor, good dive in

[@hact8907]

great explanation you help me lot

[@kevinsaffarian3017]

I liked the demo. Thanks

[@senpai70]

I watched other Flux v2 videos and this is by far the best!

[@DevOpsToolkit]

Thanks.

[@zubairhaque2706]

My friend from cloud native east Asia, said this was the best flux video as well! Do you recommend doing flux with helm ? Or just flux by itself

[@DevOpsToolkit]

You do need a way to define Kubernetes manifests beyond "pure" yaml. That means either helm, kustomize, or tools that output yaml like ytt or cdk8s. Flux works with any other those options (helm, kustomize, yaml output through other tools) so it's more a question of what you prefer. Personally, I go with kustomize for my apps and helm for third-party apps.

[@zubairhaque2706]

So I’m stuck on one thing why do you have two repos for staging and production? Why not just have one repo

[@DevOpsToolkit]

It could be different directories in the same repo. It depends on one's preferences. Personally, I prefer one repo per environment. It simplifies differentiation with, for example, PRs.

[@bobbyray8457]

Really loving your videos man. Keep it up. Also -w in kubectl will watch, no need to use watch cmd

[@DevOpsToolkit]

`kubectl ... -w` tails the of the state instead of showing the latest output. For me, the latest output provided from `watch` is easier to follow.

[@edwindavidcontreras4615]

thanks for sharing this amazing presentation !!

[@DevOpsToolkit]

My pleasure

[@90hijacked]

weekend project ❤❤❤

[@diegonayalazo]

Thanks Teacher! ❤

[@TAICHI1SCO]

Great content as always, thanks Would love to see more content on Flux and Argo CD on here or Udemy if possible.

[@DevOpsToolkit]

I was afraid thst I put more attention to Argo projects (including CD) than other topics on this channel. There are probably 10 videos exploring Argo (directly or with other tools). Do you think it warrants more? If you do, I would be happy to make some more.

[@TAICHI1SCO]

@@DevOpsToolkit Great. I see now there is a fair bit of Argo lessons to consume on the channel. Thanks

[@QnJhbQ]

Thank you so much for your videos! I really like them. It would be a bit improvement to show Git / Directory context in the PS1 of your shell so it's easier to follow. Just a thought :)

[@DevOpsToolkit]

That's a good idea. I'll try to remember that in the upcoming videos.

[@webmakaka]

Hi! Thanks for videos! Can you also write a video about FluxCD 2 + Flagger in the future. Thanks!

[@DevOpsToolkit]

Just added it to the list. I did create a lot of material about Flagger (www.devopstoolkitseries.com/posts/canary/) but not combined with Flux v2. I just added that as a topic to my TODO list. Can't guarantee when it'll come though (the list is growing faster than I can handle).

[@vincentyin6245]

Super! You said at the end of the video that you might make another video "in 6 months" about your up-to-date opinion of Flux vs. ArgoCD. That timeline has arrived. I'd be very interested to hear your current opinion. One particular problem I can't find a solution for is that Flux doesn't seem to have an equivalent to the command `argocd app wait ...` or `helm upgrade --wait ...`.

[@DevOpsToolkit]

In the meantime (since that video) I used almost exclusively Argo CD for all "real-life" projects and most of the companies I'm working with are also using Argo CD. So, I haven't been in as much contact with Flux as I should be to make another comparison video. I'll do my best to change that and spend sufficient time with it to make it fair.

[@ashhere31]

Thank god you have favorable opinion of Flux v2 now. I used it and thought it was an excellent improvement over Flux V1. And having read your books, it matters to us what you think is ok and what you think is great related to DevOps and K8s So appreciate this video and candid opinion

[@DevOpsToolkit]

One of the biggest problems I have, especially with the books, is that the tech changes very fast. I am often in the situation when I change my opinions (sometimes drastically) from one month to another. That's why KZbin videos are becoming a better format to share than courses and books. They are shorter, I can publish them weekly, and change my opinion frequently. Books and courses, on the other hand, take around a year to create and publish each.

[@DevOpsToolkit]

Flux is a good example. I do believe that Flux v1 was not a good option, but the situation changed drastically with Flux v2. Now both Flux and Argo CD are almost equally good. A month or two for now, the situation might change again.

[@ashhere31]

@@DevOpsToolkit totally makes sense. this new and innovative subject areas are still going through a faster churn, until they mature and stabilize to a point that the basic product remains same, only features get enhanced.

[@ashhere31]

@@DevOpsToolkit ha ha........i am yet to experiment with Argo CD. but for now, just want to learn one product properly and implement it for a while to see how it works in our scenario.

[@saurabhheda1302]

Thank you for the video @DevOps Toolkit by Viktor Farcic. I wanted to know if it is possible to perform a bootstrap from the existing GitHub Repo. A cluster where there are admission control policies like PodSecurity and few others requires us to edit clusterole and few other objects in the manifests. Is it possible to bootstrap using the existing GitHub repo? So let's say if we have multiple clusters where the deployment of flux should be done but a customized deployment? Is it possible with the flux bootstrap command?

[@DevOpsToolkit]

It can be an existing repo. Flux installation is all about pull Flux manifests from a repo and applying them so if you already have them customized, it'll pick them up.

[@saurabhheda1302]

@@DevOpsToolkit Yeah I do have a repo, but it makes a commit and edit those changes and revert's all the customized changes.

[@saurabhheda1302]

Pasted the link on LinkedIn.

[@PahaUsd]

werf - a good gitops tool

[@durden0]

Thanks for this. This was a great explanation of flux, much better than the official videos on the flux site itself I think. One question I have after watching is, is it necessary to have separate staging and production repos to contain the helm release definitions? Could those be in the same repository in different directories? What all can go in the same repo? I'm asking these questions because i'm trying to think through the flow a CI/CD pipeline where you commit a change to the app, but then you need to go update two different repos to deploy the change, first for staging and then for prod, which seems like an anti-pattern in CI/CD.

[@DevOpsToolkit]

It is not necessary to have separate staging and production. You can have any number of environments stored in a single Git repo. The specific implementation depends on whether each of those environments should be in the same or different clusters. If it's the same cluster, you can get away with a single `Source` that monitors the root dir of that repo and use releases to separate them into Namespaces. On the other hand, if you use different clusters, all you have to do is create a source with a different path to releases in each. The separation into sources and releases allows Flux v2 to be much more flexible than v1. You can organize repos in (almost) any way you want.

[@durden0]

@@DevOpsToolkit thanks! Gonna spend some trigger trying to wrap my brain around that this week.

[@mirkomarchiori7817]

@@DevOpsToolkit Let's say I have one cluster for each environment, what I would like to have is a single repo/folder where I store the flux-system part of gitops-toolkit so that each cluster "refers" to the same common gitops-toolkit to deploy along with the common git sources that those environments/clusters need to know. The idea is to have a sort of central gitops-toolkit configuration (a repo or a folder in it) that the clusters take in order to setup their gitops, because they are related to the same "project". Then I would have specific kustomizations and helm releases on the repo/folder corresponding to the specific env/cluster I want that "version" of the application to deploy. Is it possible? In other words, is gitops able to distinguish when to deploy something based on the env of its cluster (since in this hypothetical case it is based on the same common gitops confguration), is it even possible to do this? Or am I "limited" to have one separated flux-system folder for each environment/cluster I need? I hope my doubt is clear :)

[@DevOpsToolkit]

@@mirkomarchiori7817 Normally, you have a helmrelease or kustomize definition for each setup. It could be the same, but that would prevent you from changing the desired state of one cluster, testing it, and then applying the same in another. In other words, you can have one flux-system for all environments/clusters, but I wouldn't recommend it. It's much safer and easier to trace if releases for each cluster are separate so that you have control what goes where and when. The amount of YAML is very low since releases are only references to YAML and/or Helm charts with a few values you might want to override. Those values are another reason for not keeping it all together since there are always some differences between envs/clusters. So, I would recommend keeping the bulk of definitions in unique and not duplicated, and separating only the releases. That separation can be in different directories, different repos, different branches, etc. I'm not sure whether I answered your question. I have a feeling that I might have misunderstood you.

[@mirkomarchiori7817]

@@DevOpsToolkit Thank you for the response, yes it was on point, now on a second thought I get why they went for that route, given that everything is actually a "link" to other repos/charts, it is ok to separate things like that.

[@nikolanovkovic4103]

Hello, At @7:35 "the path where we will store the definitions, the manifests is 'apps', remember we created that directory before"... Can you explain this? You've created apps directories in production and staging repo. Now, you are initialising new repo. How are 'apps' folders between theese three repos conected? Great content btw, keep up the good work!

[@DevOpsToolkit]

The `flux bootstrap` command creates a repo (unless it already exists) and the main purpose of that repo is to hold the manifests of Flux itself. That way, any changes to the Flux setup go through that repo and follow GitOps principles. From there on, you can use that same repo for any other app running in any env. Flux does not care (much) where the manifests are as long as it knows where to find them. It gets that information through `Source` manifests that you can create manually or through the `flux create source` command. So, there is a relation between the source (Git) and a cluster and it's up to you to choose how all that is organized (e.g., different dirs in the same repo, different repos, etc.). Is that what you meant?

[@c051ne]

It would be much more clear if you had your path and GitHub repo/branch for the current directory displayed in your prompt. Otherwise good content, thanks.

[@DevOpsToolkit]

Yes, you are right. I'll make sure that, starting from the next video, it is clearer what I did in the setup.

[@Digalog]

What about: When you build your master. Let's say you have application changes. You increase your app version. Then I suppose you update the chart with the new version and publish it. Would you when updating a release in the flux- repository. manage both appversion and chartversion. when changing the appv ersion in values. a release becomes abstract in the sense of helm release. should you do helm releases instead ? i'm trying to understand how to automate helm release version updates inside the repo automatically via cicd. somehow by finding the right helmrelease file and update either the version or the tag. i'm getting a bit lost in this section. releasing a new chart and updating the version there and just rely on the helm version sounds like the best way forward to me. could mean i end up with sort of duplicate charts where only the version of the app has changed. thanks for the watch! looking at how several people are doing it. and offcourse how mainline projects are doing it. that's where i tried to base my argument on. :)

[@DevOpsToolkit]

Creating and running releases is not the same thing. You can have an application that has releases 1, 2, 3. That does not necessarily mean that any of them is running anywhere. It only means that you built binaries, helm charts, etc. On the other hand, we run specific releases of our (or third-party) applications in environment. We can have release 3 of an application running in a dev environment, release 2 of the same application in staging, and release 1 in production. Flux (as well as Argo CD), allows us to specify what is the desired state of our environment. We define which applications run where, what are exact releases in those environments, and what are environment-specific parameters/arguments. Building container images and creating helm (or Kustomize) versions is how we release applications, while Flux or Argo CD help us define what is running where. The whole idea is to have a declarative way to define the desired state of environments/clusters and, at the same time, have a process that will make sure that the actual state (runtime state of our clusters) converges into the desired state. Going back to your comments... > When you build your master. Let's say you have application changes. You increase your app version. Then I suppose you update the chart with the new version and publish it. That is correct. But that by itself does not mean much more than that. You just created some binaries (container images) and a specific version of a Chart. Actually, creating a new version of the chart is not needed unless something changed in its definition. If all you did was build a new image, you do not need to create a new version of your Helm chart. > Would you when updating a release in the flux- repository. manage both appversion and chartversion. when changing the appv ersion in values. a release becomes abstract in the sense of helm release. should you do helm releases instead? That depends. You update Flux manifests with the changes of a new release. If all you did was build a new container image (and the manifests/templates in the Helm chart did not change), you would keep Flux manifests using the same chart version and update only the tag of the image in Flux manifests related to the environment where you want to deploy that new image. For example, you might choose to run it in staging right away, and in production later. If that would be the case, you would change Flux manifests that describe staging right away and postpone changing those that describe the production. Flux, on the other hand, is monitoring Git repos and, as soon as you change the definition of the desired state it will update the environment that state describes. On the other hand, if you changed the definition of a Helm chart, than you would update Flux definition furst by changing the reference to the Helm chart version you want to have in an environment. > i'm trying to understand how to automate helm release version updates inside the repo automatically via cicd. somehow by finding the right helmrelease file and update either the version or the tag. Most of the time, you are updating only Flux manifests as part of a CI/CD process while updating Helm charts, or any other part of the code of an application is done as the development effort before CI/CD kicks in. In other words, humans are making changes to the code of an application (Helm definitions included) and CI/CD processes are building binaries and updating the definitions of desired states of environments that, in this case, are defined as Flux manifests. Please let me know if that answered your questions. If it didn't, we can set up a conference call and I can walk you through the ideas and the.process.

[@yongshengyang8144]

I have flux installed in cluster. Now I have to update the image tag inside the deployment.yaml file in code repo during my build pipeline running. Is there a good way to do it? I am trying to run some git commands..... thanks

[@DevOpsToolkit]

Thereight be tools that do that, but I never looked for them. I just execute yq inside pipelines and push changes back to git.

[@JackReacher1]

Okay I get what flux cd does but if I change a ConfigMap file can it understand it and reload the pods using that ConfigMap?

[@DevOpsToolkit]

Flux and similar tools do not do anything beyond `kubectl apply`. They do not modify Kubernetes behavior. So, if you change ConfigMap, Flux will sync it with the cluster and that ConfigMap will be up-to-date in your Kubernetes cluster. Now, whether Kubernetes will reload Pods or not is not Flux concern. It made sure that ConfigMap is synced, and that's it. If you need to restart Pods when you change a ConfigMap, you might want to modify the Deployment, StatefulSet, or whichever resource type you're using to manage Pods. You can make a trivial change just for the sake of recreating Pods (assuming that's what you're trying to do).

[@JackReacher1]

@@DevOpsToolkit Nice :) It just felt like they have an inbuilt solution like helm in the background somewhere, so they must have solved that issue too.

[@DevOpsToolkit]

@@JackReacher1 That could be the case. I just feel that would be solved outside Flux. I almost always prefer tools that solve a specific problem and let other tools deal with other issues. That being said, you should check Flux docs. It might be doing that.

[@JackReacher1]

@@DevOpsToolkit One can do this by changing the hash based on the configMap file in the annotations section. That will automatically trigger reloading of the pods, right? I'll check out the flux docs

[@Yggdrasil42]

@@JackReacher1 We use Stakater Reloader to do this. Runs as an operator in the cluster. There are other tools as well that do the same.

[@joebowbeer]

Argo Project is rolling out new versions. Now who has the lead?

[@DevOpsToolkit]

Argo CD and Flux are close. There is no clear winner. That being said, you can probably guess from my videos that I have a preference towards Argo projects.

[@koodauskanava9096]

flux get sources git is giving me this error error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

[@koodauskanava9096]

Fixed by uninstalling flux, adding more scopes to github token and deploying the bootstrap again

[@DevOpsToolkit]

@@koodauskanava9096 Sorry for not responding earlier. Glad it's fixed now...

[@--QS--]

Hi, first of all I appreciate the effort you are putting into your videos. When I tried to follow along I noticed a problem, when adding private repos. If I use the url method authentication is a problem and if I use the ssh method flux creates a dialog, which is piped into the corresponding file. Thereby corrupting it. My Solution is adding private repos manually, which works but is not 100% GitOps. Also: Flag --validation has been deprecated, this arg is no longer used, all resources are validated using server-side apply dry-run Has anybody some ideas regarding this Problem? Best Regards QS

[@DevOpsToolkit]

There are instructions in the docs how to create authentication for private repos. I'll do my best to cover it here as well in one of the upcoming videos.

[@nacho10f]

What's confusing to me is that ideally developers will simply be making changes to the app source code, and not the infrastructure yaml. The app code needs to be containerized, the container deployed to a registry, and the deployment changed to point to the corresponding tag. This solution does not seem to cover that unless I'm missing something.

[@DevOpsToolkit]

You're right. Flux or similar solutions are not covering the whole continuous delivery process (even though they do have CD in their names). You still need to build container images, push them to registries, run tests, and do whatever else you might need to do. GitOps tools are doing one thing and one thing only. They are making sure that the actual state (e.g., cluster) is the same as the desired state (e.g., manifests in Git). Typically, you would have pipelines (e.g., Argo Workflows, Tekton, GitHub Actions, etc.) that would do the process but would not apply any change anywhere. Instead, when the time comes, they would clone the repo with the manifests, make changes (e.g., a new tag) and push them back to the repo or make a PR. Alternatively, you can use one of the solutions that would do the same when an image is pushed to a registry. In any case, GitOps tools are not replacing the whole process, but only a part of it. As a side note, I would not call k8s manifests "infrastructure yaml". They are app definitions like any other, especially if you use things like OAM.

[@manasjitmohanty8959]

Your url for what is HitOps is incorrect.

[@DevOpsToolkit]

You're right. That was a silly mistake on my side. Correcting it... Thanks a ton for letting me know.

[@Digalog]

Why having separate repositories for environments ? why not use branches ? I don't understand entirely this central fleet repository. So I have to configure both a staging system and a production system pointing to this fleet repository meaning both clusters need a secret to update the same git repo. That means you need to fine grain RBAC to the secret on the staging system because if you give (rather you want developpers to have decent ) access on the staging system would give them implicitly access to the same secret to write to the fleet-system for prod.. a security risk for people to change things on the production system may they get access to this secret on the staging system.

[@DevOpsToolkit]

That is only one example. You can organize it in branches, directories, repos, etc. It is up to you to choose what fits you the best. It is not mandatory to follow how I set it up.

[@Digalog]

@@DevOpsToolkit thats offcourse true :) will have to see how it works out.

[@softwareengineer5764]

Any guide how to install Flux CD V2 on ubuntu 20.04. I got stuck :(

[@DevOpsToolkit]

You mean the `flux` CLI? If that's the case, `curl -s toolkit.fluxcd.io/install.sh | sudo bash` should work.

[@softwareengineer5764]

@@DevOpsToolkit Thanks alot

[@rubbercable]

Please increase the recording volume. I have to turn my volume to to 7 to hear you, then my speakers get blown out with the Ads.

[@DevOpsToolkit]

Can you please check one of the newer videos and let me know whether the volume is too low. It should be higher than the one about Flux. If that's still too low, I'll increase it in the upcoming videos.

[@rubbercable]

@@DevOpsToolkit My Bad. It's perfect now. :+1:

[@DevOpsToolkit]

@@rubbercable That's great to hear. It's very hard for me to see what viewers see and experience so please keep coming with comments on what can be improved.

[@eddiecisneros3256]

great tutorial, but at the beginning you are missing some basic steps

[@DevOpsToolkit]

You're right. I tried to be as fast and short as I could, so I skipped few of the initial steps (those related to the setup). That was probably a mistake. I'll do my best to create a tutorial later and keep this one more like an overview. In the meantime, you can find all the initial setup commands I used in the Gist available in the description.

[@kazakman7772]

sorry but ur video is outdated all commands not working. apart that flux bootstrap not working as well whenever u enter tokens. not connecting to github. please update it. remote add origin push -u origin main etc etc

[@DevOpsToolkit]

Can you please let me know the version of your `flux` CLI as well as the output of the command that failed? You can DM me on Twitter (@vfarcic) if KZbin does not let you paste the output.