GitOps: Branches, directories, or different repositories for the desired state of environments?

[@amoenus_dev]

I'd love a deeper dive in file organization across organization. Kubernetes config, common CRDs, templates, common pipeline steps, etc. Would love to see a vertical slice of full enterprise repo/folder/file structure from the ground up.

[@mattc2403]

same, I wish more people covered this. I would also ben interested in how to setup folder structure for terraform repos too.

[@DevOpsToolkit]

Adding it to my to-do list...

[@mihaigalos279]

I love your chain of thought - I had to fight uphill battles in multiple jobs to *not* use long-lived branches, especially not production branches different from main.

[@chrisre2751]

Question Thursday: How many GitOps (ArgoCd) Instances recommended when you staging about multiple environments and different Kubernetes versions? Further details on the scenario: If you have for example 5 environments (Dev, function-test, QA, PreProd, Prod) and around 400 Apps Management by 10 Teams. A major/minor update of the Kubernetes version is not done via in-place update, but via a parallel Kubernetes instance next to the environment and the apps are moved. Is one ArgoCD instance enough? Should you separate Prod and Non-Prod? Should you separate Kubernetes updates in a new ArgoCD? Should you separate GitOps/ArgoCD instances per team or per namespace? Is there antother recommendation if you want to use Argo Workflow or Argo Rollouts. Is it than a good idea to split the Argo Instance between environments Thanks

[@DevOpsToolkit]

Great questions. I'll add them to one of the upcoming videos.

[@MR-ew6uw]

I actually have a meeting scheduled with colleagues next week to explain exactly this to them - many thanks for creating something that allows me to avoid a 'meeting that should have been a youtube video' 😀

[@marwensaadaoui8018]

In some projects, we typically use combination of both, we use a seperate overlay for each of the environments to define manifests, but we also use different branches just to have a promotion mechanism between the envs, so these branches should be eventually in sync. We came to this setup after trying both ways, and the branching strategy is only to have a proper promotion mechanism and no other reason.

[@DryBones111]

@@marwensaadaoui8018 but if you have a long-lived configuration drift (e.g. to test the stability of something) you also then end up with long-lived branches.

[@DevOpsToolkit]

@DryBones111 not necessarily if environments are ephemeral. Even when they are not, I prefer repos or branches for gitops manifests ("real" manifests are still in app repos). Those gitops manifests can reference specific commits, tags, branches, PRs, or anything else.

[@IvanRizzante]

Thanks for another great video🎉 in my opinion developers should have what they need and care about in their repo. They shouldn't care about the insights of Argo or Flux, but they should know how to run their app. I totally agree that having a separate repo for the desired state is a good idea, I go for separate folders because they are easy to use with environment promotion. It's a pity I can't share the fantastic post Codefresh did about it, KZbin doesn't allow me. This new question and answer format is great 👍

[@chrisre2751]

Are you kidding or can you read my search history. I just started today a look up for the topic of the video. Therefore I am very interested and excited to watch your video

[@yackobi]

I have recently been struggling to work this out for myself, so thanks for the great info! The argocd best practices say "Using a separate Git repository to hold your Kubernetes manifests, keeping the config separate from your application source code, is highly recommended". It seems simpler to keep the application manifests together with the source code and just keep the environment configs in a seperate git repo. Any idea why they so strongly suggest the separation and have you hit any big problems by not doing it? As a gitops noob (but experienced dev), it's really difficult to filter all the online info to build a good dev-to-prod gitops pipeline.

[@DevOpsToolkit]

For ne it Is very simple. I have to be able to write code, build it, test it, and run it. To do those things without going crazy, i have to have code, build scripts, tests, abd manifests in the same place (same repo). If manifests are going somewhere else, it would be logical to move the rest as well and that would be just silly.

[@DryBones111]

@@yackobi from my understanding, their reasoning is/was because application code is expected to change much more frequently than configuration. This would make the GitOps controller work harder to perform diffs on applications that haven't actually changed but still triggered a sync via push webhook. There are now ways in which you can optimise the way cache is used with Argo CD using manifest path annotations to prevent this issue from ballooning with a large number of applications.

[@DevOpsToolkit]

@DryBones111 i would argue that we should not initiate argo CD sync via web hooks. That, in a way, goes against one or the four core principles of gitios (pull based mechanism).

[@cedriclamalle]

Great video again! Here we use a repository per application, with everything (monorepo...), like backend services code, frontend code, helm charts, etc. We then have a separate repository with helmfile (very great tool) and a directory per environment. As the same team is responsible for test, staging and production environments, no need for separate repos!

[@DevOpsToolkit]

I don't think that you should keep the charts in a separate repository. Put it where the apps are. If you're using a monorepo, put it there, just as you (probably) keep your build scripts, pipelines, tests, and other stuff in there. Helmfile is not a good choice if you're using one of the GitOps tools since they don't need it and, in a way, replace it (and more). You can just put Argo CD or Flux manifests instead of the Helmfile (in that same repo). Those manifests should reference your charts and overwrite whichever values are specific to whichever environment you're syncing it to.

[@cedriclamalle]

@@DevOpsToolkit The charts are indeed in our monorepo. It's been a long time that I want to try ArgoCD, will have a serious look at it soon! Thank you very much for taking the time to answer, I really appreciate it!!!!

[@DennisHaney]

We use 3 different approaches. Dir for cluster specific services. Dir for internal apps. Files for common services. And finally, a gitsha to point to last known good config of the common services for controlling when production env gets updated

[@shulyakav]

It would be amazing to se your preferred way to do gotops. With examples. Thanks!

[@DevOpsToolkit]

Can you give me a bit more info? Are you looking for a GitOps quickstart or something more concrete?

[@ChewieBeardy]

Thanks for the insights! While I loathe helm, it has something that newer "intermediary languages" lack, which is this idea of the chart as a pre-hydrated artifact. You can define your chart in the app repo and push it to oci alongside the docker image, and the gitops repo can "instantiate" it and supply the values at the last mile. Tools like cdk8s go straight from code to fully hydrated yaml, and you're left with having a CI to do a "build step" in your gitops repo to make a build per env, or rely on kustomization overlays which are a bit verbose. I'd be curious to hear your thoughts on ways to represent this 3 step process (intermediary language -> env agnostic artifact -> env specific instantiation) with tools like cdk8s or kcl!

[@DevOpsToolkit]

If you need to store it as an oci image you might want to try Timoni which does that and is based on CUE.

[@ChewieBeardy]

@@DevOpsToolkit Indeed, Timoni does have this lifecycle and packagin aspect! Unfortunately, we're still waiting for the flux controller to take advantage of it in a gitops way. The more I think about it, the more I think the "correct" generic approach is to represent that hydration phase in a CI phase of the gitops repo: the app repo provides a "package" (in cdk8s it would be a construct, in kcl a schema, ...), and the gitops repo has a CI step on main to import it and provide it with the values to generate the yaml, then push one OCI artifact per env, using tags to distinguish them. It's a bit of a heavy-handed approach though, but I don't see an easier method. It's a shame that all those tools provide docs to generate one set of manifests, but never talk about the situation where you want to make them vary per env!

[@DevOpsToolkit]

@ChewieBeardy I tend to do that. I prefer kcl to define complex scenarios but i añwsyd keep pure yaml in git as the desired state. For me it is important to be able to deduce the desired state and that's very difficult by observing values and trying to figure out what would be the outcome if they are applied to kcl/helm/Timoni/etc. In other words, write it in any format you're comfortable but store the desired state in an easy to read format (yaml or json).

[@ChewieBeardy]

@@DevOpsToolkit oh, that's a great point. Just to make sure I understand, where do you run kcl in that scenario? do you generate the final yaml in the app repo, or at the last moment in the gitops repo? It's the part that eludes me, the ideal place to merge the kcl schema with the env values and produce those yaml, and what should trigger it.

[@DryBones111]

@@ChewieBeardy this is something that I agree is sorely overlooked in a lot of popular tooling. I had to banish CDK because of it. Thankfully there's a new wave of well thought out configuration tooling coming out but I do find it very interesting that this is something that is still being "invented".

[@arniom]

I would love a (visual) example of each way in action to have a better understanding

[@DevOpsToolkit]

I'll do my best to make a similar video with visuals.

[@zoop2174]

I once worked for a company which used branches for different environments. Contained Terraform & Manifests. They ended up with multiple "prod" branches somehow.

[@andud]

One other supplementing rule is that ANYTHING that is deployed ANYWHERE should first touch mainbranch. If it's good enough for ev testing - it's good enough for main branch

[@samarnagar9699]

Love you and your Channel

[@zenmaster24]

you can use b ranches for different environment manifests that are mergable - having an environment specific named manifest can exist in all branches but can be targeted in cicd when a commit to the required branch is detected. this would be pretty much the same as the environment specific directories you mention. in this way you can have all manifests mergeable to the main branch and that wont interfere with each other.

[@DevOpsToolkit]

That's true but, in that case, I'm not sure why use branches if targeting those same files in the mainline produces the same results (which is what I do). Now, to be clear, I'm referring to repos that contain only GitOps manifests and, in that case, workflows are not triggered by commits since those commits result in Argo CD synchronization which might not start, and certainly will not end, at the time commits are made. So, if you're referring to workflow runs that validate (test) deployments to, let's say, production, those must be triggered when the app is rolled out (unless you have tests that want to validate the state before, during, and after a rollout of a new release).

[@zenmaster24]

@@DevOpsToolkit it doesnt have to produce the same results - presumably you have branches to test code changes. targeting those files based on branch and thus code is the point. the different environment specific manifests can be used for more than one branch - eg i have used the dev manifest for all non named environments - that way devs can test their changes in a known environment but dont have to merge to the dev branch. the dev branch is just the first step in code promotion - eg dev > test > staging > uat > prod might be the code promotion path. this would inbclude both app and infra code.

[@FernandoAlmeida1973]

Very important topic. And very good video. I have two comments. I think argocd defends spliting the state git repository with all the manifests and the source code it self. They discuss that on their documentation. The other it has to do with oci objects. If we have the config manifests in the source repository, what do you think about pushing them as a package in a pipeline as an oci object and configure flux to use it in the k8s instance git repository?

[@DevOpsToolkit]

I think that using oci images is okay, as long as the manifest that references them is in git. That way i can easily see what is the intention. It's a bit more complicated since going to that manifest to find the image and see what's inside is a bit of a pain it's still good.

[@KrzysztofLuczakPro]

Nice series!

[@wladyx]

My question: how do you deal with devbox (global) paths in the nushell? eval "$(devbox global shellenv)" does not work for nushell. I love your videos, thank you for the time and effort!

[@DevOpsToolkit]

I don't use devbox globally. I have a few tools I need globally (e.g. git) installed with brew and then everything else in repos.

[@DryBones111]

Some remote repositories integrate CODEOWNERS files into their user auth and pull request approvers. These implementations make keeping a seperate repository not worth the effort. As for branches, if Git Flow was a mistake, then why are we even talking about "GitOps Flow". Directories/files/flagged entrypoints are the right answer to this question.

[@Art-kz6zf]

💯 Separate folders for separate environments (/application sets). Dedicated branches per environment makes me wanna off myself.

[@DevOpsToolkit]

Totally agree!

[@Mvvement]

Thank you so much for this! Exactly what I have been trying to figure out. I really want to keep the application manifest configuration in the same repo as my app code. I plan to use KCL to create the manifests and abstract away k8 resources while setting good defaults and common config. One thing I am confused on is the deployment process with the gitops repo for my application. Should I publish my app KCL (kinda like helm chart) as a semantic versioned package then render that package in my gitops repo using environment values stored there as input into KCL options? Basically like creating a chart in apps and the values.yaml is in gitops repo, and you render that chart to a folder in the gitops repo instead of directly deploying helm to argo. I am afraid of using argo apps that point directly at the head of the app repo. If I make changes to the KCL code that is not ready for production, I wouldn't want argo to think that it can deploy those changes to prod.

[@DevOpsToolkit]

I use kcl a lot but only to simplify writing manifests. My workflows are outputting kcl to yaml and storing them in git. An important note, in my case, is that most of my definitions are in clusters (CRDs abd controllers) so what's in repos are mostly simple yaml files (generated through kcl). They rarely exceed 20 lines in total.

[@Mvvement]

@@DevOpsToolkit that's good to know. I want to make a simple interface for developers to create apps without writing yaml since they are used to writing terraform and deploying lambdas. I am trying to create a similar experience for k8s apps and allow them to configure env vars, ports, secrets etc without knowing about services or deployments. There could be several apps communicating with each other all stored in the same repo. I could make this abstraction through kcl or crossplane compositions. Either way, if a dev decided to add a volume mount to an app, how would we ensure that the change only deploys to lower environments and not prod before it should?

[@DevOpsToolkit]

@Mvvement if i understood it correctly, you have some form of abstraction (e.g. CRD) that allows volume as input but that should be allowed only in specific environments. So, thr controller itself does not know when that input is acceptable and when it's not. If that's the case, than the best bet is to have admission controller policies that allow such input in some and fissalow it in other environments. I'll do the best to dive deeper on that subject in one of the upcoming videos like the one you commented on (shorter abd focused on a single question).

[@Mvvement]

@@DevOpsToolkit not exactly. The volume is allowed in all environments, I just want to ensure the change is promoted when expected. Say there is an app that is already deployed to all environments, dev, stg and prd. A developer creates a new feature for the app and adds this volume or any other change to the manifest to support the feature. Let's say this manifest change is only compatible with the new image. Once they commit the changes, how do I ensure the infrastructure in upper environments doesn't sync those changes from the base?

[@DevOpsToolkit]

I'm not sure I understood so the answer but be silly. Correct me if I'm wrong... You'll create s PR of the changes to the code, including the changes to the manifests (if needed). That PR will build and push a new image, deploy it to an ephemeral environments, abd test everything. If the change is valid, if it passed all the tests, it will be merged to the mainline and deployed to production (unless there are some other intermediary permanent environments). If that PR contained changed to the manifests, they will be applied to production as well (but after they were validated in an ephemeral environment). Whether that includes volumes or anything else is not important since you know that it will work since you tested it. I'm sure I misunderstood so please correct me.

[@dmitriimrcat]

Dear Viktor, please show us how to use Kubebuilder, how to create own CRDs for Kubernetes.

[@DevOpsToolkit]

I'm not sure I can show it in a quick video like this one. I'll add it to my TODO list for a long one (those I publish on Mondays).

[@dmitriimrcat]

@@DevOpsToolkit I really appreciate that 🤝

[@DevOpsToolkit]

I can't promise the exact date. I expect it to go live in 4-6 weeks.

[@phillipsma]

How do you feel about Kargo using branch-per-"rendered manifests" as an option? Their intent I think is to keep all environment configuration in mainline but render the final into an env-specific branch for sync only, i.e. it's never merged back to mainline.

[@DevOpsToolkit]

I think thing that's a good idea. Also, I never grew to like Kargo. It does not feel right (and I'm probably wrong on that one).

[@phillipsma]

@@DevOpsToolkit I'm yet to try kargo out properly (if ever) but would be keen for you to elaborate on what doesn't feel right in a vid :-) . Not sure if those feelings relate to promotion but I personally opt for using Azure pipelines to manage CD promotion of CI manifest artifacts using PRs, generating the final manifests for review. Argocd watches the final manifests folder in mainline only, but could just as easily merge to an env-branch folder to keep the mainline clean of final manifests.

[@DevOpsToolkit]

I'll do my best to create a video that goes into it in more detail. Until then, what you mentioned is one of the reasons I never developed "warm" feelings for it. I have to run workflows to build, test, etc. It's almost no effort for me to clone code of a repo, execute `yq` to change a tag, and the changes back to the repo. If I would not be using workflows for other tasks, Kargo would make much more sense but, since I do, one of the main advantages of it (update manifests) disappears. At the same time, I don't have to change the way I organize my manifests to comply to whichever organization Kargo expects me to have.

[@phillipsma]

@@DevOpsToolkit Yeah it's all the other (smoke) workflow steps that often need doing post-deployment that I wasn't sure if kargo could integrate simply. And like you said, if that has to be done elsewhere anyway, then changing tags (or whatever for a promotion) is the least of the problem. Of course, determining exactly when those post-deploy steps can occur is its own problem 🙂 Thanks for the input. Looking forward to your next vid.

[@yol1982]

Can I avoid the massive amount of code duplication i.e directories containing basically the same thing?

[@DevOpsToolkit]

Yes you can. Store manifests in one place and have the argo CD app overwrite values that differ from one env to another. That way you have one copy of all the manifests and have to duplicate only the argo CD manifest with each of those copies mostly unique since the big part of them are those unique values.

[@zoop2174]

I don't know if I trust the devs with app manifests in their own repo....

[@DevOpsToolkit]

That's what testers were saying in the past, yet most of us now have tests in app repos. The same can be said for workflows, build scripts, etc. If you don't trust your fellow colleagues, there might be a problem that goes beyond GitOps or any other tech.

[@gardnerjens]

But here are some issues, if you are using Gitlab. That CI workflow will always trigger so lets say you make a new release artifact push then as a result of the new artifact you update your gitops manifest tag. That means the CI workflow will trigger again, though we dont want that because the source did not change?

[@DevOpsToolkit]

I'm advocating for gitops manifests to be in a separate repo referencing base manifests in app repos. When you change a release tag you're changing it in the gitops repo and that change is likely going to be triggered by a webhook from the app repo.

[@DevOpsToolkit]

As an unrelated question from someone who does not use GitLab, isn't there something equivalent to `[skip ci]` in GitHub?

[@Jarek.]

@@DevOpsToolkitFrom GH docs: Workflows that would otherwise be triggered using on: push or on: pull_request won't be triggered if you add any of the following strings to the commit message in a push, or the HEAD commit of a pull request: [skip ci] [ci skip] [no ci] [skip actions] [actions skip]

[@DevOpsToolkit]

That's what I was referencing. I was curious whether something like that exists in GitLab since @gardnerjens mentioned issues with that workflows would "trigger again".

[@DavidBernard31]

When manifest on the same repo as source, I configure the trigger of the ci workflow to ignore change on some path (manifest, chart's value files, ...). On github it's something like ``` on: push: branches: - "**" paths-ignore: - chart/** ```

[@makinote]

What would be a viable strategy to enforce promotion through environments? I'm tired of seeing the same modification to both, staging and production environment, in the same PR :(

[@DevOpsToolkit]

If staging and production are always the same, I'd keep it all in the same place and point Argo CD (or Flux) in each of those envs. to the same location. I'm guessing that's not the case since there would not be much use to have staging that is the same as production. As for the promotion... Change the tag in the staging manifest, test, change the tag in the prod. environment, test again.

[@amoenus_dev]

A note: best practice (as even written in ISO standard) is to always have clear separation of production from other environments. That said that separation doesn't need to be a separate file, but as Victor mentioned can be achieved by tags, artifact deployments and a variety of other ways

[@MR-ew6uw]

We solve this by using kustomizations with a 'common' base layer and then overlays for staging/production etc. But I've also seen it done in a more basic way with symlinks inside git repos.

[@illiakailli]

Why can't we use different branches for different environments while keeping configurations/manifests for different environments in separate folders? One does not necessarily exclude the other. Each environment will 'know' from which folder it should be provisioned, while CI triggers are configured to watch for changes in specific branches. This will make all branches mergeable with one another and make propagation of changes quite trivial and git-driven. I mean, it doesn't necessarily have to be configured like that, but it's definitely a viable option.

[@DevOpsToolkit]

That is, more or less, the same as if you keep it in different directories in the mainline, so I'm not sure why would there be a branch, unless you're talking about creating a short-lived branch with changes to any of those so that they can be validated on a PR before they're merged to the mainline.

[@illiakailli]

​@@DevOpsToolkit I'm talking about CI/CD that is configured to deploy changes automatically to the corresponding environment as soon as they are pushed to one of dev/qa/stg branches. This assumes a CI/CD setup in which you don't need extra tools to propagate code changes to different environments: git merge and push are enough.

[@koffiezet]

Hard disagree on the directories per environment or separate repository. Git is smarter than that, you can perfectly make changes to an environment-specific branch and still merge in new changes from lower environments without them affecting or overwriting the other environment. Directories per environment always ends the same, it's the approach I've seen the most at clients, and it always ends up in a horrible configuration drift between environments, which then it becomes the platform team's problem for some reason. Using git branches has many advantages, and there is not a single reason why you "need" to merge to a mainline branch in Git. It for example allows you to implement specific branch permissions, which for some environments can be a requirement, or you could only allow a bot and a platform/operations team to update versions from a lower env to specific branches in acceptation or production environments, for example a bot that checks if an integration test suite was successfully run. Say you have a development, test and production environment, you start off in your dev branch, with your base application manifests and specific configuration, from there you create a tst branch, where you make the necessary changes to the configuration, from there you branch off to prd to deploy to production and do the same. Any time you then merge changes from dev to tst or from tst to production, will preserve the branch-local changes. If there would be configuration conflicts or env-specific config changes when doing that from dev to tst for example, you branch from your tst environment branch, merge in the lower environment branch, make your changes, and merge that new temporary branch into your tst branch. This ensure your configuration is as close as possible between environments and you don't end up with "but it worked in the dev environment" - which has become the new "it works on my machine". Is it perfect? No, but when having to deal with dozens of development teams deploying hundreds of services, being able to enforce a way of working prevents it from becoming a massive burden on the ops/platform team because dev teams screwed up.

[@DevOpsToolkit]

Assuming that you specify in Argo CD Application manifest environment-specific variables (e.g., host), directories will not be the same. Based on your comment, I'm guessing that's not the case so I'm curious where you set environment-specific variables. Also, based on your comment, I'm not sure I understood where you keep the base manifests. Are they in the app repos or in a "GitOps" repo? If it's the latter, that is the fundamental difference between what you're doing and me.

[@koffiezet]

@@DevOpsToolkit I prefer the approach that for every single application/service having there are 2 repositories, one source code repository, and one "gitops" repository with branches per environments. In small environments you could probably have a single git repo containing all service configuration, but that doesn't really scale well, and blocks you from doing fancy stuff like auto creating Argo Applications in a certain env when a branch with a certain name is created, and routing traffic there using a service mesh for client requests with a certain header set. Not sure what you mean with "base manifests", but most of the time, my approach is that teams use a base helm chart they include as a dependency in their gitops repo, and configure that using the values file and optionally some additional templates and files for specific customization. These base helm charts are versioned and pushed to a helm repository. And naturally there are differences between the branches for each environment, but once changed for a specific environment/branch, it will not be overwritten when merging in new changes from a lower environment to that branch that don't touch those specific lines of configuration, and with the proper branch protections and pipelines, you can't have the "oops we forgot to add this new config part to production", or not updating this base helm chart version with changes they rely on. And on top of that, so many things here can be automated. One of my big gripes with the whole gitops world is that it's called GITops, not copy-paste-ops. By using Git, you get a wealth of tools that I feel most of the gitops-y setups are completely ignoring because git is often not well understood.

[@DevOpsToolkit]

@koffiezet By base manifests I mean manifests for an application (yaml or helm or customize or any other format with all the resources like a deployment, service, ingress, etc.). The only thing missing there are variables or overwrites with "stuff" specific to an environment (e.g. host, tag, etc.). I keep those in a separate environment specific repo and those are Argo CD Application manifests that not only overwrite but also point to wherever the "real" or base manifests are.