Branching Strategies Explained

[@ultiumlabs4899]

finally, someone summarizes these different branching strategies in an easy way to digest. thanks a lot.

[@homegrownluv8106]

My dumb ass stumbled across the wrong video. I literally have no idea what your talking about and found no clue in the comments. I was just trying to figure out how to create more branches on my cannabis plants. Hats off to u geniuses.

[@DevOpsToolkit]

I would gladly change my branches for yours :)

[@PrinceRambade_Official]

It's a great video, very crisp and concise !! However, in a corporate development environment there can't be only 1 approach. Probably, it makes sense to use combination of Trunk + Fork + Feature Branch. Here is the reason : Reason : 1) Trunk based - For frequent small feature changes to main branch. 2) Fork Strategy - To limit the developers access to main repository and allow merge into main repo only via PRs. 3) Feature Branch - To have development for major features which tends to break almost everything until they are entirely completed. e.g. TECH stack upgrade etc I know there can be different approaches with Trunk based development, to disable/hide incomplete features (e.g. Dark launch, Branch Abstraction, feature toggles etc). but that requires very high level of maturity of team (which is not always possible with new developers joining once-twice year).

[@DevOpsToolkit]

I don't think that there is one-fits-all solution to anything, including branching strategies. While some work better than others, it ultimately depends on what one is doing and how mature the teams are. What really matters is not to pick one, but to be able to make a decision what to do and how to do something based on circumstances. There is nothing wrong to use different tools and strategies depending on specific situations, hence, it's quite OK to choose the right strategy on per-case basis.

[@Zeelo123]

Exceptional! you are a legend, and the way how you explained these strategies, I could not control but ROFL. We started off with GitFlow, and then adapted Release branching and now taking the path of Trunk based!

[@bobbyray8457]

This is a great meta topic in our industry, thank you so much for taking the time to dd on this.

[@DevOpsToolkit]

Glad it was helpful!

[@n4870s]

@@DevOpsToolkit great video. Also you forget that in case of microservices it becomes insanity level 999999.

[@_indrid_cold_]

This was outstanding, I loved every word of it. I’ve worked on most of these methods and cannot agree more with your evaluation.

[@codesrinivas2816]

you solved my confusion by summarizing the branching strategies once and for all tysm

[@SaeThunder]

A git branching strategies video that talks about git branching strategies. And on top of that it explains sums them very well. This should be higher on youtube's list.

[@DF-ss5ep]

Personal opinion: The testing needed for trunk based development isn't a lot of unit tests. What's needed, in my opinion, is to have a robot equivalent of a QA tester, meaning that the thing will be deployed in an environment just like production, and some component will drive the program like a human would. Teams I worked in didn't like this or even see it as a possibility because it sometimes requires thinking outside the box and being multi-disciplinary in order to implement.

[@henryboy19]

Something like cypress? This seems to be the robot equivalent

[@privatename3621]

Correct. Unless you are dealing only with a rigid API (and no UI involved whatsoever) where a very mature set of deep unit tests (whitebox/blackbox) and be unleashed that will verify all possible inputs/outputs to confirm behavior is not broken for each of the API interface elements, then sure. Unleash the robots. But for _anything_ else, if you are not putting a set of human eyeballs on it and actually manually stepping through the application, then you are 100% going to shoot yourself in the foot. That tiny little CSS change you made which does the right thing on one page, but is shared around 50 other pages, has now right-shifted your logos to the wrong location. That input drop list menu that worked just yesterday is still there and the first item that the automation test relies on is still there, but that automated test robot will not tell you that the same menu item is now repeated 20 times. That automation robot will also not tell you that the beautiful white color you chose for that css element not just made half that text on another page completely invisible on a white background. The automated tests still see that text as being there, so what's the problem? Automation is and always will be just a backup sanity check. Real manual human testing is ALWAYS required before releasing publicly available software to the world, else you are going to have some very pissed off customers.

[@DTQC]

@@privatename3621 In your scenario it seems that the fear of walking on a mine is the main problem, not the failings of testing automation. Feature toggle is what allows you to check for a mine before stepping on it.

[@Gaming214-y3g]

What you're saying is end to end (e2e) test, which is an integration test. Look at the testig pyramid, you'll need different test to cover. E2E test are slow and it's white box test. Unit Test are super fast, gives your immediate feedback and helps you identify problems on the unit level, whereas if an e2e test fails, if doesn't tell you what's wrong, you only know the feature didn't work as expected. And if you don't cover using unit test and instead write them in e2e test, you'll need to covert a lot of scenario which could is not ideal and not efficient.

[@bigfishoutofwater3135]

Often the production-like environment ends up not being very production-like. It can also be very expensive. Production is always the most production-like environment. Feature toggles were mentioned and can enable dark launches, staged rollout, A-B testing, etc. Ongoing observability that includes health checks combined with a fast hotfix capability is also critical in my opinion.

[@DaveBizz]

Very well illustrated. I don't think you should waste your time watching any other videos about branching strategies

[@mvargasmoran]

I'm on the Environment Crap. and have deployment team... when I mentioned "branchless" they told me that doesn't scale or other idiocy.

[@aldyj4733]

"...because it will feel like I'm on vacation" I ROTFL there! Great explanation btw, thanks

[@oliverbruckerskau7047]

You just saved me hours preparring my last notes for my exam.. thank you

[@deap5193]

Well done! You should do a talk with the guy from the channel "Continous Delivery", I just saw a video in which he is more for trunk-based development rather than branching. Would love to understand better in an in-depth talk between you two! Btw, the git flow part was really fun!

[@DevOpsToolkit]

That's a good idea. I haven't seen him since one of the pre-COVID conferences. Let me check whether he has time for it.

[@Checkitwithamagnet]

Excellent video. Thorough and clear enough for someone who has almost no experience with this!

[@ndev92]

Wow I spent ages reading on articles and other videos and I could never fully understand branching and versioning until I watched your videos. Thank you!

[@codinghusky5196]

It's a great video. But, for anyone reading this... the video is an absolute gold to help you synthesise what you know or understand what you're getting into; but the only way to truly grok versioning and branching is to simply - use it. The first time you delete some code you didn't save into some backup file or OneNote or something, or you did save it but spent an hour looking for it among all the gddamn backups you have in various notebooks or text files, you will understand the need and the practicality of a versioning system such as GIT. A Git GUI tool, like GitKraken, can help you just browse your commit history and look through things in your code (or other people's code) without even reverting back or changing the files. It's invaluable. The first time someone interrupts you while you're doing something on a project and says "I want you to stop that and do something else, because this new thing is now a priority", you will understand what feature branching is for and how awesome it would've been if you didn't just do 6 buggy commits to your local trunk repo but had made a feature branch instead. Of course you know you can just go back to before your 6 commits, create a new branch, and work from there. Etc. My point is, absolutely use this video to synthesise any systematise your thoughts on the issue. But don't let yourself get stuck in Paralysis by Analysis. git init & go!

[@X-llllllllll-X]

Stayed for 14:45 my favorite branching workflow 👌😉

[@InspireAZ]

the best video ever i watched about branching strategies. Thanks a lot

[@bendunaway8296]

Great rundown. I LOL'd when you mentioned people using complex strategies for job security. I have people like that at my company.

[@miletacekovic]

Just to be fair to the Release Branches Strategy: Everything Victor said about it is true, but when it is applied in the Project or Product as a Service business models, when there is only one production environment. However, when business model is Product based, where product is shipped to many customers and there are many production environments, then Release Branches Strategy is viable complementary branching strategy to cover customers that are using different versions of our product (for non-technical legal, regulatory, business... reasons). Actually it is must-have additional branching strategy, in addition to say trunk based development or feature branches that are used for the main stream of development on the current version of the product.

[@DevOpsToolkit]

Agree 100%. When you do need to maintain multiple releases of a product, you do need each of those releases accessible separately, so you do need a branch for each active release, similar to what Kubernetes itself is doing.

[@SynthwaveMindset]

There is a big chunk of strategy that was missed here in relation to trunk based development combined with release branches and environments. My preferred workflow uses trunk based where main/master deploys to a staging environment (or UAT). Then once acceptance testing has completed, a release branch is created to trigger a deployment to production. Hotfixes then can be cherry picked from main into the release branch, as needed, avoiding the branching mess described in this video. This is similar to Gitlab AutoDevOps, but inverted from their default (main/master targets k8s production and other branches target staging environments). This also works well outside of k8s deployment targets.

[@DevOpsToolkit]

You're right. There is close to infinite number of permutations one can do with branching strategies.

[@Ginjitzu]

My team _started_ using git flow shortly after I joined the teama few months ago. I was brand new to the team, and don't have a tonne of experience with branching strategies and yet even without knowing exactly why, I just had an almost instinctive revulsion to the new strategy, but as the team newbie, I kept my concerns to myself. Tomorrow though, I think I'm going to share this video with my managers and just let them watch it and take from it what they will.

[@DevOpsToolkit]

I'm curious to hear how it went.

[@Ginjitzu]

@@DevOpsToolkit Well I shared the video with my team. I think one person looked at it and it was basically forgotten about. Don't worry though. I still haven't given up this particular fight. I just need to pick my time.

[@ChrisGWarp]

Interesting video; I liked it. With my 40 years of experience, I agree with most of it. There is an unstated assumption in what is presented here, especially with the trunk based dev. The assumption here is that whatever is in trunk/master/mainline = production. That is not always the case, and it is where I diverge from agreeing. This is due to Continuous Deployment, which I personally, am not a fan of. I prefer a human in the loop. I follow the Apache Maven approach. Yes, Forking and PR's are used [to good effect], but commits to master can (and sometimes do) create a dirty/broken master. But this is not a problem, as we choose when to perform a Release - it is not automatic - why I don't like Continuous Deployment. Once is a Release is performed, it's out in the wild, and we prepare for the next release. Always moving forward. Cut a Release, moving it through environments until it fails testing or hits Prod. You make this point well, where you point out that we're not deploying source into production (unless you're using Ansible!), we make packages and deploy them as need be. Why don't I care that trunk does not equal production? What if there is more than one? OS Libraries and some of my work places have more than one production, so having trunk represent 'production' what ever that is, is meaningless. One thing that I'd add/challenge: I understand why you say that you need a disciplined team for trunk, but my experience shows that feature branching requires a higher level of maturity and discipline so that you DO limit branch lifetimes. Otherwise you end up in merge hell and everything comes to a sudden halt due to the inability to deliver any change. So, I take the KISS approach for everything that I've ever done, trunk based dev has worked for me, and what I recommend. Only go more complex if you absolutely need too. TBH, there was ONE exception, where there were monthly release branches with overlapping dev (due to testing resources), but we simplified things by never having to merge back to master (no point). R2210->R2211->R2212->R2301 etc.

[@DevOpsToolkit]

You're right about "multiple productions" like, for example, when maintaining multiple releases of something (e.g., Kubernetes). In those cases, there are multiple branches that represent "production".

[@fahadahmed5620]

agreed, feature branching goes insane as well, depending on the number of features being worked in parallel. Toggle flags is a life saver. Feeling sad for teams working on environment strategy. That's a nightmare.

[@tabliqatchi6696]

Baby dev here: I learned git flow just yesterday and I was so proud of it for a complete day until this video broke my pride bubble :D But seriously, git flow makes a lot of sense, it's not this difficult. Also there are tools that can automate its flow. But I'll trust this video b/c experience talks first.

[@Kabodanki]

Make sense…. For your context. Every solution works better in specific context, good engineer chose the right solution, not the most shinny tool

[@uneq9589]

This explains all the strategies very well. But I think it should also include rollback strategies and which branching strategy makes it easier.

[@DevOpsToolkit]

I believe that, in most cases, we should roll forward. If you do need to roll back, tools like Argo Rollouts or Flagger are great choices and do not depend on git branching strategies.

[@uneq9589]

@@DevOpsToolkit I use GitHub Flow in all my services. We have automated tests, but we still need to rely on manual QA. We try to make features backwards compatible, but that's not possible everytime. I am thinking to start tagging the main branch with a release tag upon a successful release and in case a rollback is required, just create a build with the older release tag and get it deployed. Looks like it would work in theory. I am still figuring out the edge cases that would increase madness (in your language ;) ).

[@vedadburgic7530]

Well done for summarizing and comparing all of them 👏👏👏

[@nathanhawk1580]

Excellent video! I think you left out the biggest madness of all -> Not having a documented strategy, ending up with some crazy mix of all of the above and hundreds of branches -> Madness Level = Suicidal... Might be worse than using Outlook for source control.

[@DevOpsToolkit]

It's planning and experience. The more you have of one of those, the less you need of the other.

[@Blkhole02]

I giggled a bit at the environment branching part, since the idea was pretty intensely circulated in our team meetings a few months ago, as a possible strategy for our IaaC repositories. In the end we went for something most closely resembling feature branching - looks like we dodged a git bullet there :)

[@cyclogenisis]

you may have gone to a mental institute

[@mohdsaif1798]

@@cyclogenisis after working for few months you can call that a vacation lol.

[@AncientMarinerNY]

I've worked in many teams that use short lived feature branches (that exist for a day or two) and they call it trunk based development. And I agree with them. What you call feature branching is essentially the same as trunk based development since you need feature switches because you are essentially pushing to the trunk regularly.

[@DevOpsToolkit]

Technically, trunk based development means that you push directly to the mainline without branches and PR. In practice, short-lived branches are very similar in terms of the objectives (get to mainline and production fast). Feature switches are not a requirement of trunk-based development. They help a lot, but are not necessary.

[@AncientMarinerNY]

@@DevOpsToolkit Yeah, so making them different branching strategies is just academic at best. I still need to hear a good case for why using short lived feature branches with feature switches is not considered trunk-based. All I've heard so far is just splitting hairs. I think I am in good company given the previous teams that I've worked with and given some "authorities" like Martin Fowler who considers using short lived branches as trunk-based development too. Of course there is no official source of truth. In the end, whatever is adopted by most people will win out. PD: Just in case, when someone says "trunk-based" it is not implied that you are pushing to the trunk. That is your personal definition. Another valid interpretation is to base your development on pushing to trunk as quickly as possible. So technically, it does not mean pushing to the mainline without PRs.

[@DevOpsToolkit]

At the end of the day, what does matter is how frequently we push the code to the mainline. Whether that is done directly or through PRs is not that important. So yes, I agree. If you are pushing to the mainline frequently, you are effectively doing trunk based dev. The only thing I would correct is when you say "as quickly as possible". For some, quickly could be once a month.

[@ChandanKumar-ou9fr]

You are awesome. The visual & pictorial representations were awesome. Though I spent more than 2 hours understanding this complete video. But this was so much worthy and concentrated content. Thanks for the video 😃

[@pinkeshpatel1940]

Excellent video, I like the way you delivered truth with a pinch of sarcasm. Kudos!

[@avishekbehera]

Hey Viktor, Trust me this is one of the best video explanations out in internet. Super thanks for making information as well as fun. I totally get your points and the options for different purposes. And the last takeaway was awesome, Haha.

[@SD-ko4tz]

In our BI team we use release beanching because things are interconnected. The thing is… you have to commit often

[@turboguzar]

7th of April, 2005. Release date of Git. However, Victor is using branching strategies for 30+ years. LEGEND, still think you're the best DevOps youtube and can't stop wathing and learning from you! Keep the great job mate!

[@DevOpsToolkit]

You're right. That statement had some artistic freedom in it. I have not been using branches for 30+ years. If my memory serves me, and, this time, I calculated it correctly, I've been using branches for 27 years, so I should have said 25+ years. I think I started working with CVS in 1995. The "pain" caused by working with it still haunts me. Discussions about branches started long before Git was born.

[@turboguzar]

@@DevOpsToolkit WoW, my appolagies, I've haven't heard for CVS before and thought (my bad) that branching was invented with git (feel ashmed). Meaning your statement is more than correct!

[@outifaout]

Not only super instructive but also funny :p Excellent as always, thank you !

[@kebabimpaler]

Great video thanks. Though I will step in to defend git flow a little here since it seems no one else wants to :) ... When I use git flow, the release branches / hotfix branches are like a stepping stone to main, then main (which one would tag / release from anyway) gets merged back to dev instead of the original branch. It makes things a lot simpler vs oft-documented git flow per-branch stuff and cuts out the "more or less sometimes eventually up to date" bit (1 tag = one merge back to dev rather than depending on the merger to know how to / remember to merge individual branches back to dev - which admittedly they wouldn't remember to do otherwise). I think it depends a lot the sensitivity of the software you're writing, the manual testing / team involved, the stability / confidence required etc (after all, unit tests / integration tests get bugs too). I do a lot of specialist development so those factors have been important pretty much anywhere I have went. Effectively it depends on whether or not you need release branches to stage for extensive and/or expensive non-automatable QA while ongoing bleeding development testing can continue in isolation. It also gives you a more formal and well defined version of environmental branching too without doing an environment strategy (automated deployments for dev = integration, release-* = UAT / staging, master = production). It even can give you long release branches should you need them. That's the thing with git flow - it pretty much gives you everything you describe in the list, but gives you a well defined framework in which to do it. As another example if your target is an embedded or specialist hardware system, you can't always automatically test every aspect of it, and testing everything in a test plan is resource intensive where you have a separate QA team testing it. So you don't want to do it too often (and as a developer you don't want to be pressing buttons on a hardware device either). If you're lucky enough to have a simulator for CI, even then it won't necessarily behave in the same way as the real thing so you ultimately still need manual testing anyway. I think git flow gives you everything most mature projects need and it's ubiquitous enough that must developers don't need it explained to them. In saying that, I usually would start trunk-based for new projects, then move to git flow once the project starts getting deployed because it *does* bring with it more overhead, but I've used it in several companies, none of which needed a release manager and none of which caught fire after I moved on.

[@DevOpsToolkit]

That's a great write-up. Ultimately, it all depends on what you want to do and what your requirements are. I tried to provide general rules and ideas which certainly do not work for everyone. The important thing is to understand and have practice with different tools and processes. After that, one can make a decision what to do and what to use based on first-hand experience. It does not matter if that ends up being X instead of Y. What does matter is that decisions are not made blindly or "because that's how we were doing it for years". Beyond that, decisions differ from one organization to another.

[@cathalsurfs]

Excellent presentation. Immediately subbed. Thank you from Ireland.

[@r3jk8]

Great job on this video! It was spot on and also had some really funny moments. “Cake made of mud..” 😂

[@DevOpsToolkit]

Thanks! 😃

[@reefhound9902]

More like a cake made of shit.

[@jaganpandurangan8121]

What no one realizes is even in feature branching or trunk based development or release branching we all encounter merge conflicts etc., because the business team wants 3 big features in the same javascript or java or typescript file at the same time which will be taken in same sprint and the leads and devs are supposed to deliver that.

[@francistembo650]

I laughed at Git flow segment like I was watching a standup show haha.

[@daveslee7726]

Luv this! I appreciate your humor ... very helpful.

[@def1nt]

This was a good explanation and illustrated even. Thank you for helping.

[@KnThSelf2ThSelfBTrue]

I think release branches can make sense for mobile applications or IoT software where you simply can't deploy updates on-demand, or force users to update remotely.

[@DevOpsToolkit]

You're right on that one. I forgot to mention that my points do not apply to those orgs that distribute software in a way that users download/install some binaries. In those cases, multiple releases might need to be maintained and that requires release branches. A good example is Kubernetes which promises to maintain the last 4 releases and that means patches to all those. So, it's not that much whether you or others are updating but, rather, how many major/minor releases you promised to keep up to date with bug and security fixes. If there is no such promise, users can always install the latest.

[@milonga332]

Wonderful explanation! You have saved my day.

[@DevOpsToolkit]

Thanks a ton, @milonga332.

[@thosethere]

you make the topic interesting, thank you.

[@CarolinaSilva-h1r]

You deserve a million thumbs up!

[@Peter1215]

Ah, good old branch per environment merging. I still have nightmares! Happy feature branch user now :). Great video as always.

[@DevOpsToolkit]

I still remember spending weeks in creating diagrams, meeting with people, and doing whatever else should be done to convince decision makers to simplify the insanity we had with branches. Every once in a while, I remember those days and then I spend yet another sleepless night.

[@chrisjaimon9137]

@@DevOpsToolkit relatable, going through the same.

[@barefeg]

So you don’t have staging environment? Or do you release manually?

[@DevOpsToolkit]

@@barefeg That depends on the strategy. When doing feature branches, there is no staging, at least not in the traditional sense. Each PR from a feature branch can get an environment that is removed when the PR is closed. In a way, there are as many staging envs. as there are open PRs.

[@ha5h]

Loved the video, subscribing your channel :) , especially the extreme hatred for Git flow & Env. branches :)

[@GM-qv1ql]

Nice video, Viktor. Liked the simplified articulation. Great job!

[@CC-ij3ky]

That madness caused by git-flow, I am experiencing it in my veins these days as I switched to a team that uses git-flow. When I share my opinions with the more experienced engineers, they keep producing arguments against me which freaks me out.

[@kevon217]

Appreciate this breakdown, thanks!

[@vinaysingh5469]

What an easy and interesting explanation!👍

[@Jubilee1989]

Great video, very helpful. Thank you.

[@davidg.3226]

I disagree with the "you don't care what goes to production" in trunk-based development, because the feature toggles are exactly to avoid that. Other than that, great video!

[@DevOpsToolkit]

You are right as long, as you said, there are feature toggles. The only thing I would add is reliable and automated testing.

[@thigmotrope]

@@DevOpsToolkit also you don't have to release everything that you commit to main immediately. You can still test it as a release candidate. It's just that the review process is different, and that's why Dave Farley suggests pairing instead of pull requests for trunk based dev. I think this is a bit unrealistic for most teams

[@agilethoughts6619]

Nice presentation! Thank you!

[@yekohein1528]

Thank you so much for explain.

[@sergeykorobkov9080]

excellent explanation! Thanks Victor for sharing!

[@Pjblabla2]

Great summary and actionable insights

[@akimbbo_upnext]

Really nice overview. Thank you:)

[@bartekr5372]

So eature branches with feature flags. For me it has everything as soon as feature flags are being used. As soon as project is non-production you can give right to merge pull requests without a review. It's quite close to trunk based development - gives you more speed via simplicity while avoiding any dedicated pipeline logic.

[@DevOpsToolkit]

You're absolutely right. Feature Flags (Feature Toggles) are the perfect companion. I skipped mentioning them in this video mostly to avoid extending the subject too much. I am planning to have a video (or a couple of videos) dedicated to Feature Flags.

[@ogyct]

I strongly disagree with opinion on Git Flow. There is a good reason, why this strategy is being used. First of all, it is very well documented on atlassian web page. Despite the fact, they themselves tagged this strategy as deprecated, I didn't find anything, that is documented better. Git flow also works well on enterprise projects, where release cycle is slow, and there is a need for a stable main branch with occasional hot fixes, while active development happens in develop branch. The release branch is arguably not a good idea, imo it's not necessary, but other than that, Git Flow is a very viable solution. You have to understand, that projects are different, and there is no solution that fits them all.

[@ruess]

Best video on guy I’ve ever watched

[@olenamaksymiv7754]

Thank you for the video!

[@GreyDeathVaccine]

The threat of migrating my code to subversion worked ;-)

[@DevOpsToolkit]

A while ago, I used a similar strategy, except that my threat was migration from SVN to CVS.

[@xavierdubruille]

Loved it ❤ thanks! But you're not completely fair on gitflow: it's not that complicated : it's like the feature flow but with extra steps (and, from what i've seen, some steps, like the release branch are, often, not even used !)... ... And, often (e.g. if you don't have strong tests, ci/cd ) you don't realy have the choice :(

[@radekcrlik5060]

Great explanation even today :) I've been feeling lately that every other company developed their own branching strategy with its own name :D Even in our team, I have no clue where to categorize our strategy. It is a mix of feature, release, and even maybe an environment strategy. I guess I will never know if there is a team that uses the similar technique :D Cheers!

[@brianmills5289]

What do you recommend when using GitOps style IaC? It seems to be often recommended as a branch per environment, or sometimes using a tag per environment. Love to hear your thoughts.

[@DevOpsToolkit]

I do not like the idea of using a branch per environment. Branches are supposed to be mergeable to the mainline and not used as a permanent placeholder disconnected from the trunk. I tend to use either a directory for each env. in the mainline or a separate repo for each env. In either case, it's the mainline that matters, and everything else is temporary.

[@Blkhole02]

Not sure how applicable this approach is to your use case, but if you're using something like Terraform Cloud or Enterprise, you can separately version your modules using their registry. Then you can reference those (sub)modules by version in a main IaaC repo, to which you can apply something like feature branching, since making a "code change" would simply imply a version bump to said modules.

[@DevOpsToolkit]

@@Blkhole02 Exactly. That is a general approach to releasing something. Every release is versioned and stored somewhere. Using a release of something should have no relation to branches. That is the part that people mix. There is no need to have, for example, a staging branch with the source code of an app if you already built that app and created a release.

[@wodan742]

@@DevOpsToolkit Nice video as always! We are using the following strategy for application deployment (application code itself is in a separate repository, can therefore be ignored here): Feature branch for any change no matter which environment -> dev-branch for deploying to dev env -> main-branch for deploying to test env -> tag on main-branch for deploying to prod env (Each branch has directory structure containing configuration of all environments) 80% of the time we are quite happy with this because of the following reasons: - We can prepare changes for all environments simultaneously, but roll them out independently. - The standard way of merging is predefined (dev->test->prod) so there are almost no merge conflicts - We have multiple pull requests which get reviewed before deploying to prod effectively Things which are problematic: - Some applications have more environments than these 3, meaning more branches. This is where things get laborious. - The code is effectively stored as many times as there are environments (-1 for prod, because its tagged) which could lead to confusion what is applied effectively. What would you suggest for in our case? Especially the first "advantage" described above is huge for us.

[@DevOpsToolkit]

@@wodan742 I think that branches should be temporary and should be mergeable to the mainline. From that perspective, I don't like the idea of having a dev branch for dev env. Instead, I prefer having a dev directory in the mainline, or a dev repository. That way, mainline is always the source of truth and all PRs are made towards mainline. If sharing is important, different dirs in the mainline simplify things a lot. The alternative is to use some kind of libraries/modules/plugins that contain the shared code. Your painpoints are one of the reasons why I love tools like Argo CD and Flux. Manifests can (and should) be in the repos of the apps while the references to those apps and env-specific values are in the repo of that environment.

[@НиколайБеляшов-в6к]

Thanx a lot for your work!

[@myronww]

So your madness levels only take into account the developer perspective. Typically the madness level will be inversely proportional for the Quality organization for these branching strategy. For example, you rated trunk based development with a madness level of low. For QA, the madness level would be high because the difficulty of ensuring that a bad change does not go out the door is much more difficult.

[@DevOpsToolkit]

That depends on testing. The more is done manually, the more likely it is for trunk-based development to be a disaster. Even if more (if not all) of testing is automated, it still depends on the level of trust we have in that automation. Trunk-based development is a good option only for those who have very high level of trust in automation and, just as important, those who do some form of test-driven development. It does not matter much whether we trust tests if they are written after we push something to the mainline.

[@myronww]

@@DevOpsToolkit Its the opposite because when you test, you have to make sure features are turned on and configured correctly to excercise the code.

[@DevOpsToolkit]

I'm not sure I understand what you mean. By features being turned on you are referring to feature flags. Right?

[@chrisjaimon9137]

haha, thanks a lot for this video. Awesome take on this subject. I'ma send this video to all the old heads i know, who still believe Git Flow/Environment branching is the right way to go. Tired of convincing :P

[@rupeshpatil6957]

Great Video. Thanks Victor

[@dinobulja]

Do you have videos explaining the workflows of feature-branches and release-branches little bit more in depth?

[@DevOpsToolkit]

Unfortunately, no 😔

[@dinobulja]

@@DevOpsToolkit perhaps a good reference to look at? Much appreciated again

[@DevOpsToolkit]

It's been many years since the last time I read a book about version control so i not even remember what it is. Since then, it's mostly personal experience. I other words, i don't think I have a useful reference 😔

[@LokeshSharma-zz6pm]

Outstanding !!

[@SaudBako]

There are millions of ways to branch. All the strategies you mentioned fall into a single family. Branching with git opens up so much potential. Yet, the industry is stuck on one flavor, as always.

[@prasad_yt]

Very informative ! Thank you!

[@barefeg]

I use a single mainline where everything is merged. Any commit to it is deployed to staging environment, and any tag is deployed to production. Insanity level seems low even with this git flow. The only problem is when devs merge to the mainline but that change is not ready to be tagged at any moment. It doesn’t happen that often but then it’s better to keep the feature branch a bit longer until we know it could be deployed to prod right away or wait. I have to mention we do automatic deployment to staging when the PR is open too. This doesn’t use feature branches but instead it’s a new pod with slightly different service name. The infrastructure can route to that version at runtime by passing flags. It’s kind of a feature flag but for the microservice routing infrastructure rather than the code. This makes the deployment a bit more complex but it’s fully automated now. I would still like to try code feature flags to remove the issue I mentioned of the mainline not being ready for deployment as well as not having to deploy multiple pods for each PR. Then we would release to both staging and production on commits to the main trunk. But the default state of the feature flags would be on in staging and off in production. A tag In the main trunk would then happen when a feature flag is flipped to on in production. I haven’t really thought how that looks precisely but it sounds interesting

[@DevOpsToolkit]

If a change is not ready (to be tagged or anything else), it's probably not a good idea to be merged to the mainline anyways.

[@barefeg]

Oops I submitted before being ready. I edited the main comment. But I agree it needs to wait if it’s not ready, even if we try to be as agile as possible

[@cabanford]

Really nice overview. Thanks

[@cabanford]

I've been getting back into development after about 5 years off and wanting to bring my basic skills up to a usable level. Git specifically and branching (CI, CD, testing) strategies are turning out to be the biggest area of learning and can be rather overwhelming at times. You have a direct, easy and fun to follow manner. The information comes across incredibly well. Kudos

[@codinghusky5196]

Hvala Vix.

[@mohamedaminchairi]

hahaha loved the conclusion ! RUN !

[@GernotPokorny]

You forget the backup aspect of the central repo. When you only push to the main line and do not create branches, then you do not have that backup component and at some point some work will get lost if you do not push your work to the central server. Often people want to push to a server before they leave the office or it is an requirement of the company.

[@DevOpsToolkit]

That's true only if you work on something for a long time without pushing to the mainline.

[@GernotPokorny]

@@DevOpsToolkit Not every team member will push every day day to the main line, not even every two days. Some features need a few hours dev time, but some also need a few days and you should not push half done work to the main line and you still have push requirements at many companies. Please don't say now that you should split up everything into a few hours tasks, because that is impossible to do and also no good idea to do that ^^ It would be a project managers wet dream to see that every feature and bugfix is done in 4 hours and fully tested and operational. Devs always completely understate their development time. I do not even want to mention what figures I've heard in regards to development time at companies and what the reality was and this isn't due to inability of devs, but just the reality of hour industry that execution time is often only a small fraction of dev time and often devs are challenged with new things and problems arise. This scenario where we all only have execution time does not exist in reality, because then we'd all build an app in a matter of 2-4 weeks. Also if somebody has to pack his things right now after 1 hour of work what then? It's just a characteristic of this one branch model that it will not be easily possible to fulfill this requirement.

[@DevOpsToolkit]

@user-du6hs8fe8x you're absolutely right. That's why we use feature branches.

[@GernotPokorny]

@@DevOpsToolkit yes

[@WallyBricks]

great explanation!

[@henryuta]

Awesome video, nice and easy explanations

[@Vinod_Kumar827]

thank you, this was great

[@devnetindonesia9716]

OMG, thx a lot Papa Varcic

[@jacekbiaecki8076]

I liked this video very much. Very informative

[@fenarRH]

Rather than no one left behind (Trunk Madness), I prefer you will never walk alone (Release Branching).

[@angelgutierrez240]

What about the continuos delivery using feature branching or trunk based to dev, staging and production environments in an automated fashion. Think of GitOps and GitHub actions to k8s cluster. How would you do it? Git tags?

[@DevOpsToolkit]

That's a great question. Would you mind repeating it in today's AMA session on the channel? I think it would be of interest for a wider audience.

[@angelgutierrez240]

@@DevOpsToolkit I did but my question got skipped several times. I even sent a super chat 😞

[@DevOpsToolkit]

@@angelgutierrez240 Sorry for that. I can assure you that it wasn't intentional. Darin (he's putting messages on screen) probably missed it. I'm in a meeting now... I'll answer later tonight in here or, if you prefer, pick any time available in calendly.com/vfarcic/meet and we can have 1-on-1. Terribly sorry for missing your questions. I feel really bad about that. I hope there's no hard feelings.

[@angelgutierrez240]

@@DevOpsToolkit would be great if you could reply in here so that everyone with the same doubt be able to see it.

[@DevOpsToolkit]

@@angelgutierrez240 I tend to keep application manifests in the repository of the application, and Argo CD or Flux manifests in env-specific repos. Those are referencing application manifests and overwriting variables with env-specific values. Inside the pipelines (e.g., GitHub Actions), I checkout env-specific repos, modify Argo CD or Flux manifests, and push them back. The logic is, more or less, the same no matter which branching strategy is used. If, for example, we have feature branches, I'd create new Argo CD or Flux manifests and push them to the previews repo whenever a PR is created. When a PR is closed, I'd remove those manifests so that the preview environment associated with that PR is removed. The builds triggered by commits to the mainline (no matter whether those are made directly or by merging PRs) would perform some actions (e.g., building, testing, etc.) and, each time I'd need something deployed somewhere, push changes to the env-specific repos. At the end of the day, it's all about modifying env-specific repositories that reference the repo with the manifests and overwrite stuff with env-specific values. It's a set of simple `git clone`, `sed`, and `git add/commit/push` commands. I do not have a strong need for Git tags since the history of env-specific repos is giving me insights what was and what is running in my clusters. That being said, I do still use tags, but that's mostly for the purpose of distributing releases to third-parties.

[@lebronkiddz]

I completely agree with the Test Coverage of >95% should graduate their Strategy to Trunk Based…..otherwise, I can say lots of Engineering Manager or LEAD, pretty much hiding behind Waterfall with few merging every other day….I’ve seen lots of self-proclaimed DevOps engineers for big enterprise proudly thinking they are a DevOps engineer….true and true…. 😂

[@esbrasill]

hmmm. Were i work they want to do environment branching, but keep some branches in different repositories

[@proprajyot]

This is a great video! We used to follow gitflow, but now swited to release based braching. But the issue we only have 2 environment. As per gitflow, develop and prod. How can manage these in release based branching.

[@DevOpsToolkit]

I don't think that there should be release branches. Tags are a better option since there should be only one release (one binary or one container image) that moves through environments.

[@simonrussell9238]

Great video, thanks really found it to be informative.

[@xnullz]

The last 2 blank minutes are for meditation over the code-to-subversion? :D Cool video as always

[@pointerish]

Introspection, I'd say. To ask yourself if the whole software development shebang is worth it.

[@DevOpsToolkit]

Good catch. I must have messed up the editing part. It should be fixed in a couple of minutes.

[@cihanyilmaz4474]

I was gonna say best video but after reading some comments of Viktor, I am gonna say "good video" :)

[@DevOpsToolkit]

I guess "good" is still good :) What was the comment that lowered the score?

[@cihanyilmaz4474]

@@DevOpsToolkit I do not want to be misunderstood. I was just joking, it is a great video. (I was referring your comment about the term "best practice" & "good practice")

[@DevOpsToolkit]

I missed my own reference :)

[@joehirn2022]

The most difficult thing is keeping things simple.

[@CloudScience-z4l]

thank you for this

[@miletacekovic]

One question: While hopefully everybody agree that Environment branching is a horror for application development and application code repositories, what do you think about using it for DevOps repos, like infrastructure-as-code repos, environment configuration repos etc... ?

[@DevOpsToolkit]

I believe that a branch is something temporary that should be eventually merged to the mainline. If you define environments as separate branches, they will be independent forever (not mergeable to the mainline). In those cases, when you do not merge, I think it makes more sense to have a repo for each environment or, if one repo per env is too much, different directories in a single repo.

[@miletacekovic]

@@DevOpsToolkit In this case there is no mainline (or you can consider dev branch as mainline), but there could be some value in having a relatively easy way to check what config changes are done in dev/QA/UAT... environment since the last release, in order to know what changes need to be applied to the next environment in the release lifecycle, including finally production, and to cherry pick these changes to the next env. But as you said, its more natural to work with multiple repos or folders in the same repo. Though Git worktrees can help to have easy cross-environment view of the repo (where each branch is checked out in its own worktree), while still being able to cherry pick from one branch to the other.

[@DevOpsToolkit]

I was referring more towards something like Argo CD or Flux manifests that contain only the references to the repo with manifests and environment-specific overwrites. In those cases, there is nothing really to compare since almost everything in those is specific and common specs are somewhere else.

[@edd6927]

OMG I honestly LOL'ed. Sadly because I am at the end of the spectrum he described xD :(

[@andriyun]

I can easily share your feelings 😀

[@taylow1]

Great video! Could you clarify how we can handle with hotfix using feature branch strategy? I’m not sure how to not interrupt the development flow of new features? (In this case there is only one development environment)

[@DevOpsToolkit]

Hotfixes are actually easiest ones to deal with since they tend to be very short lived. Create a branch from the mainline, push the hotfix, create a PR that will run whichever pipelines steps you might need to run, and merge it back to the mainline.

[@PelenTan]

Ok. I've got something set up for my team that is a bit different than all of those. Our current release schedule is every two weeks, but that's primarily because what we inherited from the last team was a total mess. No testing. And deployment is: merge to master on the public repository -> merge to a feature on the private repository -> merge to an svn repository -> go to all the locations that have code and run svn up. All command-line. Inside containers hosted on a single-worker k8s cluster. Which has tons of code that isn't stored on git or svn. Container images? Surely you jest! So... yeah... that's what I had to start with. As soon as we get things sorted out, with automated building and testing, ramping up the speed of deployment should be easy. Anyway, if you're still reading, here is what I set up. There is a fair amount of redundancy, but three sprints in, it's pretty stable. Master branch. Off master, a dev and a qa/testing branch. Also hotfixes come off and merge to master. Dev branch. Feature and bug branches off these. All work must be broken down into bites that only take a few days to complete. Once ready, PR back onto dev for dev review. Once passed, PR same branch to qa/testing. Currently all testing is being done manually. But they are setting up the specs to automate things. Once the testers get notification of the PR, they run the needed tests, including making sure the ticket fixes what's requested. They approve and it gets merged onto qa/testing. Once the QA team is happy with the number of features/bugs added, they run final tests to make sure nothing added breaks anything else. Then a release branch is created. Release branch, once we get the new architecture completed, can be pushed to a "public" test-deployment for user acceptance. If all is good, merge to production and deploy. I know this may seem a bit convoluted. But I think it will allow for complete automation with needed checkpoints built in. Ideally, once the work branch finishes dev review, there is no more human interaction until the UA point. And then it's out to production. Thoughts?

[@DevOpsToolkit]

I think it's amazing what you did considering where you (your company) was.