Continuous Integration vs Feature Branch Workflow

🎁🎄🌲🌲🌲🌲🎁🎄🌲🌲🌲🌲🎁
HAPPYCDNEWYEARFOR2024
🎁🎄🌲🌲🌲🌲🎁🎄🌲🌲🌲🌲🎁
Continuous Integration and Feature Branching are both very popular ways of organising work in a development team, but they are mutually exclusive for most cases. This is counter to what many, maybe most people think. In this video Dave Farley explains the difference and why the two are largely mutually exclusive, and then explains how to live in the CI world by describing three different approaches to keeping the software working as it evolves.
Software engineering, any engineering, is all about trade-offs, the trade-off at the heart of CI is that to avoid conflicts our aim is to integrate our code with that of our co-workers as close to “continuously” as we can, that means that we can’t afford to wait until we are finished. If we take that idea one step further into Continuous Delivery, then every change may end up being deployed into production, so we need to grow our software, through many small changes, committing multiple times per-day, and be comfortable that, at any point in that process, the software may be released into production. How do we use ideas like dark-launching, branch-by-abstraction and feature-flags to help?
RERERENCES:
Trunk Based Development ➡️ trunkbaseddevelopment.com/
Read More About Branching ➡️ martinfowler.com/articles/bra...
-------------------------------------------------------------------------------------
🎓 CD TRAINING COURSES 🎓
If you want to learn Continuous Delivery and DevOps skills, check out Dave Farley's courses
➡️ bit.ly/DFTraining
📚 BOOKS:
📖 Dave’s NEW BOOK "Modern Software Engineering" is now available on
Amazon ➡️ amzn.to/3DwdwT3
In this book, Dave brings together his ideas and proven techniques to describe a durable, coherent and foundational approach to effective software development, for programmers, managers and technical leads, at all levels of experience.
📖 "Continuous Delivery Pipelines" by Dave Farley
paperback ➡️ amzn.to/3gIULlA
ebook version ➡️ leanpub.com/cd-pipelines
📖 The original award-winning "Continuous Delivery" book by Dave Farley and Jez Humble
➡️ amzn.to/2WxRYmx
📖 "Extreme Programming Explained: Embrace Change", Kent Beck ➡️ amzn.to/2GpQRjE
📖 "Accelerate, The science of Lean Software and DevOps", by Nicole Fosgren, Jez Humble & Gene Kim ➡️ amzn.to/2YYf5Z8
📖 "The DevOps Handbook", by Gene Kim, Jez Humble, Patrick Debois & John Willis ➡️ amzn.to/2LsoPmr amzn.to/2LsoPmr
-------------------------------------------------------------------------------------
📧 JOIN CD MAIL LIST 📧
Keep up to date with the latest discussions, free "How To..." guides, events and online courses, and get Dave’s FREE guide on “Continuous Integration Top Tips” here ➡️ www.subscribepage.com/howto-c...
Dave Farley's Blog ➡️ bit.ly/DaveFWebBlog
Dave Farley on Twitter ➡️ bit.ly/DaveFTwitter
Dave Farley on LinkedIn ➡️ bit.ly/DaveF-LI

[@Crozz22]

This assumes all my changes are always incremental. In practice I often tear down most of what I did after I've cemented the general idea. To introduce my initial ramblings to the team before I even know if it's the direction I want to go, sounds like a whole lot of work for nothing.

[@_skyyskater]

Great point, but you can still do CI here. If you plan on throwing away the branch, that doesn't count and is totally fine to take your time on it. Just be aware things could change during that time. Create your "draft" branch, do your thing, then when you are satisfied and ready to write it for real, check out a new fresh branch and do your changes there. Then commit and submit PR. Done.

[@MidnighterClub]

I've worked for a large firm (close to 100 developers) and we were divided into smaller project teams (10 to 20 people) each responsible for a particular feature set and launch window. "Assumes all changes are incremental" is the problem here. Large changes are not useful to the rest of the team until they are completed, or at least nearly complete (ready for system testing). Working on a branch gives the project team time get their stuff into a usable state, then it gets merged with top of tree, but not before.

[@_skyyskater]

@@MidnighterClub You can break work into smaller pieces. It doesn't have to be useful to merge, it just can't cause regressions.

[@JD-mz1rl]

Work on your local branch and don't push to origin until it's ready. When complete, do interactive rebase, fixing up all the irrelevant commits into one (or as many as logical), then PR that set of logical commits.

[@_skyyskater]

@@JD-mz1rl That only works if much hasn't changed, or if your changes aren't massive. If this is working for you, then either your team is probably already breaking things down small enough (good), your application/team hasn't really scaled yet, or you aren't making many big changes.

[@aprilmintacpineda2713]

As someone who has experienced both sides, I can say that there's no perfect solution, it's basically a "pick your poison" situation.

[@bartdart3315]

So true.

[@RadioWhisperer]

Especially true as one thing that's missing from this approach is code review. If every developer on a team integrates every day that's 1 code review for 2 or 3 other people every day. Granted they're small changes but reviewers have to understand the big picture behind changes, otherwise all they're doing is reviewing syntax variation and coding style. A big part of review is does the change fit the spec and the desired end result of the bigger change. Granted review against bad coding practices is important, but it's not the only thing.

[@jordanfox3782]

I am pretty sure most of us have experienced both sides. There are planty of things that can go wrong in either side, one is less efficient but has the upside of hiding merge conflicts until the end and gatekeeping the codebase from bad developers.

[@adityaprakash5093]

We changed our process after this video few months ago and that instantly solved lot of code conflicts problems that we were facing and allowed us to visualize the progress of the day. Thanks a lot for the content

[@ContinuousDelivery]

Thanks for the feedback, I am pleased that you found this helpful.

[@gregzoller9003]

Branching is valuable. Many times I use a branch for experimentl changes/refactorings that I’m not yet sure should ever be merged-at least until they prove themselves. If it doesn’t work out, I just delete the branch. Other times I use a branch to make significant refactorings I don’t want to disturb the main code, which is still moving. Its not always possible to think of work as a stream of small incremental changes.

[@Thejasonlessard]

How do you do code review before a commit to the main branch if you are not branching and submitting merge requests ?

[@m.x.]

Having a dev branch from which you can create one single, big PR to be merged into master? Not sure tbh

[@ultrasoft5555]

I guess with the more frequent merges team members are more aware of the work of each other. And smaller, daily merges mean quicker reviews. Merge requests are not the only way to review code, anyone at any stage of the development can just read the diffs. Maybe reviewing is also better if continuous.

[@whitewittock]

what do u mean, you don't need to commit changes to create a code review you can use something like Code Collaborator to make a code review based on your local changes

[@johnkeck]

Code review does seem rather more complicated in the CI context.

[@SuperSlugger94]

Gerrit accomplishes exactly what you are asking. You push your commit to the main branch but Gerrit does not push it to remote origin until it has been reviewed.

[@msintal1308]

Im not even a programmer but, I'm loving this channel. Found it through Cyberpunk criticism. Very fascinating!

[@thereal_maximg]

cool, don't stop learning

[@MrThogin]

Me too...

[@kaitsurugi3280]

Me three! :D

[@ContinuousDelivery]

Glad you enjoy it!

[@destroyonload3444]

Sounds like your destined to become a programmer...

[@johnnyw525]

I was about to write a comment on how this was clearly bullshit. I watched the video instead. By the end I got completely what you were saying, and it makes perfect sense. CI isn’t something you can dabble in, and this demonstrates the level of commitment required to do it right. But if the data says it’s the most efficient way of working, then it’s worth seriously considering. Thanks!

[@AlexAegisOfficial]

My preferred way of dealing with staying updated with feature branches is frequent rebasing, this also allows me to have a linear history instead of a git history looking like a weaved basket. Gerrit is a very interesting Git host, it automates a lot of fiddling with branches to the point development can go on for months without even thinking of creating a feature branch. Because in Gerrit you work with patchsets (internally they are just branches where only the head is used, so it acts like a single commit, with all its history still available) and just "replace" commits, and you can do this as long as you like, rebasing it, running CI against it, until you merge it back to the parent branch.

[@laughingvampire7555]

rebasing is cheating

[@Inversions51]

Thank you for sharing this! When the video started I thought, "this is a terrible idea", but the methods you presented for safely dealing with partial features make a lot of sense. I think branches still seem to make sense for open source projects because of the distributed decision making process in the community. But inside a development team I can definitely see this being a lot smoother.

[@henristahl7203]

At what point do you perform code review if you commit to trunk every 15 minutes?

[@ContinuousDelivery]

All the time, with pair programming is my preference. I talk about that in this video: kzbin.info/www/bejne/l3rXh32mp7GMock

[@vyli1]

​

[@JulianSildenLanglo]

@@ContinuousDelivery But with pair programming you end up with two authors of the code so now you need a third person to review it.

[@ContinuousDelivery]

@@JulianSildenLanglo Actually my prefered way of working is pair-programming with daily pair-rotation, so there are more than 2 authors and for big stories nearly everyone on the team will have worked on it. At which point an extra review is redundant. I work a lot in regulated industries, there is no regulatory framework that I know of, anywhere in the world that I have worked, that won't accept pair-programming as a code-review.

[@danielevans1680]

@@ContinuousDelivery Have you always lucky enough to have an even number of developers in a team, or have you had cases where one dev a day had to do some lone working? (I'm coming from an even worse situation, where I'm in a team as a single software dev - nominally there are two of us, but the other person has an astoundingly large project management workload, and hasn't committed code in months)

[@BigCarso]

So if Bob is working on the same file as me, then I'm supposed to pull Bob's changes every hour, understand why he broke my feature then rearrange my feature and break Bob's feature the next hour? Sounds like no fun. Never had a problem with merge conflicts and much prefer to have complete ideas to compare and merge.

[@ContinuousDelivery]

Yes, you and Bob should merge your code at least once, but ideally many, times per day to qualify as practicing Continuous Integration. Sure, you may like the way that you work, but the data says that you are more likely to be producing lower-quality software more slowly as a result. That is not me saying that because that is just my guess or preference, that is based on an analysis of 35,000 software projects. My video is an attempt to explain why that fact is the case.

[@BigCarso]

@@ContinuousDelivery Did you provide a link to that data?

[@BigCarso]

The data doesn't say that at all.

[@MiningForPies]

@@ContinuousDelivery I’m sorry, you have not analysed 35,000 software projects and come to that conclusion - that is utter utter rubbish.

[@Luxalpa]

@@BigCarso Ridiculous response for someone whose beliefs got challenged. Nobody requires you to adapt your work style to new information. If you like what you were doing or are too ignorant of realizing the problems you have, that's fine, just keep doing it, no reason to change. However, many companies are struggling with their systems, and in particular the problem you mentioned is one of the main causes: If you two break each others feature all the time, then something is already going seriously wrong, and in such a case, a slower merge (like a 300+ conflicts merge) would be a complete disaster both in terms of time investment as well as in terms of keeping the error rate low. It also seems like you haven't watched the video, because it's pretty damn obvious that a change with fewer changes takes less time to merge than one with more changes. Really not that difficult to understand.

[@jefferygriffith]

I feel like this would have been better received if it had been about branch lifespan/scope rather than “don’t branch” since branching is the typical means of code review. It seems to me branch/review/merge often is compatible with the CI definition and provides the same benefits you describe. Thanks for the video!

[@objarni]

If you have a team that enjoys reviewing code, that'll probably work! Also, if reviews are done in pairs, even better - then you will get higher bandwidth of communication than the async nature of typical "chat based" reviews.

[@objarni]

If however either of those are not true (code reviews take 1-3 days to complete or reviews are done through comments on a pull request), you are missing out on something here

[@lvleminckx]

Can you get your devs to do code reviews DAILY though ?

[@objarni]

​​@@lvleminckxthat's why pairing or mobbing is my preferred way of working. Code reviews just create lagging or queues at a system level = low flow

[@AdarlanTeixeira]

Your videos are great and inspiring! Thank you!

[@ContinuousDelivery]

Thank you

[@mmick66]

I just want to express my gratitude for your content. It’s really professional stuff. As a software developer myself, yours is one of the very few channels I can actually still learn from. Bravo!

[@ContinuousDelivery]

Thanks, I am pleased that you find my stuff interesting.

[@akemp06]

Thanks David for all your explanations! Very helpful

[@alittlelightcse2762]

Best programming content on KZbin, can’t get this level of experience and amount of useful info anywhere else

[@strangnet]

You can always integrate your work by syncing/merging the trunk with your feature branch. By keeping your work current with what happens in the trunk you ensure that other features that are introduced work with your changes all the way until merged back into trunk.

[@ContinuousDelivery]

Yes, but that only solves the problem as far as what has been committed to trunk, if everyone else is Feature Branching too, then you have no sight of what changes they are making until they merge to Trunk. This is certainly better than not merging frequently from Trunk, but it is not the same thing as CI.

[@strangnet]

@@ContinuousDelivery that is true, but the only thing I need to consider is what the trunk has, which is "the truth" that is in production. Same for others working in their feature branches. This is however when people work on things without knowing anything what others are doing, which should be rather uncommon unless there is no communication in the team - the occurrences where different team members work on the same bit of code without each other knowing should be rather minimal so "collisions" should not happen unless the code is badly structured as well. By continuously testing my code with the latest changes introduced in trunk I do get CI, I don't see how it isn't. This also helps when my changes are about to be merged into trunk with a pull request, so that other team members get to see if any changes are in fact in line with the current code base and follow the guidelines and principles/best practices the team has decided.

[@ContinuousDelivery]

@@strangnet The problem is that if you have a team of people all working like that then except for the last person to commit, trunk isn't really trunk in the sense of the version of code that will go into production. So everyone is testing against a version of the code that wikll never end up in production. In true CI - the definittion of which includes the statement "merge changes into trunk at least once per day" this is not true, at the point of every merge, trunk is definitive. All this is just our opinions though, the data says that if you have feature branches that last for longer than a day, wether you merge from trunk or not, statistically you produce lower quality.

[@DarrellTunnell]

Some features don't pan out or take longer to complete or require several user stories to be done before the feature could be considered complete. If that code is integrated immediately then should it not pan out you have to "unintegrate" it. Sure you can use feature flags but if you have to pull the feature you have have to remove / undo that code. Forward integrating feature branches on a daily basis is the cleanest solution imho.

[@FlaviusAspra]

@@DarrellTunnell a branch should not live longer than a what a sprint is. It can start in the middle of the sprint, and be finished in the middle of the next sprint.

[@zpinacz]

Nicely explained! Thanks

[@Lascarnn]

Very well spoken, constructive and informative. Thank you!

[@alexanderoh1847]

I really enjoy your videos and content. While I don't feel I'm radically enlightened I see these as a great opportunity to share them with less experienced engineers, as the clarity of your arguments is amazing. Thanks

[@ContinuousDelivery]

Thank you!

[@egor.okhterov]

Create branches and rebase often. Happy life :)

[@AndriiMuliar]

Fail othen and fast

[@c64cosmin]

@@kajacx It means you were the only one doing the rebase, if all the team does rebase often you will rarely get in that state, I've been using git rebase for several years and very rarely hit points where it was a hell. It is hell if you are the only one doing a rebase.

[@thomasBura]

The premise is that my small daily commits doesn't break the app even it is not in production, so my colleagues can work on their tasks. But some tasks with a lot of refactorings take days to finish and it could make harder for others to work. Also you would get rid off pull requests that should be reviewed. But don't get me wrong I hate long term branches and dealing with conflicts so we try to minimize size of branch to minimum. Also feature flags are good approach we implement very often. What I prefer is to pull from development branch often so I keep up with current state.

[@andrealaforgia]

I would argue that refactorings are the best use case for trunk-based development. If you are refactoring a large part of the code, you’d better ask other people to stop working on that code or you’ll end up with a disastrous merge hell. If you can organise a refactoring as small batches of work to be integrated frequently, definitely do it.

[@ContinuousDelivery]

Yes, I agree with that approach. TBD is a MUCH better way to support refactoring.

[@carveratutube]

@@andrealaforgia so you arguing that if you doing a refactoring the 5 other people in your team need to have a pause 🤷🏻‍♀️ Well I agree that while doing some heavy refactoring, generally, nobody else should work on the same code part. But that doesn’t mean you need to do it in the trunk. Refactoring can sometimes not work well and it’s much easier to verify and to revert changes which were done in a separate (feature) branch.

[@ronaldomorillo3789]

If your feature requires changes to shared libraries and database schema, how would you handle potential breaking changes to other areas of the code if you don't isolate them by branching?

[@jonnyevason2219]

Love Dave's channel.

[@ContinuousDelivery]

Thanks 😎

[@simonegiuliani4913]

I worked with both branches and incremental changes on master. They are both valid. If there are more than 15 devs in you team it will be very difficult not to spawn new branches.

[@retagainez]

One of my favorite channels to watch, I enjoy the style of the videos :)

[@ContinuousDelivery]

Thank you.

[@IanStevensZA]

or just merge from the main branch into feature branch every day or even after every other person merges into the main branch. It still keeps your code close to the truth and avoids stepping on other people's toes

[@75yado]

well it's merge up from MAIN and run the tests but the problem is when you merge your work to the MAIN, Others should get the changes and run tests on their work and if they are close to release they could get to the state that their long work is no longer relevant and they would have no more time to fix it so they cannot integrate their work to MAIN and the reason is lack of informations about what do the rest of the team doing

[@AloeusCapitalManagem]

@@75yado uh that shouldn't happen that sounds like dependency hell

[@75yado]

@@AloeusCapitalManagem no just too many people working on same system. Usual problem was when someone changed something in the geometric kernel or similar module which everyone used

[@redhotbits]

that does not work

[@RandyLutcavich]

That only solves half the problem. You need others to take your changes as well, that only works if you push to Main.

[@andyd568]

You are the programming mentor i was always looking for but never found. Well now you are here. Thank you for your videos.

[@ContinuousDelivery]

Thank you, welcome to the channel.

[@precumming]

Don’t listen to them, they don’t know what they’re talking about and what he is proposing would just get you laughed at by developers because it’s a stupid approach where he assumes that nobody can communicate. He was late to version control and is confused by it, teams can manage branches and reduce merge conflicts down to 0, especially with microservices (which he is against for stupid reasons)

[@Adams456]

@@precumming I fell the same, seems he never worked in a big team, it is so horrible in a real world, he talk about downside of feature branch against continuous delivery but miss to mention the pros

[@CottidaeSEA]

The definition of continuous integration sounds like 10 developers coding live on the same computer but with different keyboards.

[@Martinit0]

It's like working together on a Google Doc

[@KevinBronsdijk]

Interesting topic. I believe that this is the way forward; however, one concern I have is related to code reviews. Im not worried about syntax relates issues; this can be automated using static code analysis. I'm mainly concerned about the architecture/implementation of the feature as a whole. When you make all your feature changes within a single branch, you will have a holistic view of that feature's design and implementation (something which can also be used as a reference). This is something which you will lose when committing directly to master. Just wondering how you have solved this problem. Maybe it's as simple as tagging the commits with a feature id, but review tooling just isn't ready for this scenario.

[@ContinuousDelivery]

Yes, I think that needs more care. The best way to achieve that design overview is pair-programming linked with regular pair rotation. I describe that, and some of the reasons why I value it so highly, in this video: kzbin.info/www/bejne/l3rXh32mp7GMock

[@betowarrior_]

I was going to ask that exact same question.

[@milavotradovec1746]

@@ContinuousDelivery Nice video :-) I have to say I usually have different opinions, but I really appreciate your work and ideas. They make me think :-) Pair programming is usually advised as a solution for absenting code review. However, I would be interested in your opinions about pair programming in terms of distributed team - not only geographically, but also across multiple timezones. There might very small overlap suitable for pairing. Thanks again for all your work, I really enjoy thoughts it sparks in my mind.

[@ContinuousDelivery]

@@milavotradovec1746 Thank you. On remote pair programming, I once worked for a financial trading company, based in Chicago, I worked out of the London office, but the team that I worked closest with was based in Chicago, a 6 hour time difference. We paired during the couple of hours of timezone overlap and found it helped strengthen the bonds in the team. It is different to full-time, local, pairing though I think. I think that it helped the team a lot, but there was also a lot of work that was done not-pairing. The way it worked best for us, was I had some stuff that I was working on and the people on the team in Chicago had some stuff that they were working on. I was the more experienced dev, so that it nearly always ended up that I paired on their stuff. I'd give it a go, and experiment a bit with what kind of things you choose to pair on to find out what works best.

[@milavotradovec1746]

@@ContinuousDelivery Thank you for your reply and sharing your experience. One more question - when you worked on your stuff alone, were you still committing directly to the trunk (main, base, whatever :-))? If so, how were your colleagues in Chicago aware about the changes you did? Thank you very much for sharing this, I find it very inspiring.

[@nelsonmacy1010]

I have your books and all the Devops , Martin Fowler series. You ability to speak clearly and know all the pitfalls is A+. Steve McConnell is also someone I deeply respect. Maybe you can comment on his work in a video or have a joint discussion. Thx for all your hard work.

[@fauredaniel57]

Espectacular Dave! I loved this

[@OhhCrapGuy]

Agreed that they should be very small integrations, integrating at least once a day, but committing and pushing to master allows junior developers to add flaws to master without code review. Merge commits from work branches to main allows associated work (first attempt, write tests, second attempt, add comments) to be grouped as part of a single commit. The commit graph is much more clear about when and why certain changes were made, and a rebase to remove problems can be MUCH simpler.

[@ContinuousDelivery]

Or do pair programming, even simpler, and the junior devs get to learn faster.

[@RowanGontier]

I like the principle of continuous visibility of changes. However, regardless of feature branch vs CI, there are few people who will 1) actually look at the commit logs of changes pulled, or 2) see the changes made by others when working on their own ticket. The only thing that wakes people up is merge conflicts. Must say, among the merge conflicts I face using feature branching, very few are due to mismatches based on a fundamental approach; it is minor things like imports/formatting. I like feature branching because a reviewer can see things like architecture of a feature within a PR. I fear that CI will make people approve minor changes with less critical thinking about the big picture.

[@mydadletsmeshootatcats6754]

I have the same thoughts about code review on larger features. It seems like it would be impossible to do proper code review. I would love to hear Dave's thoughts on this. The other thing I don't like about committing incomplete features to the master branch is that it makes it difficult to identify zombie code. If you see code that doesn't appear to actually be used, is it zombie code or is it a necessary part of an incomplete feature? You'd have to actually ask the developer who wrote it. What happens to features that were abandoned half-way through? Seems like clean up for that would be a PITA as we can't simply revert the merge commit for a feature branch because the feature was committed via countless tiny commits.

[@danepane527]

this 100%

[@foxy_robin]

@continuousdelivery I would also greatly appreciate your thoughts on the above situation. A FB PR gives a more holistic view on what the problem being solved is and the architecture decisions made to deliver it. The reviewer can more easily see if there is an issue. However a senior reviewer would likely notice an architecturally poor choice very early on, thus saving the junior dev from continuing down the wrong path.

[@richardcomblen5928]

Hi. Big fan Continous Delivery (the book) and Accelerate (the other book) here. I've been implementing build processes for more than 10 years for a big variety of projects and I'm a fanatic on software quality. I agree with all the branch by abstraction, feature toggles and dark launch. I have a fundamental disagreement with forcing to work with a single branch. Let me explain my point of view: I want to have the highest software quality. To do so, I automate all the possible quality checks in an automated quality gate, typically: - stupidity checks (duplicate DB migrations, leftover console.log etc ...) - all the necessary linters and static code analysis to spot code smells - unit tests with high code coverage metrics (for code with cyclomatic complexity) - architecture rule enforcement tests (using ArchUnit) - mutation testing - software dependency scanning for vulnerabilities - end-to-end testing for most of the features of the application - UI regression testing with screenshot comparison Well implemented, those checks take 40 minutes wall clock time on a web project after 10 men-years of development (using a standard Spring + Angular stack). I want to deploy to production at least multiple times a day. At any time, tip of the main branch should be production ready. No-one should ever hold his breath before a change getting deployed to production. I've been doing that successfully, with high throughput, over multiple years. How: using short lived branches that have the full quality gate running before being merged. The development cycle is the following: - pick a task to work on - start a branch - as soon as you have some change whose quality is high enough, push, make a self-code review, and get it merged (after pair-review if needed) as soon as the quality checks are ok. - You don't need the task to be complete to merge, you just need the quality to be right, to merge. - You'll typically make quite a few merge requests before completing the task Typical cycle (between branching and merging is less than half a day). If someone wants to keep a branch longer than that, no big deal, he merges the main branch into his development branch as often as necessary (I recommend at least once an hour). He'll have the burden of merging, as he is the one choosing to move away from the "short-lived branch" recommandation. His choice. The higher the threshold of the quality gate, the more frequent the checks are going to be red. If it's red on the main branch, we cannot deploy, so we test before merge. You understand that with such a test battery, considering pipeline 10% of the pushes leading to build failure means the main branch will be red more often than not. This means team blocked hunting for the change failiing the build. Testing on the branches allows as well to: - test the build process update before merging (new compiler chain, different code coverage tool, etc ...) - review app: review UI changes live before they get merged by deploying the branch to a short-lived live environment - code reviews for knowledge sharing (I've read your arguments for pair-programming rather than code review. I'm a big fan of pair-programming, but it's just not always possible, sometimes simply because the work schedule is not uniform among the team members). Conceptually, a developer is always branching. He's working on a local copy that IS kind of a branch. Let's just consider the short-lived branch the equivalent of his local copy. If one is force to push on trunk, he'd better run at least a significant part of the quality checks before pushing. Running well written unit tests for a big project can easily take 5 to 10 minutes (we have many thousand of unit tests). I should run that locally before each push ? Why doing it manually, when tools like Jenkins, Gitlab-CI and many others can do that for you ? Hardware is much cheaper that developers, auto-scalable build farms are just a click away. Short lived branch allow this process to be asynchronous, and let the developer do something more interesting than waiting for the build to be ok on his laptop before pushing. I've now moved on a project that advertise doing trunk-based development (based on your very arguments). Tools have no build-on-branch support (GOCD). Quality gate only contains unit tests and some end-to-end tests. There are just a few deployment to production a month because confidence on the quality is not high enough. There are more days where build gets red at least once than not. I have a strong feeling trunk-based development gets in the way of having a strong quality gate. You can't have both, and without the later, you cannot safely do continuous deployment for mission critical apps. I suggest the following motto: "Long live short-lived branches" (no copyright). I'd be glad to read your feelings about this.

[@ContinuousDelivery]

I think that we are 90+% aligned here. A few thoughts... If your "typical cycle (between branching and merging is less than half a day)" then you are doing CI, so I have no argument with that at all. I think that you are doing more typing than me, I work on master locally and then merge all my local commits immediately that I have made them to origin/master. So I don't have to create any branches or merge to them (less typing for same effect that you describe). To be honest, this doesn't matter at all, if you want to type a bit more it doesn't matter. "The higher the threshold of the quality gate, the more frequent the checks are going to be red" Not so! The slower the feedback loop the more often the checks will be red! Adding branches slows the feedback loop. The problem, as you correctly point out, is the efficiency of the feedback loop. One of the more subtle effects of feature-branching, in my experience, is that it gives teams more room to slip into bad habits. Inefficient tests being one of the most important ones. If everyone is working on Trunk, then slow tests are a pain, so the team keeps them fast. I think that most of the rest of what you are describing is about feedback efficiency. Let's try a thought-experiment... If you could get the answer (is it releasable) in 1 minute, would you bother branching? If not, then what we are talking about then is where is the threshold where the efficiency gains of hiding change (branching) outweigh the efficiency gains of exposing change (CI). I think that CI wins hands-down as long as you can get a definitive answer within a working day. Practically, the shorter the feedback cycle the better, but my experience is that under 1 hour is the sweet-spot. That gives you lots of chances to correct any mistake during the same working day. I built one of the world's highest performance financial exchanges and the Point of Sale System for one of the UK's biggest retailers, and we could evaluate the whole system in under 1 hour. So my preference is to spend time on optimising builds and tests, rather than branching.

[@richardcomblen5928]

@@ContinuousDelivery Thanks for the feedback!

[@Ays536]

Your topics are very informative, these are hot topics and our teams discuss them in the retro. I personally, enjoy reading the different perspectives in the comments.

[@JimmyGee]

Wow! I'm happy to have found this channel. Great videos!

[@beepboop533]

Can't wait for all the supporters in here to get in the field and deal with the struggles this methodology lends itself to. Hopefully they won't become cultists over this and one day will be willing to give branching a chance.

[@ContinuousDelivery]

Maybe you should try it? I have tried both and know which I prefer.

[@FlaviusAspra]

@@ContinuousDelivery I highly value your ideas and videos, really. But this no-branch attitude lends itself only to really senior developers. Many other people have mentioned code reviews. The code needs to be reviewed. Why? Because programmers are sometimes too creative with their design, especially the mid-levels who see themselves as seniors. I do agree with a few ideas and it's how I do things: - all branches have to be releasable at any time; developers merge the master branch into their branch every 15 minutes / once a day. This is continuous integration - once the code has been reviewed, it can be merged into master. This can happen once a day, or once every few days - the developer switches back to his branch and continues to work and the cycle restarts This way: - we protect juniors from their own mistakes - we protect the design - we can do pair programming, but doing it all the time is just too much; trust me, I know my juniors - we have design meetings; problem: the junior of the team does not really see the same imaginary code as me when we do the design - and yes, I explain a lot, we do enough pairing, but at the end of the day time efficiency also matters. And releasing often bad design is not time efficient I'm a CTO at a smaller company btw.

[@PrinceRambade_Official]

This is much closer to what I was looking for....while youtube is flooded with basic to advanced GIT tutorials but nobody talks about professional implementation and use of CI/CD with lot of confusion on branching.

[@MiereTeixeira]

I've worked with a bloke that had the opportunity to work with you back a decade ago. When I'm watching to your videos I can totally imagine he explaining his values, approaches and techniques. Don't mind the people that didn't get what you're trying to explain them. They will, certainly, learn this one way or another. As more videos comes, I'm sure you'll tone down a bit and reach to wider audience. We need this wisdom. Keep pushing mate. Awesome content.

[@ContinuousDelivery]

Thanks, who are you working with that I know?

[@trigun539]

Thanks for the great videos, but there is a basic flaw with your logic here. That is that the feature branches are somehow completely broken apart from the main branch or trunk. There is nothing wrong with having a long lived feature branch as long as you keep merging the trunk back to it. Your graphic is not quite complete, it only shows the merge of a feature to the trunk. But there are many trunk to feature merges along the way while the feature is being built. This can be done fairly easily and this way your feature is always updated with latest code. The feature itself can be tested completely through CI and if tests pass then you are good to merge your feature to trunk. Adding feature flags, dark features, etc involve huge amounts of extra code/tests which are not necessary. Most of the time this will probably add performance issues and extra unneeded code. In some instances this could be impossible if the new feature demands a DB migration or something similar. Having feature branches or even long lived features branches doesn't mean you are not continuously integrating.

[@ContinuousDelivery]

Well, syncing with trunk for a long-lived FB is a good idea, but it doesn't solve the problem of what is going on on everyone else's "long-lived FBs". I am not saying that an FB is "completely broken" I am saying that it is not the truth and you can't tell wether it is completely broken or not. You can reduce the chances, but you can't tell until you merge into the version of changes that represents the "truth".

[@upgradeplans777]

Edwin, I think you are correct in the observation that dark launches, branch by abstraction, and feature flags sometimes require extra code. But I think the extra code is likely to only lead to a higher quality code base and/or better user experience. With extra code, I mean facilitating code by the way, I think you meant the same? Because there will of course be multiple "feature variants" in the code base, but that is not different with FB, just in branches instead of files. For dark launches, actually no facilitating code is strictly required. For branch by abstraction, the facilitating code (when done right) are mostly interface definitions and IOC mechanisms. Chances are the project will have these anyway, or otherwise are a good addition for other reasons. Lastly, feature flags are a bit more complicated to evaluate. First, they require facilitating code that many projects do not have and would only be a low priority for other reasons. Second, naive implementations impose a configuration burden on the ops team (aka yourself if devops). In my view, feature flags should therefore not be configured, but be exposed as a user preference setting (this can be an internal user of the code, or the end-user). If this is not desired by the user advocate (ux designer, productowner, whoever), then I can probably be convinced to use one of the other strategies entirely. But if the feature flag is desired by the users, then the facilitating code actually improves the user experience, and is not a burden at all. For these reasons I completely agree with Dave that feature flags are more "risky" than the other mentioned strategies. That said, I have in the past dark launched a feature-toggling interface to great effect, and I can't imagine any user advocate not wanting this for themselves (even if they want to keep it hidden from their users indefinitely). When done correctly, the facilitating code should not introduce performance issues, or this indicates other problems. With DB migrations, that should be done by a branch by abstraction on the DB schema itself anyway :-)

[@dainiusfigoras]

pain with long lived branches are not in master, it's in branch. So you can update your branch as many times as you want, but others will not see your changes. And you get merge conflict not because of you pulled changes a week ago, but when you try to merge 2 files that edited same lines. And when you have long lived branch chances that someone else edited that is higher and these people introduced changes in their own branches and then when you want to merge you get a mess, because you have one logic there, another logic here and who is right? If you have highly decoupled work, where you don't work with same code as others, then long lived branches will not create conflicts. But they (long lived branches) will increase business value delivery, regardless.

[@mynxeram]

just merge the current "develop"-branch (source of truth) into your feature-/bug-branch before starting to work on it again, at least every day. It's that simple. Nothing else ever worked with > 10 Devs on one repo. Channels like this show, how far away the theory could be from anything practical :)

[@Mio2k10]

This approach kinda assumes that no one makes mistakes. Imagine someone is committing every 15 minutes. After 2 hours someone realizes that there is a huge issue with the new code which affects everyone's ability to work on their own tasks. Writing tests won't always help. Things can slip through. Also writing tests that early is often times a waste of time because you don't even know what the final version will look like. Thanks for your video tho!

[@defeqel6537]

Tests should be written on the feature-level, the implementation details don't matter. This is not always possible, but I'd say in 99% of cases it is. Not that I'm convinced that branches should be avoided, but I do think that frequent merging would be useful (again, in 99% of cases), but needs to be done right.

[@Mio2k10]

@@defeqel6537 Well, there are different kind of tests. Every test kind has its place. I don't see how this is referring to my comment btw. Frequent merging is good, indeed! But you want to merge a fully refined increment not a WIP state to reduce unnecessary friction.

[@defeqel6537]

@@Mio2k10 I agree, and disagree. Generally, yeah, fully refined increments should be preferred, but as long as it is flagged out, or similar, I'm starting to see a benefit with WIP state changes being merged too. I do think merging WIP requires care though, otherwise may lead to just frustrating your team members. As for testing, my point was that the tests (usually) shouldn't rely on what the code looks like, so they may be written quite early on, after a bit of planning. As for test types, I prefer unit tests, by which I mean code I control the inputs and outputs for. IMO people should avoid micro-testing, where they try to test each function or class separately (unless those are your deliverables, in which case thoroughly test them).

[@pocoloco453]

This was awesome!!

[@rogerpetruki4759]

Great video, as always Dave. It's good to know that some of the strategies to avoid branching have to do with a great effort I've been putting into releasing a more intelligent and manageable Feature Flag platform. Thanks for sharing your thoughts.

[@ContinuousDelivery]

Thanks

[@MuhammadAbusaa]

I think , this is good for small teams that consist of experts and seniors , but when it comes to large projects with many people and the skills vary, then we need a strategy "called PR" to review juniors code as example. Thank for your video . I enjoy it

[@ContinuousDelivery]

Well it scales very well, and I would argue that hiding what people do from one another (branching) may be a defensive strategy for less experienced teams, but is not going to help them improve. My own preference is to combine this with Pair Programming, that way you can mitigate the risk of inexperienced people making a mistake by pairing them with someone more experienced, and they also get the chance to learn more quickly. I describe my thinking on this in this video kzbin.info/www/bejne/l3rXh32mp7GMock

[@precumming]

@@ContinuousDelivery it does not scale well at all, you shouldn’t be teaching something that will lead to more problems. There are no benefits to not branching that can’t be solved with some planning and communication, just because you can’t doesn’t mean that everyone else shouldn’t. People should learn to use branches because that works best, and merge conflicts only mean there was a hole in communication.

[@Stylpe]

@@precumming Are you speaking from experience of having tried proper CI, or are you just pessimistic because you have no confidence or trust in your team or having been disappointed too many times?

[@edwardcullen1739]

@@precumming Notwithstanding that every working copy is a branch, every commit a merge... less experienced developers will more often mess up and take longer to complete complex merges... There are secondary benefits to the frequent integration approach - protection against device failure and interruptions, for example. I think the point here is less in the branching itself and more in totality of the approach; small changes, backed by automated tests... Smaller changes means less to go wrong / less to fix if there's a problem...

[@coderlifer4870]

Yes, this only works for small teams, small projects, or projects that are not mission critical. People who believe in this will be in for a big surprise when they joined a big company or a big project. Just wait and see.

[@KevinONeillPlus]

I see a lot of discussion that feature flags solve the parallel development issue, and I believe they do, to an extent, but they also increase the complexity of the code and make it more difficult to reason about. How does flag x impact flag y? Ahhh I hear you say, well that’s all handled in testing and ci. That’s true of feature branches as well. Branches increase process and development infrastructure complexity. Feature flags increase process and code complexity. Both require coordination and communication between team members. I’ve used feature flag extensively for rolling out experiments in the UI/UX space and they work well. They’re isolated and have a finite lifecycle (the length of the experiment). Cyclomatic complexity has long been shown to a have a major impact on system maintainability. Feature flags by their very nature increase cylomatic complexity. PS: Buying Daves book, reviewing considered opinions are always worth your time.

[@ContinuousDelivery]

I think that you can implement Feature Flags in different ways, particularly if you use them in combination with one of the other strategies. Feature flags don't have to be in the guts of the code if the design is good. But I agree with the point. I am wary of Feature Flags for a related reason the explosion of potential code paths - what do you test?

[@so_to]

Thanks for this awesome video. I hear what you are saying, my question is how should you handle code reviews in this case? If I understood correctly, the desired process here is to push to master as often as possible instead of hiding the changes in a branch, but it also means the code needs to be reviewd by someone else(teammate) every 10 or 15 minutes which in reality is very disturbing(in terms of context change for the reviewer) and blocking. What's your point of view on that?

[@mkschreder]

I would suggest using upstreaming for this. In such an approach the whole team is committing into a develop branch regularly - this becomes the trunk. Then separate "merge request" branches are created and relevant code is checked out into them using "git checkout develop -- " and then these are reviewed and merged into master. After that master is merged into develop to bring the two branches in sync.

[@d1morto]

@@mkschreder Then you're back to multiple branches that aren't always in sync. I would suggest using short-lived branches that are merged within 24 hours. Then, you'd need to make sure your team is reviewing and approving PR's daily.

[@mkschreder]

@@d1morto yeah but that doesn't always work to review daily. The idea here is to commit to develop and then move commits into review branch (or just move changes) I agree that pushing the team to review daily is important but some teams are very slow at this and can take a week to review something that they may not filly understand.

[@d1morto]

@@mkschreder The devs would sort through the develop branch looking for their individual commits, and then move just those commits into a review branch? Wouldn't that be tedious?

[@ChrisG-sw5zm]

Another benefit to CI compared to long-lived feature branches - if there is a merge conflict, it is with the code that the developer wrote *in the last 15 minutes!* This code is fresh in the developer's mind, and they will have an easier time fixing the conflicts rather if the conflicted code was written days or weeks ago.

[@haseebs.7286]

9:30 (DevOps Handbook) I am not used to this workflow but I can see it’s merit. In my experience, the biggest bottleneck with feature-branch workflows tends to be code reviews (not branch maintenance or testing feature branches). However, code reviews are also the most important step for ensuring code quality and readability. Google/Facebook do NOT bypass this. I think the benefits the DevOps report outlines might be true with trunk-based: - higher throughput - better stability - higher job satisfaction - lower burnout However I don’t think it’s particularly comforting knowing people can commit anything at anytime without any oversight. I’ve seen codebases use this sort of style with just a linter and the commit history was a really long mess (and the code was a mess too). Really was a nightmare having to work off of that. feature-branch workflow compartmentalizes work much better and has a controlled process for integrating. Trunk-based might get you to the end faster but throughput isn’t everything. For example, a project that’s “released” but is being maintained with occasional bug fixes or other minor changes would not benefit at all from continuous integration. I guess it could work on completely green-field projects but most jobs are working on existing money-making products. Just to re-iterate, even Google/Facebook who supposedly use this trunk-based workflow STILL have code reviews. Ultimately the biggest bottleneck is waiting on others.

[@VictorMartinez-zf6dt]

You can do code reviews without needing to branch and can be done on each commit.

[@GaryvanderMerwe]

I'm a dev with 20 years professional experience. I disagree with this. Code should not land in master if it's not ready. It doesn't have to be perfect and bug free, but it shouldn't be horribly broken. Where do you keep this code while you are getting it ready. But I think there are some important rules for when working in branches: * Merge/rebase master into your branch often. * As soon as you have some parts of your code ready, get those commits merged in to master asap. A git rebase -i is useful to move the commits that are ready to earlier in the commit graph so that these can be merge without bringing in the commits that aren't ready yet. * Do pay attention to what the other devs are doing. If two devs are working on the same bit of code, then they should maybe merging each others branches.

[@paul-bouzakis]

Where did he say it would be horribly broken. This is who’s point. Master is stable, is up to date.

[@PhilipJReed-db3zc]

He didn't state the assumption here, but I think Dave lives in Blue Sky World where there's good unit test coverage and you can see in 10 seconds or less whether you've broken something in main, so you fix it before pushing back in. Lots of high functioning teams operate that way, but it's not always easy to end up on such a team without orienting your career specifically around that goal. In any event you can introduce change incrementally, and I'd say adequate (and fast!) test coverage is prerequisite to even a hint of "pure" CI as described here.

[@VictorMartinez-zf6dt]

Then you’re not doing continuous integration.

[@coderlifer4870]

@@VictorMartinez-zf6dt , CI is just a means to an end. The objective is to have a quality code in the main branch. You can thoroughly test the main branch, which you should but you also need to do due diligence in making sure you don't continuously pollute it.

[@gpzim981]

WTF. why this channel doesn't have 1 million subscribers yet?

[@j3r3mybr00ks]

I like the focus on true CI/CD workflow here as the ultimate goal. Personally I do like and currently use feature branches day to day. But we also try to keep our changes small and commits frequent so pull requests are small and they do go into prod on merge to main. Maybe this is a middle ground? If a team could ensure pair or group programming most of the time, then committing frequently to main would work nicely, but in reality that can't be maintained just because even the most committed pair programmer wants their own space sometimes. Anyway, my key takeaway from this is that implementing proper CI/CD is the key and how ever that is achieved then happy days! Nice video Dave I'll be sharing at work 👍

[@patrickroza3073]

I try to minimize branching, especially long lived branches. I also think a good starting point is to have only few branches at any given point in time. To facilitate this; do pair programming or otherwise divide the work, to work in parallel.

[@timseguine2]

First off I agree with a lot of what you are saying. Certain problems (on an otherwise amenable project) tend to require workshopping before release though, and I feel that in cases like that feature branches and a bit more integration work can be worth the headache in order to get things right. Also, maybe I just don't know how, but I can't figure out a great way to do mandatory code review in a "commit anything that is tested as often as possible" codebase without basically mandating pair programming. That wouldn't necessarily be the worst thing ever. I like pair programming, but I also enjoy programming alone, and I think they are both beneficial. In the projects I have worked on we tended to stay somewhere in the middle and the communication in the daily standups generally kept merge conflicts to a minimum, even when some branches lasted for a week or two.

[@paulprescod6150]

Yes, he didn't mention code reviews at all and I infer that they don't have mandatory code reviews.

[@mrspecs4430]

Yes, and on GitHub for example code reviews are intended to be done via pull requests

[@timseguine2]

​@@paulprescod6150I am not so quick to jump to that conclusion, but I am sceptical of that aspect nonetheless. The biggest thing is in my experience developers are more willing to make improvements to code that isn't in the mainline branch yet than something they already got merged. Analogous to how most developers won't write tests for code that "already works".

[@tedfitzpatrickyt]

a fascinating and reasoned argument.

[@coder8515]

Yes, this is definitely very interesting. I think that your release/deployment strategies could be integrated also with a more traditional use of VCS. Thank you for sharing.

[@Gr8thxAlot]

This is really great and makes things very understandable, thank you. This seems like an optimal strategy. But, isn't frequent merging to main going to generate a pile of merge/pull requests? Who typically reviews and approves these, and how do they manage this volume?

[@vitormanfredini1205]

From my understanding, you'd commit directly (without making any pull requests). Ideally, the code you're commiting and the existing code base are covered by tests verifying all behaviours, so it gets a lot harder to introduce bugs without tripping the alarms.

[@lucas_badico]

I really love this idea. But, we have developed an alternative way. Can I make a response video and than we talk about the trade offs?

[@stephanbranczyk8306]

Why ask for permission? Just do it.

[@ylazerson]

Very insightful!

[@imanamani3924]

Wow, your video is amazing! I wish there was a way for me to give you a thousand likes for it. Keep up the fantastic work!

[@FTropper]

How do you implement code reviews if you commit every 15 minutes?!

[@cheetah100]

Compulsory review on commit.

[@RolandvanderHeijden]

This may work in a "code-only" feature, but in my world, many features require data changes in persistent storage like databases. I'd rather hold on to a feature branch for an extended period of time and (more than) daily pull in changes from main. This way I'm the one solving my own merge conflicts and push to main would be easy.

[@diegosasw]

Fantastic explanation!

[@ContinuousDelivery]

Thanks 😎

[@laughingvampire7555]

I've been professionally coding since 2008 and the best way to organize code I have seen was the one I thought was the worst. It was bank using IIS as the web server, and the site of the bank looked like a single site, it wasn't, it was hundreds of tiny little sites, each of them with its own repository, the manager assigning the tasks was the one controlling who was using each of the modules or tiny sites. And we needed to make a change, we had ownership of that module for as long as it took to make the change.

[@hearnjohn]

If you are not using pull requests or mobbing with everyone on the team then what is to stop unreviewed code is getting into the mainline in an uncontrolled way? I know XP is explicit in saying that (at least) pairing is necessary for this to work (partially). What if even pairing is not an option for whatever reason?

[@OggerFN]

Branching shouldn't be responsible for making sure code is reviewed before it's deployed to production. Use jira or whatever to track status of commits and only deploy the application to production when tasks have been code and feature reviewed. CI is supposed to encourage collaboration as everyone is working on the same codebase.

[@yenenehmulatu5707]

What would happen after you have merged your small changes to master and part of your code is in product but not yet release. You get told the feature is no longer needed or you need to pause work indefinitely. How do you make sure your partly finished function doesn’t pollute the code base.

[@ContinuousDelivery]

Well, I’d question the sanity of the product owner who thought this was the most important thing to do a couple of days ago, and now wants to pull it. Then we could either leave the changes there, if we thought that we’d come back to it, or more likely delete them.

[@IslandHermit]

@@ContinuousDelivery So it sounds like you're not a fan of rapid prototyping, where you iterate frequently with user input to determine the final shape of the product.

[@jasonzuvela]

Maybe they cancelled the contract or the client folded, or funding was lost. Your initial response shows the tunnel vision of a dogmatic approach.

[@ContinuousDelivery]

@@jasonzuvela I don't think I am dogmatic, I have tried both approaches, so have an opinion on which works better. Sure, rarely, there are changes that completely invalidate the assumptions that your team has been working to. That is always going to be disruptive. If that happens after a CI approach vs after a FB approach I don't see that either approach helps more than the other to untangle the problem. You are, presumably, going to have to extract code that may or may not have been built on top of. This is not as trivial as merely reverting a branch. So I don't think that CI makes this worse.

[@coderlifer4870]

I've seen developers committing directly to the main branch and never doing a feature or a test branch. The changes are fine and safe. They don't break anything. And then I see the changes reverting what they committed a week ago with a comment saying, "I've made a mistake, this design turned out to be a bad idea or I found a better way to do it." The next thing you know, so much crappy code are already in there that are non-functional. They're harmless but dead code and it just pollutes the code base.

[@ggir9979]

Since we switched to a model.broadly similar to this, we almost never have merge issues any more. We do not merge branches every 15 min, compared to merging once a day I do not see the added value. Developers usually do not work on the same piece of code on a single day, and having to synchronize your repo with the origin every 15 min seems like an overkill. But as said before, with a more manageable timetable, it's indeed a very good practice. But every video I see on the channel I can recognize myself in most if not all the advices given, good to see someone with some real experience take on youbute!

[@Rekettyelovag]

At my previous job we used SVN with a trunk based approach, and we constantly had red test problems or even bugs/crashes that weren't even covered. This resulted in a stop that blocked 200+ people. Later we switched to Git, revamped the package and release system, and we were able to use feature branches as well. This solved many of the issues during development as we had coverage on our branches. OC we frequently synched with master, and we merged back the feature branch as soon as it was done which happened within a sprint. Since we had good code separation among modules the merges were fairly easy. The hardest part was figuring this whole thing out and creating the custom package manager plugins and Jenkins scripts, etc. But because of the previous experiences nobody wanted to go back to trunk based development although we could have solved the problem. With that said I think I understand your point, but we always have to evaluate our situation. Currently, I work on an enormous 15-year-old monolith software with a lot of legacy and new parts on top of it with more than 1000 devs. A code change in one place can break a lot of stuff at other parts and it's very hard to identify what went wrong. Not to mention that we need to wait until the next day to get test results. In this situation I wouldn't concentrate on trunk based development. At least not at this state. :D

[@ContinuousDelivery]

There are certainly times when adding FBs can reduce immediate pain, but I don't think they make much sense as a long-term fix, for the reasons in the video. I have seen the opposite too. I saw a company that did what you did, divide up a large code-base, each team began working on FBs, but they had lots of problems because of unpleasant dependencies in their code-base. So they made the branches last longer, to ease the pain. It did ease the pain, locally, within each team. But when I saw them, they had not been able to build their system as a whole for over 18 months. The need to integrate is real, the longer we defer it, the more difficult it will become.

[@Rekettyelovag]

@@ContinuousDelivery I completely agree. If we hadn't been able to merge back frequently we would have had the same problem.

[@detlevspitzbaard1653]

Although I'm fairly convinced that the way we develop our application (News Site Backend) is fairly optimal for what we do and how we do it, I watched your video with an open mind and tried to see the benefits of it. However, I couldn't find a way to make this work in our environment and being beneficial to stability, increased efficiency during development and deploying realeases. There was always a tradeoff at some point. So my argument is that concepts which were developed maybe one or two decades ago are still valid concepts but with different constraining parametres. Your argument was that a code should be always in a deployable state - which of course is always given for any branch which is deployed on dev, stage, release or however you may call it. On the other hand it doesn't make sense to commit unfinished code with unfinished unit-tests to a dark release or feature switch - even in the dev-stage - when it is simply not finished because it will inevitably break things. Just for the sake of commiting code every 15mins (you don't open a bridge to the public if its only halfway built). So my argument is, that it is probably a good idea to work in that way but adjust the constraints to the environment they apply to. Maybe commit the code not every 15mins but only every 1-2days when a feature is in a state where it actually could be integrated without breaking things. Another parameter (might) be the actual size of the codebase in relation the changed lines in every commit and the number of devs working on it simultaneously. Either way...good content and nowadays it became equally important to actually know how to organize your code compared to know how to write it.

[@MahmoudAbduljawad]

So, I still am pro-branching, but I minimise it to a semi-daily merges that guarantees the work is always very close to the truth. Having unfinished work in production doesn't seem to work for me, as it generates bugs more than good results for the devops process.

[@redtela]

Exactly! I mean, just take the argument that branching is "hiding" implementations and therefore cannot be continuous integration... equally, Dark Launching (or Blue/Green) hides the implementation from being in the Production system. The whole "don't branch" thing (and then going on to say that we should minimise branches) just pushes the problem(s) elsewhere in the development pipeline, just so we can continue to use the term "continuous integration." I'm happy to just have "semi-continuous" integration, I don't see why me or my team should be under any pressure to continually put things into Production.

[@ped7g]

@@redtela there's a mild difference that in dark launch the other developers and users don't run your changes in main line, but already may be executing your unit tests along the stuff they work on, so if they modify something your new changes will plan to use, it may break earlier, than brach-way. That said, I quite agree with your sentiment that branching and dark launch are quite similar, probably depends a lot also on the team, their overall habits, and the SW itself, what it does and how and how the work is split between developers. I can easily imagine situations where is no practical difference between the two whatsoever.

[@redtela]

@@ped7g that "difference" (or lack thereof) simply tells me that the advice of "don't branch" is just a synonym of "make it someone else's problem." At some point, we all must take some form of acceptable risk. We follow gitflow (at my insistence), and so, we use feature branches. We also use unit tests, pull requests, integration tests, regression tests, manual QA and UAT. We use Pair Programming, BDD, TDD, and other methods. We have well over 1000 components in the system, each probably has 17 different "versions of the truth" and a hotfix can leave a developer laptop and arrive in Production within 30mins. Could we do it better? Sure. Are we "statistically producing worse code than if we didn't branch" - I doubt it. Could we release faster by not branching? Possibly, possibly not. Do we "hide" information from other developers? No, because a pushed commit can be pulled into any branch, at any time.

[@redhotbits]

@@redtela you are doing it wrong

[@OthmanAlikhan]

Thanks for the video =)

[@aaronmacandili7149]

Commiting unfinished work to production is kinda scary. I also feel that the “behavior hiding” can introduce unwanted bugs and will need more efforts of removing these hidden behaviors in case the requirement didn’t push through or the requirements have been changed totally in the course of development (dev will have to look for each and every commit just to remove these tiny pieces of hidden behaviors). Also, does this approach considers the need for QA to test the changes? Does this approach requires QA to test the production version whenever there is a new change?

[@chronic_adventure]

Do you have any videos that explain where QA processes such as regression and feature testing would fit within CI and CD?

[@ContinuousDelivery]

Not specifically, but I should do something on how roles change for CD. I recommend that you aim eliminate dependence on manual regression testing. That doesn't mean that QA has no place, but the role shifts to exploratory testing of new changes, not going back and checking that old stuff still works. This works most effectively when QA work closely as part of the dev team, evaluating changes while they are bening built, rather than waiting until they are finished. This is easier than it sounds if the team is following the CD principle of "work so that your SW is always releasable". I will think about doing a video that cover some of this in more depth, thanks for the suggestion.

[@richard_luke]

This channel deserves more subscriptions and more views in the videos

[@ContinuousDelivery]

Thanks

[@johnronaldraviz4052]

Great advice. :)

[@pansrn]

Very thought-provoking. Thank you.

[@georged8644]

This was a very fascinating and informative video! I learned a lot from it but I have a question. How do you handle speculative developments where you are trying out ideas to see how feasible they are and how compatible they are with the rest of the system without committing them to the central trunk? I don't see how to do this without a branch. This is because you don't want your speculative work to interfere in any way with the production system until you are satisfied with it because you may end up discarding it entirely if it proves to be a dead end.

[@andrealaforgia]

Using branching for that specific activity is totally fine and probably the only usage that is acceptable. Branching is not necessarily evil per se. It’s a Continuous Integration anti-pattern, though, so to be avoided when you want to integrate often with other developers in order to deliver changes as fast as possible and avoid merge conflicts (typical of long-lived feature branches). However, bear in mind that the longer your speculative branch staying unaligned with master, the more it will diverge with the rest of the development hence the more likely it is that your speculative work becomes less and less relevant.

[@ContinuousDelivery]

For real speculative things, treat them as what they are, experiments, focussed on learning. Aim to always throw away any code you write in these experiments, the value is the learning. This means that you can be more radical in your speculations and aren’t constrained by having to make your experimental code “production quality”. This approach tends to focus you more on what it is you really want to learn too. Try and clearly state what you’d like to find out before you begin.

[@georged8644]

@@ContinuousDelivery Thank you both for clarifying things for me! This is kind of what I expected the answer might be but I wasn't sure. I really liked the way the method presented in the video prevents huge divergences in the production code. I've been through a few nightmarish project merges and that was back in the SCCS days. I can see where GIT could lead to even worse ones if you don't impose discipline.

[@StefanKGr]

Hey Dave! Love your channel! This is great for on-premise teams. If the teams are distributed all over the world with a single code base, not only that pair programming isn't an option, its that some PRs will have to wait a day to get merged due to time-zone difference. Any advice there?

[@andishawjfac]

Why have PR's in a continous intergration workflow? The whole point of CI is to get rid of PR's so you don't have to wait for somebody to approve. Build a good integration testing environment and let everybody commit to that, create a secure and quick feedback loop for devs.

[@JadeSync]

Would you suggest something like the "Live Share" extension in VSCode? Where entire team can work on same code and you can see what everyone is doing in real-time.

[@chernojallow6666]

I think u making the assumption that there is one way of implementing CICD and that there is only one place where all feature branches converges which is PROD. I think with a good branching strategy (feature/(123) -> develop -> release/(1234) -> production ) with each branch or branch prefix mapping to a environment, life of that dev team would be much better. This allows for less conflict, two convergence points (merge) for all features (in develop and release) before prod and 3 test env (where 2 env: develop and release have the integrated changes) . Pretty sure there are other ways but a development team of atleast 15 members-without branching it would be a nightmare.

[@jack6539]

Having implemented ci and cd last century, I can tell you this is a gross oversimplification. Branches are necessary when the need to isolate changes justifies the overhead for having the branch. There are many reasons to branch ( yes, even by feature) and there are many reasons to not branch. The problem lies with knowing when to apply the method and when not to, and unfortunately many developers have such a low understanding of branchinh that they even conflate directories with branches. The factors involved with these decisions can involve everything from the sw architecture, the competance of tge team, the competance of the vc tools merge capabilities for tye files concerned, and yes, even the release management process. Over the decades I have seen so many people try to oversimplify this and have had to clean up the mess afterwards. At the very least your video should provide caveats like " if the merge tool for the code files is not reliably automatable for >95% of merges ( and yes there are tools that exceed this, and git ootb is not one of them) and your development is project based, not product based with product variants and/or your system is not a safety critical system and/or your dev teams are poorly trained on the effective use of branches etc etc etc. To give a real world example of the impact of branch avoidance, I jave seen an entire product line have to be redeveloped as each product variant was a copy pasted directory of the source code (12 product variants). Despite having the same people maintaining the codebases over a number of years, each copy of the code was so completely dissimilar that a core change across all of them worked out to be cheaper to redevelop the entire product line from scratch ( a $12m decision just taking into account the costs for re accreditation of the variants - let alone the cost of redevelopment and opportunity cost to the business.

[@ContinuousDelivery]

Naturally, I don't think that it is a "gross over simplification". It is certainly a simplification, because it is a 15 minute video! The approach that I have described is, and has been used, to build large complex systems all over the world - I have worked on several myself. Occasionally we found the need to branch, and it felt like a failure when we did. I don't think that there is much about software development that is easy, but I also think that, as an industry, we are exceptionally good at over-complicating things. Branches being one of those times where we are prone to mis-using them. Any branch is "running with scissors", it exposes you to risks with BIG consequences if you get it wrong (as you point out). Of course, this doesn't mean that every time you create a branch you will fail. It means that your chances of failure are increased. CI has a different set of compromises and fails more gently IMO. My intent with this video was to point out the costs, and benefits, of both, but naturally my advice to any team is to "prefer CI to branching" and "always branch reluctantly and with care". (I probably should have said that in the video - sic!). I agree with a lot of what you say when you describe naive approaches to branching, cloning different versions of products is a horribly common practice - and terrible solution.

[@jack6539]

@@ContinuousDelivery an excellent clarification. It is vitally important that branching strategies are applied by skilled people who know how to balance the various factors within the constraints provided. Unfortunately, there simply aren aren't enough people who do. CI and CD are good methods when applied appropriately, however, I continue to be concerned about how CI and CD are being pushed as being the default or only way. A recent example of the consequences is actually the recent solarwinds hack. It looks like they had a fully automated CD system with a view to releasing quickly, which on the surface seems like a good idea. That is, until hackers put a backdoor in the IAM modules and let tge CD systems punch it out to prod, and then to 16k of the worlds most crucial orgsnisations. Having a fast means of punching through to production is a good thing to have in general, however for some systems, and for some areas of systems, a more rigorous control process is necessary. The approach of fail fast, fail early, fix quickly is simply not viable for some systems where the failures are safety critical. Also, with monstrously large dev endeavours like I have been involved in ( 1600+ developers concurrently, for example), effective and consistent branching strategies become essential for the efffecttive coordination of change implementation. The result of not branching in these situations leads to high numbers of regressions and an explosion of inconsistent environments.

[@tube4Thabor]

​@@ContinuousDelivery I can't take seriously someone who considers themselves a failure for using a branch, or compares it to running with scissors. That is the sort of holy war language that doesn't belong in a real discussion. Branches work best when they are used like transactions in a database. They should be small and short lived, but they should also be consistent and complete changes (not complete features).

[@RadioWhisperer]

@@tube4Thabor As an embedded software guy I couldn't agree more. There are so many different kinds of software development, creating one nail to solve this problem turns all projects into hammers. And sometimes forcing the project to be a hammer when it's not is way more expensive than choosing the right process in the first place. It's nice to have this tool be available, it's not a solution for all projects.

[@chrisfogelklou7136]

@@tube4Thabor "Holy War" Yes! It's surprising because most of the developers that get religious about any "development philosophy" including functional, OOP, branch strategy, code coverage, TDD, BDD, Scrum vs Kanban, etc, are *junior*. It should be about applying the best practices in the right amount, but not *all or nothing*.

[@bijukthomas1981]

An eye opener for me 🙏

[@ContinuousDelivery]

Thanks 😎

[@precumming]

Don’t change from branches, he’s full of shit.

[@runonce]

Sounds too ideal to me. Never worked on a team that is even near to have what it takes to do true CI. Thanks for the vid.

[@thekodomo]

I know nothing of programming, but this is just so relaxing to listen to and so interesting :)

[@ContinuousDelivery]

Glad you enjoy it!

[@muteza]

This is a big problem. Unsafe code ending up in production - waiting to get exploited. I think the demand for instant relief has gone to far.

[@ContinuousDelivery]

Nope, this is how some of the biggest, and some of the most reliable, systems have been built.

[@Stylpe]

@muteza You write that as if you think it replaces all other common practices like code review and static analysis and so many more tools

[@rhakka]

@@Stylpe Well, it changes code review to pair programming, according to the video and comments. If the best developers/teams merge many times per day, the code reviews have to go away. Even if you have the automated testing to cover your code confidently, you're still then missing out on reviews of a feature as a whole, and instead getting reviews or pairing on small changes without the context of the entire feature. Oddly, even Martin Fowler himself says that for actual features, not just small changes or bug fixes, the best practice is short-lived feature branches.

[@Stylpe]

@@rhakka I'd argue those are issues with better solutions than feature branches, and which this style of CI will force you to learn, for the better. Better separation of concerns, dark/dormant features and feature flags, realising that you don't have to, nay, shouldn't, be producing new code 100% of the time but also review and explore other recent changes both for your own learning and as a kind of code review, having the whole team paying attention during sprint demos, understanding the difference between deploying and shipping.

[@RommelManalo]

This guy is making things complicated!

[@ContinuousDelivery]

Nope, things are already complicated, but not everyone notices that they are😏

[@Tall-Cool-Drink]

in a perfect world "Continuous Integration" is possible, but our world is anything but perfect. :-)

[@ContinuousDelivery]

The nice thing about software, is that it is easier to change than the rest of the world, maybe not perfect, but certainly "good enough for CI" 😁

[@esotericbeep5923]

Ok so I'm 8 minutes in and it seems you're suggesting to minimize the size of the branches, not to be rid of them completely.

[@karsh001]

I like branching, but then I work in a field where features are truly independent on each other. Usually the branches in my projects live 2-5 days.

[@ContinuousDelivery]

You can like it, but the data says that you are likely to be producing lower-quality software more slowly.

[@karsh001]

@@ContinuousDelivery That is dependent on what type of software is produced and how it is structured. As long as the features are truly independent it is not problem to let the branches live a bit longer.

[@nicat6153]

​@@karsh001 Isn't branching used to prevent getting in each other's way, where one might change a code breaking someone else's code? What is branching used for in your case, where things are completely independent? Just curious.

[@karsh001]

@@nicat6153 Its a good reason to use branching, but not the only one. Controlled delivery to production and simplified dev testing is the reason I use it in my projects. Due to the way the problem space work it simplifies some stake holder management and approval processes and we can quite easily drop features if they are rejected somewhere along the line. Dropping features mid implementation is very common, and the ability to just drop the branch becomes a value in itself. Once implemented, the features live a long time, usually many years with none or very small changes. Its all dependent on how the solution is structured and I wouldn't recommend long branches for a more standard SW project where you risk breaking each others code. For my projects, though it works great.

[@jonobaker]

I have to agree. We manage a legacy system that is very broad in scope. This means Multiple features can be worked on at the same time without overlap, so feature branching works well, and turns out to be the simplest system to manage in this case. Having a sql server database at its core combined with being a legacy system full of bad design means feature flags tend to be difficult to implement confidently. And we’re rarely working on new elements that would allow dark features. We also really battled against a dev/master branch method, there was just so much drift between the two, after a month or so we had no confidence that dev was anything like what was in master. So I’m totally on board with the frequent merging of continuous integration, it’s just turned out to be a easier method to use feature branches for us, as conflicts are rare

[@SaiyanJin85]

Branch in behavior, not in code. That’s a great one. Eye opener.

[@davidbroadhurst4031]

I would be concerned about a project that required merging several times a day because the code I was integrating into was being changed that frequently. I suspect the need to constantly integrate is s symptom of much bigger problems. I've not found a big issue with either approach but I also try to manage change chaos.

[@test1594]

I think he is totally wrong. A branch is only a pointer to a commit, nothing more. Feature branches is a developing method for mainly creating a clean commit history. If you have conflicts in your team because too many developers are on the same code it's better to try to find a different WoW e.g. swarm on that task or don't work on the same code at the same time. So don't stop using feature branches it's the only thing that keeps the commit history clean which is really important thing for finding and understating how bugs were created

[@habiks]

Yeah... I love it when I get coworkers changes right away and it breaks the code for whole team....

[@habiks]

Also it's not like: #1 teams don't talk what the end plan is, #2 we can't look at other branches, #3 create our feature branch on top of another unmerged feature branch.. #4 rebase our work on top of any current branch.

[@ContinuousDelivery]

Write some tests and gate the commits!

[@precumming]

Exactly, avoiding branches just causes a massive amount of problems. The only “problem” with branches is merge conflicts but they arise from a lack of communication and planning and it highlights what needs fixing for it to not happen again, handling the merge conflict itself isn’t even that bad and it’s less inconvenient than everything breaking suddenly because someone pushed in a rush and the bug wasn’t contained in their branch.

[@nero1375]

@@precumming But if the team writes tests, this "pushed bug"will not be committed until fixed, right? I think that will happen if the project does not have the test coverage

[@levileal4500]

@@nero1375 Nope. You changed the Client history functionality and updated the tests. It's all working for you. I'm creating functionalities A, B and C that uses that history you've changed. Your code works and is tested, the moment I download it it can break my code, because I'm not expecting to receive your changes.

[@FlaviusAspra]

Dave, I think the real discussion should be: code review vs continuous deployment. It feels to me that this is where the friction comes from, not the branches themselves, which are used as a means for code review. Maybe it's worth a video?

[@oneheadead]

we tried CI/CD with a small team of relatively green developers and it worked surprisingly well for game development, BUT this required a lot more management than when we did feature branches, and pretty much continuous communication between the devs. I can't imagine how a workflow like this would work with a team of over 4/5 coders or without a team lead that's keeping a lookout for issues 24/7.

[@ContinuousDelivery]

I can only say that it does. I have done this many times with teams of 30 - 50 people, and often with teams in the hundreds. Google does a version of this with 25,000 people working in a single repo.

[@crushingbelial]

I am reading this as a team of people were in learning mode all at the same time. This is expected.

[@j.j.9538]

I feel like once a competent person goes into devops or management, they start trying to make the development process more efficient. The problem is that software development is a creative labor. is complex and messy at times. There's a lot involved and there's no way to implement source code control without affecting the development process. Eventually, there will be so many constraints around developers that it will become impossible for them to do their job. I understand where all of this management/organizational techniques come from. They don't want to depend on a few talented individuals, so they want to hire a bunch of code monkeys and enhance their abilities. Basically, streamline software development. But the best you can do is to try to get the really good developers and hope that they are smart and have good communication skills. If you hire of bad developers, no amount of organizational techniques is going to solve your problem.

[@minastaros]

The general idea to be only short time away from the master, and to integrate often and early is surely a good advice However, I disagree on one aspect: testing new code should _first_ be done on the feature branch, not on the production master (where it might break something). But, of course, you need to make sure that the feature branch is as close as possible to the master, thus, first merge master into the feature branch, test, and _then_ integrate to master.

[@crushingbelial]

Straight trunk to production without any automated testing is a recipe for disaster. Testing new code can and should be handled by feature flags with selected enablements allowing for environment testing to occur. You effectively solidify the trunk by using toggles as such to prepare integration of deltas after your test scenarios have run.

[@jeangautier1224]

I have a another question, what's youre opinion on using this technique in a mono repo with more than 50 dev working together ?

[@szilarddoro]

Thanks for these thoughts, it's really good to see a different point of view on how to approach delivering features. There is a question though that came to my mind. How would you revert changes of a single feature? Wouldn't it be a pain in the neck if its changes were really fragmented into a lot of small commits?

[@ContinuousDelivery]

Not really, it’s usually pretty simple. It’s not really different to any other revert. You need to confirm that everything works afterwards as you would any other time.

[@ddtalks2821]

I am curious how this helps/improves/makes worse QA efforts. While I can see it helps reduce the "merge conflict" issuses, are you suggesting that deploying each time a commit to the master branch should be put into QA environment for testing ? (assuming the code commit doesn't "Break" anything) also, assuming testing is being done on the code commit to ensure nothing is broken, how much testing is that ? Unit testing is such a small part of testing (only makes sure the snippet of code doesn't have errors and "passes" whatever tests have been created) Usually there is not integration testing done or more comprehensive testing (regression). So this makes me think that there is still a large effort (once the complete coding is finalized and "ready for Production") that regression/integration and such testing is performed - and if failures found then it is not released into production. Isn't part of CI/CD supposed to be the "code is in a releasable" state? So to me that means a full regression could be run (should be run) every time a commit happens to "ensure" that the code is "releasable". That takes a lot of time to deploy to QA environments and run tests. If you are committing every 15 min, there is NO way a regression testing could be completed in that time.