Real Programmers Commit To Master - Jakob Ehn

Continuous delivery and DevOps is all about delivering value faster to our end users while keeping the quality at a high level and reducing the downtime to a minimum. Traditional development strategies such as feature and release branching allows us to work and stabilize in isolation, but by doing so we are no longer integrating continuously which in the end will slow us down. So, how can we whip new code as often as possible (even every commit) without breaking stuff in production?
In this session you will learn about techniques such as Feature Flags, A/B testing, Canary releases and Dark Launches and how we can implement this using Azure DevOps, Microsoft Azure and Application Insighs to be able to ship new functionality into production as fast as possible, and learn about how these features are being used so that we can learn and improve along the way.

[@DefenderPuma]

I really wish I'd seen this earlier. We are still using feature branches and it's not working for us. I immediately shared this with my team and we are already in the process of adapting our processes.

[@skipodap1]

Great video and presentation.

[@CuriousCyclist]

Thanks for uploading all the good videos. You deserve more subscribers.

[@LeviNotik]

This is a fantastic talk.

[@jankinsics]

Love this talk. So far the best I have watched here. I used to use git flow(the very first version). Caused lots of problems. One of the big problem is the long lived branches. It’s a nightmare. Finally, we ditched that. Our way is similar to trunk based development model with minor differences, it’s been working great.

[@saikrishnasiliveru4350]

Thanks Jacob

[@davidknoernschild6053]

This is the best case I have ever seen made for trunk-based development and CI. Great talk.

[@danielcooke3243]

Feature toggling is something i've never considered doing - wow, it really does solve the problem of long lived feature branches

[@jamiemchugh3279]

From my experience feature toggling adds complication and overhead. If you don’t schedule clean up of feature toggles once a feature has settled, you end up with repeated forking of your business logic. If there’s overlap in different parts of the system, this can have a multiplier effect that violates Clean Code principles. Furthermore, clean up is a challenge as you’ve weakened your tests by introducing ambiguity of truth, and you don’t know when a feature will settle, so those cleaning up in the future may not have the context to effectively remove redundant code. Maybe I’ve done it wrong though… happy to hear arguments for it…

[@TheEVEInspiration]

Feature toggling is a terrible idea for anything that has some real complexity in it. There is so much that can and will go wrong with that, its a massive no no! And for the love of ***, stop thinking about code, think about data(-models), states and state changes/information flows. Any data-structure that changes or different ways in which a state is kept or how different parts of code interact, cannot be thought of as just code that one can toggle on or off. It becomes a whole different thing and having half finished stuff in the codebase will be confusing AF and cause many errors in one way or another. The examples of big changes that require months to implement, are most likely those kind of changes, so plan them right. It will not be brand new functionality that is just glued on like a new data-entry screen of some sort (kid stuff). It really becomes maintaining N different variations of the same product in the same codebase (includes database). How is one to deal with performance issues when some new solution gets introduced? How much of the unfinished code is in the way or also needs untestable changes? It is hilariously stupid. In a way it is the same as only using global variables.

[@ForgottenKnight1]

Feature toggling is NOT a solution for long lived branches - YIKES ! STOP ! :)))

[@aayushiagarwal8063]

When we are committing in main trunk, is it necessary to create tags for every commit. When we are going for a release, shall we cut a branch out of it or just tag??

[@chrisalexthomas]

ok, 2 minutes in I can already say that the first mistake was that if you need to have a long lived branch like this, you need to constantly sync master->branch and take all the low hanging changes and in addition to that, possibly refactor into the long lived branch things from the master branch that are not 100% compatible. You can't let them diverge for months and expect good results at the end

[@samuelmothwa1720]

My thought exactly, why don't you keep up with master.

[@tiagodagostini]

Pulling from master is something that MUST be done daily .

[@andrealaforgia7699]

​@@tiagodagostini Not if Continuous Integration is what you aim for. In the case of CI, you must integrate into master multiple times a day. It's a much simpler and effective model than feature branching.

[@tiagodagostini]

@@andrealaforgia7699 That is the point where I completely disagree. COntinuous integration doe snot mena continuous deployment. The place where I worked with the best quality results (Huge software, bigger than photoshop and we kept under 3 bugs detected by clients per YEAR) we used feature branches, and every branch had to pull from master every day. When a smart ass came with this new model the company FAILED hard and the number of bugs and problems on clients skyrocketed to over 200 per year.

[@andrealaforgia7699]

@@tiagodagostini Not sure what you are disagreeing with, exactly. The definition of Continuous Integration is "the practice of merging all developers' working copies to a shared mainline several times a day", so exactly what I said. It is NOT merging trunk into the branches multiple times a day. Also because, if two developers branch off trunk and work independently on their branches for days, merging trunk into each branch daily has absolutely no effect. No matter how successful the company where you worked was, feature branching is a Continuous Integration antipattern and I have witnessed many situations where feature branching was creating problems. If CI is what you aim for - and you should aim for - then feature branching is the WRONG way of working. Equally, the fact that the person you call a "smart ass" was unable to implement proper CI does not mean anything. Continuous Integration is not at all a "new model". Officially, Grady Booch talked about it in 1991, but there are fundamentally equivalent practices being implemented at least 30 years before then. I suggest reading "Accelerate" (Kim, Humble, Forsgren), "Modern Software Engineering" (Farley) and the "State of DevOps" reports.

[@ForgottenKnight1]

22:30 - It should be obvious - Darkly is super easy to implement in code and super easy to use, not to mention the other stuff.

[@RikiPoon]

9:08 by branching out here, does it introduce one extra commit (the blue one right before the branch) to production? for i assume after creating R1.0 the production just need 2 yellow bug fixes, this seems to be a problematic setup to me

[@danielcooke3243]

Yeah - what he says he is not possible. If you tried to merge that feature branch back into master, you would have to merge it with the 2 missing commits, therefore introducing new features to production

[@ok-it4li]

yes, its a vague diagram.

[@tarankaranth8782]

can we do without release branch with feature toggle?

[@VictorMartinez-zf6dt]

yes

[@JamesJansson]

Trunk based dev is always premised on a lot of good tooling being deployed, a minimum level of employee competence which is quite high, a strong belief in honesty of the employees that they aren't malicious actors. This is why it works well at big companies.

[@JerreMuesli]

Git workflow, when using short-lived branches and rebases, isn't an issue. Trunk-based development is handy at the start of a project when you both need to e.g. extend the domain model or you both need to work on the same modules (records/classes) I don't like feature toggles as it instantly creates technical debt in both code and unit/integration tests and also it increases complexity. I don't say I hate trunk-based but saying that it is a silver bullet when using git, and git workflow is bad is just ridiculous

[@ono48]

So, you actually need to work on a branch forked from main trunk and merge back to it via pull requests. We just want to avoid long lived branches, call it a feature branch or a topic branch, it’s just semantics. Working directly on main trunk implies no Code review / pull request, which is a terrible practice. Main trunk needs to be protected with a branch policy

[@crushingbelial]

Pair program or use PRs. Yes don't break trunk always have it ready to go but use trunk for your lowest level environments for fast feedback then prep release for controlled environments

[@ramdenze]

The examples of feature branching are maybe not the full story. The most used way of doing feature branches right is called git flow. If you know how to use it you would not end up in situations like the ones illustrated. For example: You are creating a mobile app. Trunk / master has version 1.0 and you already work on 2.0 on develop branch. Now suddenly, something bad happens and you make a hotfix 1.0.1, master will be on 1.0.1 and develop branch will have 2.0 WITH fixes for 1.0.1. Works pretty well

[@krajekdev]

I upvoted, that is what I was doing and still do. I do, however, start to see the advantages of trunk-based development. In ideal world feature flags of course give you more flexibility than feature branches.

[@peterpopov2937]

I would recommend that you watch Dave's commentary on GitFlow and why it's a bad idea that goes against CD principles. Here's the video: kzbin.info/www/bejne/laiZhaqkgah5raM

[@xXxDiukexXx]

Git flow is considered deprecated

[@andrealaforgia7699]

GitFlow is a CI antipattern. Look at the videos Dave Farley has published about it.

[@andrecampbell7365]

I only know trunk based development.

[@user-gm4he1yo8l]

You said a benefit of Trunk based development is that it "encourages refactoring" but never explained how that could be so. From everything I've seen and experienced, smaller and quicker prs to the main branch do exactly the opposite. All in all, though, I found this talk very informative.

[@tiagodagostini]

I am so against the concept of integrating continuously if the objective is to release fast as hell. As a costumer I hate when a company does that to me, as a software development company I hate the concept of so much entropy in the system. As I developer I am amazed by the drugs people take to have the idea that ANY practice can be said as the CORRECT way to develop.

[@matthewgreen3107]

100% Code quality, tests and regular merging not overdoing the tickets and work is the key. We tried this with over 100 devs it fell apart very fast as people kept breaking master and holding others up and issues being spread across others unrelated like a virus as people start fixing things. Utter headache especially when people start adding feature flags and expect a database or API still to be the same.

[@tiagodagostini]

@@matthewgreen3107 One of the things that did not work for me is... you do nto knwo if what you are doing is what is really wanted or will be accepted. To know it you need a trough and detailed pass of the product department/owner and the result is.. that is the most anti agile thing ever. Notice , I am talking about software that is really developing something new that cannot be simply described as "make me a new form to insert new registries of this stuff". If the feature is "we need a new polygon split and subdivide interface that can combine the method X and Y " you are NOT going to do it correctly the first time, and therefore it should NOT be in master. As I said I would , as a costumer, drop ON THE SPOT that product if they start releasing stuff like that to me. No development branch should ever deviate too much from master, but you achieve that by everyone PULLING from master every day! I would FIRE any of my developers if they pushed to master a NEW IDEA without first presenting it in another branch to see if it is approved.

[@matthewgreen3107]

@@tiagodagostini spot on reply. You only have to pull from master whenever you feel you are deviating and need something. Making it a hard set rule is also not good as you may not want to integrate features from other devs and are simply demonstrating a concept feature. I have always promoted feature branches being properly tested and demonstrated to the clients as we go and when they feel they are happy they go into master. A proper feature should have tests and so when two features or more get merged into master the tests will pick up any merge issues and most of the team should have work planned to not step on each others work or they should be working on the same feature branching off it and pushing to it regular this is called quality sprint planning months ahead and understanding the product roadmap. We have NEVER had any issues and had code thrown out by clients after a sprint which would have been a real headache trying to take it out of a master merge as they went along. All this nonsense of feature flags as well introduces masses of potential for issues. Sure if it is a shopping cart go for it on a shops website but if it is a hospital system you want that rolling out with experimental stuff in it with flags for paths that people have been throwing into master?... crazy. Life and death do it right anything else is dangerous. If you want to design software for the local chip shop selling fries and it crashes sure go for main line. Also have you seen that they are now trying to advocate not doing code reviews as it takes too long and everyone should be able to review and fix as they go! wow crazy. It's like they don't want to do quality anymore and just want to be hippie devs playing with tech imho! Scary tbh that the standards are dropping in todays devs rushing to release constantly as if that is the answer to giving a client quality when all it does is introduce risk. As an aside I work on life / death systems often so perhaps this is why this scares the crap out of me also given the quality of the code some devs produce as well and the bugs they introduce even on features and having at times multiple code reviews and refixing.

[@matthewgreen3107]

@@tiagodagostini 100% would work in your team as you sound like you understand the real world.

[@tiagodagostini]

@@matthewgreen3107 That seems to be the standard.. peopel that support that are mainly WEB stuff. People that work on defense, energy or Helathcare would put fire on you if you said to commit on main/master.

I think all the problems you're exposing arise because of a bad architecture and a lack of strategy when planing the work in parallel rather than the version control flow.

[@tothestars3958]

Yeah I'm starting to feel like the issue the trunk-based crowd has is with their code, not with their git strategy. If your code is designed reasonably, there won't be these massive conflicts, OR these major integration issues.

[@m.x.]

Problem: young generations reinventing the wheel again and again.

[@gpzim981]

This approach as well have sooo many problems

[@berkes]

There are always tradeoffs and downsides. The question is, are the 'many problems' you see, worse than the benefits that trunk based development offers? Can you find solutions for them, or for most?

[@MasterTrollazo]

@@andrealaforgia5066 * Expensive all the system and automation around this for most of the projects do not worth it. * The logic of versions moved from git to feature toggle, you are maintaining all the possible combinations of all features toggle. If you have 5 features toggle. 2^5 versions = 32 (If I remember maths correctly) * Unused code in the main version, you need to clean and reduce your artificial incremented codebase. In the feature branch, you can discard completely the change or incorporate it, but is not going to generate unused code. * Many big companies with a lot of concurrent users are using it. However, not in all of their products. Also, we are still suffering a lot of bugs in their products.

[@casper0at0ease]

@@MasterTrollazo you use TBD for SaaS and therefore only have one version, which makes your whole argument moot

[@MasterTrollazo]

@@casper0at0ease if they behave different for the same circumstances for different users, are different versions.

[@lauyuhin]

Agree. Fixing bug in master could cause problem as well.

[@SM-ok3sz]

Why is it always assumed branches are always out of sync with the parent branch? Does your git pull not work? Sounds like a case of poor development practices.

[@emyrulz]

This! Also, it seems to me that there is this assumption that people are working on stuff that 100 other people are working on at the same time and that is being changed hourly somehow. This is nuts. Part of it stems form misunderstood practices where features are broken down way too much and too many developers work in parallel on them. What happened to one dev working on something from start to end, alone?

[@lauyuhin]

Total agree. If everyone git pull frequently in their feature branch, feature won't be out of sync.

[@andrealaforgia7699]

It's not continuous integration.

[@VictorMartinez-zf6dt]

Do you pull your colleague's changes from their branches as well?

[@bsuperbrain]

feature branching is great, you just need to rebase often

[@berkes]

Do you rebase all your colleagues' open feature branches "often"? How do you know what feature branches you must rebase on? Do all your colleagues push their feature branches often? And can they then still push them --forced?

[@TommasoLumare]

@@berkes I think @bsuperbrain was meaning that it would be better to *rebase your feature branch onto master often* to pull in all the bugfixes and the features that your colleagues bring into the master which is not a bad idea if you think carefully.

[@carabouzouklis]

Feature Flags are better

[@rc3043]

@@carabouzouklis I think rebasing often during a day and then merge your current work daily to the master branch from that feature branch is behind a feature toggle is a good middleground for this kind of work :)

[@m.x.]

Rebasing is not continuous integration.

[@Olga-pc3bm]

Also, feature toggles makes code unclear at all. It breaks a lot of common programming rules like don't create zombie code or zombie comments. What will happen if 3-5 developers left the company and their features are in the intermediate, non-working state? Nobody can understand what they were trying to do and what was the idea. Or the implementation way could be wrong. And all this code we have in master branch! Each reader of this branch should spend time on trying to understand fired developers' thoughts until he comes to the idea that it was a wrong implementation of smth.

[@lcppproductions7910]

4:42 "You haven't really tested this combination of features yet" umm... Pull from master incrementally, so you get the previously Merged feature, and then test it before you integrate? Is that not a thing?

[@redhotbits]

we used trunk based and our new boss forced us to use gitflow

[@tommcannally1895]

Bummer

[@howardh6857]

And what’s your opinion now

[@redhotbits]

@@howardh6857 i see many teams do not practice feature flags, some boses do not like that... thats an issue and prevents trunk based to be used

[@jankinsics]

We switched from gitflow to trunck based.

[@redhotbits]

@@howardh6857 trunk based ofc, even on atlasian page it states that trunk based is the best practice. But practices are evil, sometimes you need to unlearn the old best practices and learn new best practices.

[@matswessling6600]

feature branches simply periodically merge main back into it. problem solved.

[@VictorMartinez-zf6dt]

no, not really.

[@matswessling6600]

@@VictorMartinez-zf6dt yes, really.

[@ddanielsandberg]

@@matswessling6600 No, not really. The point is that when you have 30 developers, each working on their own branches for days (or worse, weeks) on end, what does it help rebasing from master? You still won't know if anyone else's changes conflicts with all other changes until they're merged into master.

[@jamiemchugh3279]

@@ddanielsandbergyou will once they request a PR, or further on, push to master, then those changes are pulled to the feature branch. Then conflicts can be resolved in the isolation of the feature branch, rather than on master? I remain unconvinced here that the problem trying to be solved is due to a combination of messy code / architecture and a lack of coordination/communication between different teams. That said, I never advocate for long lived branches. I just like the temporary isolation and PR mechanism.

[@agile2academy]

I guess Linus Torvalds is not a real programmer.

[@TheEVEInspiration]

Its always possible to do trunk based changes, even when using feature branched for other stuff. So that negates any argument in favor of CI, let alone CD. Really I hate these videos made by amateurs, trying to solve problems that already have working solutions. Just because they stumbled and messed up in obvious ways before using one method, does not mean another is somehow superior. I only see downsides to CI, let alone CD, not even customers want it (they hate constant change). It simply cannot work for most kinds of projects, every showcase is about an existing product undergoing insignificant changes. Not because they are broken up in small pieces, but because they really are that insignificant.

[@tiagodagostini]

I will give you a single example why real development teams do not commit to trunk as you say. ON the field I operate, it is REGULATORY BOUND that every single feature must be tested and must be available in a state that is the exact previous released version of the product PLUS the feature itself. If you MIX 2 features development release cycles, you lose your license and your company is SHUT DOWN . That is how healthcare operates in several countries. You CANNOT effectively have 2 teams comiting to the same branch because you violate these rules.

[@haraldhacker]

so what concept you follow? Thanks

[@xXxDiukexXx]

That law says nothing about branches, with feature toggles you are not releasing an active feature therefore is not available in that release, unless activated either remotely or in a future release

[@tiagodagostini]

@@xXxDiukexXx The thing is I must have specific versions with EACH new feature separate from all other new versions. That is unfeasible in a single branch and alarge team that is working on several teams at same time. That law does not care if the feature is active or not, but the feature existing in the artifact of software or not.

[@JonathanYee]

The principles of trunk base is the big take away. He did mention about release branch + cherrypicking. That would be best suit to your architecture.

[@VictorMartinez-zf6dt]

Two words, continuous integration.

[@vickmackey24]

"master"?! 😱🤯

[@FudgeYeahLinusLAN]

You must be an absolute joy to work with...

[@julianosouza8382]

real devs commit to master and deploy stuff in friday night N O T

[@chrisalexthomas]

You clearly have never worked a day in your life! 🤣🤣🤣 (just kidding)