When 10,000 Software Engineers Work on the Same Code

Try the Stacking workflow: gt.dev/codingwithlewis
How do software engineers at big tech companies like Google and Microsoft manage repositories with billions of lines of code? 🤯 See how thousands of developers collaborate and ship updates to hundreds of millions of users daily.
In this video, we dive into techniques like virtual file systems, stacking, trunk-based development, merge queues, continuous integration, and continuous delivery. We learn how developer productivity tools like Graphite and Bazel enable rapid iteration at massive scale.
Whether you're a solo developer or part of a huge engineering team, understanding these software development workflows can help you ship faster and more reliably. See how code review, testing, building, monitoring, and deploying happens behind the scenes at industry giants.
MY DISCORD: / discord
MY NEWSLETTER 💌
thebetter.dev
------
CONNECT WITH ME ON SOCIAL
📸 Instagram:
/ lewismenelaws
🎚TikTok:
/ lewismenelaws
🐣 Twitter:
/ lewismenelaws
-
My gear 💻
liinks.co/lewismenelaws
-----
Timestamps:
0:00 Intro
0:32 The Beginning
1:23 Working on Software
3:03 Stacking
5:21 Graphite
7:08 Trunk Based Development
8:25 Merge Queues
9:22 Continuous Integration
10:50 Continuous Delivery
12:32 Outro
Sources/Inspiration:
Why Google Stores 2,000,000,000 lines of code in a single repository: • Why Google Stores Bill...
Microsoft Windows Source code using VFS for Git: • Scaling Git at Microso...
Using BOTH Blue/Green and Canary: / blue-green-or-canary-w...
Merge Queues: www.uber.com/blog/bypassing-l...
Stacking: stacking.dev/

[@CodingWithLewis]

This video took a bit :) this video is meant to be a light introduction to this concept and we made a fake company that’s inspired by the real practices of MANY different companies. Enjoy 👩‍💻

[@mazyr5908]

thank you

[@alexandrupantea3170]

Thanks a lot for your content!! 🎉🎉

[@med.brunofreire]

VIDEO IDEA: How Apple's developers tests their native apps on every iPhone version?

[@foxedex447]

thank you !

[@VolodymyrInTech]

👍👍👍👍Great video as always!

[@Julio860JVL]

I got a merge conflict in my git and I was the only one working on it.

[@siteantipas6837]

😂😂😂😂😂😂😂 Hilariously funny. You made my day with this! 😅

[@anees2410]

😂

[@Wolfwuf]

I laughed my ass off to how relatable this is

[@jakubo1853]

This is so real

[@codewithshriekdj]

same happened to me also

[@avi12]

Dude, almost two years ago I joined a startup as a junior front-end software engineer, and if I had watched this video back then it would have saved me so much trouble

[@meanlowwater]

That was bloody interesting, mate, and the production value is unreal. Big thanks to you!

[@samuelmwendwa4327]

The promotion fits well into the video. I love such creators

[@christoslazaridis7128]

It is amazing the ways we have found to automate one of in my opinion the hardest problems out there, having many entities work independently on the same project, especially with something so big and hard to scale as modern software... now you got me hyped for my first internship!

[@robethius_]

Just learned git commands, downloaded git bash, made my first repo on github and pushed it up for PR using git commands. This video popped up right after, and now being VERY relatable. Thanks for all the info mate!

[@RyanLynch1]

i can tell this is well made because you actually researched and found some of these newer tools! good explanation based on my experience working with larger codebases and wrangling git

[@David-ss8bo]

I really liked the concept of this video, great mix between detail and accessibility!

[@mickoalhwint.andrada8080]

Great video as always! More power lewis!

[@wonderstruck.]

Words cannot describe how good Google’s internal code review & version control tools are. I’m now at a different company and I miss Critique & Fig every day.

[@aanchaallllllll]

1:02: 📚 The video discusses how big tech companies manage and update their massive repositories of code. 3:44: 🔀 Stacking branches allows for parallel development and easy rebasing onto the main branch. 5:52: 🚀 Graphite's developer productivity tools and stacking workflow help engineers increase developer velocity and ship code faster. 8:08: 📝 The video discusses methodologies like feature flagging and merge queues that allow for efficient code development and deployment. 11:16: 💡 Continuous delivery is crucial to avoid downtime and ensure a seamless user experience. Recap by Tammy AI

[@adib-the-noob]

Enjoyed the Video too much!! Thanks Lewis.

[@keenanwekesa1723]

I understood less than 5%, but I'll come back as a senior dev and hopefully it'll make sense.

[@CodingWithLewis]

Don’t worry! It’s extremely complex :)

[@shinsha_]

This was so well made!

[@chilusoftcorp7762]

this is so wonderful, free content with such good quality??? this is another level 🔥🔥🔥🔥🔥

[@alexaneals8194]

It would be nice if having CI and CD was the default for most enterprises, but in my experience it is generally the exception. Of course, I work at the lower backend (closer to the database level than many developers) and mostly with legacy code.

[@willi1978]

all code is legacy

[@alexaneals8194]

@@willi1978 I would not consider code written in the last one or two years as legacy code. The code I work with has been written over a decade ago in languages that are no longer in popular use.

[@cofucan]

Very well made video, great editing!

[@williebaxter4910]

These graphics are awesome! Great videos .

[@cexploreful]

Nice audio-visual introduction thx

[@SwankyShanky]

Absolutely beautiful video! Thenks

[@marcel_kogs]

Biggest challenge for big mono repos is still devops in my opinion. Pipelines and shipping usually don't scale that easy and can cause a lot of pain

[@grzegorz.kocjan]

The simplest solution is the hardest - make small, incremental changes. You don't need stacking if you can hide unfinished features behind feature flags. Do small MR-s and merge them to the main as fast as possible. Maybe stacking is good for 10 000 devs when not all of them have the skill to do small changes, but in smaller teams, I would rather invest in teaching people how to work smart ;)

[@MehmetKoseDev]

What an inspiring video and quality storytelling

[@user-re3de3nt5g]

Really insightful!

[@martijnp]

I'm personally really not a fan of mono repos. I do understand that there's been too much of a swing to microservice structures and I see that even my team suffers from that. But there's always a balance. Just splitting things up into libraries and then combining that into one mono repo that packages everything together could be a good solution and might prevent duplicate features, code, structure and comprehension. Another thing is that in some cases running services can help localize outtages or bugs. If everything can run separately and incrementally upgrade its own dependencies and systems you are less likely to run into issues. Of course with proper testing you *should* see most issues, but in practise things will slip through

[@infiteri6030]

Great video, and well made, keep up the great work!

[@mohamedaathik1060]

haha i have been waiting for a vid like this lewis

[@larsiparsii]

Recently I've been asking myself this question a lot! Doesn't even have to be on an enterprise scale. If a teammate and I both work on the same couple files, what's the best way to make sure we don't screw each other over?

[@sirynka]

Communication?

[@macsmola]

Talk to each other.

[@MartinMadsen92]

Communication really is the best way. If at all possible, this is the easiest (and perhaps the only) way to avoid screwing up each other's work. All that branching strategies do is adjust the timing of the screw-ups.

[@CoderDBF]

Try to stay away from each other within the codebase. Try to have some separation of concerns. You work on this feature, they work on that feature. Commit and merge often. Any merge conflicts that will inevitably emerge should stay small in scope and ideally be a matter of whose code goes above whose… rather than who is correct and who’s wrong.

[@MartinMadsen92]

I agree with @@CoderDBF that merge conflicts should ideally be a matter of whose code goes above whose, but I personally find that conflicts only really arise when someone is refactoring something that another person is working on at the same time. Merely adding new features is not a problem, as those conflicts are very easy to solve (it's a matter of choosing the order of each person's chunk of code). They are conflicts in the Git-sense, but not in a logical/behavioral sense. "Real" conflicts are conflicts in a logical/behavioral sense, and they may or may not result in a Git conflict. These conflicts should ideally be avoided rather than "solved". This is only achievable by coordinating the work with the other team members in some way. However, having to plan and hence postpone this kind of work also comes at a cost.

[@roromaniac8]

This is some incredible editing! Mind sharing what you used for the edits?

[@CodingWithLewis]

Premiere and after effects :)

[@muscular777man]

Amazing knowledge 😎

[@potaetoupotautoe7939]

finally some quality video

[@twiksify]

Very interesting, thank you for sharing

[@frogery]

my man, do you do all the editing and animation yourself?

[@Android-17]

Production quality is skyrocketing 🚀

[@sanchitgumber9676]

Wow what a great video!!!

[@ninojamestan8895]

This video is really high quality, as if I'm watching a documentary

[@InvisibleSquids]

Anything you make mistakes on in Git can be undone up until you do something that changes the history. Hard resets, force pushes, and anything forceful are 100% foot bullets that can cripple you, so use them carefully

[@vipulchaturvedi]

Great editing 🎉

[@CodingWithLewis]

Thanks 😁

[@justind4615]

@@CodingWithLewisThis is the best video so far!

[@codingvio7383]

I feel so good not having to worry about this kind of stuff. Mainly because I have no clue what it is going on in the first place LOL

[@McRinato90]

Very useful video, thanks for it. I also have a small question that interests me: what do you think about blockchain development? Now many applications on the blockchain use web3, openness. For example, NEAR Protocol, this BOS uses rust and java. Thanks for the answer.

[@ignaspan]

1:00 it’s not monolithic, it’s monorepo.

[@Rt.hype646]

In the realm of digital light, I dwell, A solitary soul with tales to tell, Before each comment graced this screen, I, in timeless verse, did intervene. A Shakespearean ode, I shall compose, With words that dance like poetic prose, My comment predates, a timeless grace, In this cybernetic, boundless space. "Thou art a pixel, mere fleeting glance, In this vast expanse of virtual dance, Yet, let thy words be noble and true, For in this realm, our hearts shine through. With keystrokes delicate, we do converse, In this modern sonnet, a universe, So, let us speak with wisdom and grace, In this digital age, let kindness embrace." Before the threads of discourse did flow, I scribed these lines in days of yore, In the spirit of Shakespeare, I convey, A timeless message, for night and day. For comments may come and comments may go, But this poetic echo, it doth bestow, A reminder to all who venture near, In cyberspace, let love and wisdom steer.

[@CodingWithLewis]

I think you got it

[@nikobalek]

yo bro I love you you are so good!

[@davidadesina3321]

Can you make a video on how KZbin stores videos and comments

[@newdiary6978]

We are a small team working on a POS system for various businesses such as malls , resto, etc. But still getting merge conflicts xD

[@yashgupta417]

we are also building a loyalty and engagement system for SMBs and are actively doing POS integrations. Are you serving in India, if yes we would be interested in partnership with you.

[@_Nonines]

I still love your vids!

[@DakotahMiskus]

By the title I thought you were going to talk about the Linux kernel

[@dx243_]

I thought this video was gonna be like a r/gitcodeplace experiment where anyone can edit the repository and there would be hundreds of trolls

[@gingeral253]

Which companies have thousands of software engineers outside of Big Tech?

[@aethro4375]

ASML

[@miguelgargallo]

An ad from Graphite as a video? but interesting 😂

[@MartinMadsen92]

The explanations given in this video for concepts like branching, continuous integration and continuous delivery are confusing at best, misleading or outright wrong at worst. Stacking of branches is presented as a solution to the problems that arise when thousands of engineers are working on the same codebase. But working in isolation on branches doesn't solve any of those problems, it only postpones them (till the point where you have to merge). In fact, I would argue that stacking branches like that is a *terrible* idea. Not only are you working on a feature branch that is not up to date with the main branch, you're actually building more feature branches on top of the assumption that your "base" feature branch will be approved without changes and merged into main, in which case you can continue working on your stacked branches. But what happens if one of the other 9,999 engineers requests a change to your first branch? Then you would have to apply that change not only to that branch, but to every branch that builds on it. Of course, that alone takes time, but it also significantly increases the risk of you having to rethink and redo something you already implemented on your other branches, because you're building your implementations on false assumptions about the quality of your first branch. Trunk-based development is presented as a branching strategy where you just merge back the branches "quickly". This is not correct. Trunk-based development is working _on_ the trunk/main branch. Continuous integration is presented as being about running lots of tests when the code has entered the main branch. This is also not correct. Continuous integration (as the name suggests) is about continuously integrating all code changes in the repository, i.e. merging to the main branch at least once per day (if you're not doing trunk-based development, that is). Continuous delivery is presented as a weird mix of Infrastructure as Code and blue-green deployments, both of which have nothing to do with continuous delivery. Again, the meaning of continuous delivery can be seen from it's name: It is about continuously delivering value (code) to production. The tools you use to achieve that are decoupled from the concept itself and should not be mixed with it, as it just causes confusion or will mislead someone who doesn't already know what those concepts mean. I can recommend watching some of @ContinuousDelivery's videos, David Farley does a decent job at explaining concepts like these precisely and sometimes even with references. Animations were great though.

[@CodingWithLewis]

Hey Martin! Thanks for the feedback while also keeping a professional tone. You are right, there is A LOT to cover for something that is so complex. Mostly, the video was targeted towards a demographic who are: a) extremely new to software development b) who don't know software development at all. So the idea was to always keep a casual tone in explanations while focusing on the storytelling aspect (from code editor to production). That being said a couple of things: - I wasn't saying Stacking necessarily was the be all or end all, I just explained how that was something they did to counter-act it. There are for sure some caveats to this method that you mentioned. - Trunk based development is actually about staying as close to the trunk as possible. Not working directly on it. This is with short lived branches. - You are right about the CI/CD aspects as I did confuse a couple of the methodologies within this umbrella of work back and forth. So I appreciate the feedback on that. Thanks for posting your critiques! Looking to improve this style of video for the next time around :)

[@MartinMadsen92]

@@CodingWithLewis Thanks for your reply. I get your point, I just think one should try to be precise especially when explaining academic or technical topics like the ones here. And I personally found your walkthrough a bit confusing even though I'm a senior dev, but it may be because I'm not the target audience, as you say :) Regarding trunk-based development, I found multiple sources agreeing with you as well as multiple sources agreeing with me. I guess there is no strict definition of the term, so my bad for correcting you on that.

[@amogus3023]

​@@MartinMadsen92 Hey, just want to comment on your concerns about stacking leading to more issues than what they solve. There are clear downsides to stacking especially when it's done with bad scoping or if you don't have good tooling for it, but many of the problems are mitigated by good tooling and scoping of the work. As you noted, changes being requested to a lower part of the stack you're working on can lead to having to change every subsequent branch in the same stack, and that can be a pain to work with. Regardless, it's still often a better approach than a single large PR would be, as what you're essentially doing is shifting the workload from the reviewer to the developer. From my experience it's the review specifically that tends to become a blocker, and anything that makes reviews easier improves overall productivity even if it's at the expense of time spent actually writing code. Consider a scenario where you could have a single large PR with 1000 lines of code in changes going through a single review, versus a chain of PRs where each individual PR adds a well encapsulated isolated module with around 200-300 lines of code changes, but just depend on each other in a chain. Dependency chains should be avoided when possible of course, but sometimes they become necessary, and just choosing to not split the work in smaller pieces is rarely a better option (though there are scenarios where it is the better option of course, such as large autoamted code style updates.) I suppose the main point is, most of the problems that PR chains/stacks have are possible to mitigate with good tooling that support stacks & updating of them easily, pre-approved plans of the higher level architecture (you know what modules you're adding and the exposed interfaces of them, implementation PR generally only reviews the implementation, meaning the exposed API is already something you can build on top of), and that it's a smaller problem to have to update your branch than it is to have a massive PR that's hard to review. From my experience, a lot of the potential issues of stacks can be mitigated by having a general higher level plan/architecture of what's being built ready & reviewed before moving to the implementation stage, but it's also been pretty rare to see individual developers utilize that level of planning in their workflow unless they're forced to. Having planning done on that level would of course be a good thing in any project that has multiple developers working on it.

[@MartinMadsen92]

@@amogus3023 That's a good point, and I completely agree. In general, I think developers should put as little load on their reviewers (and testers, if that have any) as possible, and instead secure their own work as much as possible before asking others to spend time on it. However, I wasn't comparing stacking to one large PR, which contains all the changes of the imagined stacked branches, only in a single PR, because yes, that would be even worse. What I think is the better alternative is to have as short-lived branches as possible, perhaps even trunk-based development, and focus on minimizing the time from commit/PR to review. If you're doing blocking reviews (such as PR's that cannot be merged until they have been reviewed), then prioritize reviews over development. If you're doing non-blocking reviews, then prioritize it anyway, because the potential blast-radius of any changes increases over time. To your point about planning, personally I don't find that relevant in the context of reviews, since I kind of take it for granted that the contracts are always in place before development begins. That is, the contracts that could lead to a change request due to it being "wrong". After all, how could it be wrong if it hadn't already been agreed upon?

[@amogus3023]

@@MartinMadsen92 Hey, yeah that all sounds reasonable to me too! I think stacking is most likely to take place in an environment where you're attempting to make small PRs, but reviews take long so you end up being blocked from working further unless you do stacking. What I was referring to with planning is that you can try to plan all the work steps in a way that there are minimal chained dependencies and more sections that can be worked in parallel, and the parts that are chained are less likely to bounce back from review in a way that'd lead to conflicts (conflicts only happen if the contract changes after all, internals are easy to just merge into the deeper branches in the chain without risk of conflicts). Parallel branches are always better to work with than chained of course.

[@gagne.michael]

That bazaar development model lol

[@Jodasi_ig]

My hero

[@evelinodavidarevalo1692]

I don’t understand this but something inside me does. That’s why I’m here…

[@GetShwiftyInHere]

I'm new to coding and when I see stuff like this or other videos pertaining to high-level coding or all these different things outside of the languages themselves like React and Angular, it gets daunting. When I first started this, I assumed it was as easy as learn a language and have a code editor. And hearing how you can google a lot of the answers but then doing coding as a logical process seems lost on me. Why would I learn the logic behind my code when I only googled the answer and it worked? It feels like it defeats the purpose in actually learning and just copy/paste with tweaks to what everyone has already done. It's like seeing the same movie come out with a new cast. I'm still going to learn, though.

[@CodingWithLewis]

The good news is, this video isn’t meant for you to take anything away to learn (specifically) but more to understand how complex these systems get! Don’t be discouraged :)

[@GetShwiftyInHere]

@CodingWithLewis As my feedback is important for your channel, your feedback is important for my personal growth. Thank you for taking time out of your day to reply! Keep up with the great content though!

[@eligbuefelix7988]

I love your presentation style. Absolutely amazing

[@Abenezer__]

from ethiopia ❤

[@RedstoneHair]

Explain the ducks!! I know but, I need your humor plss

[@foxedex447]

i love your titles very interesting to click

[@SandhyaRani-np9be]

❤from India

[@KarunanidhiVijay]

Me too. 💗 from India.

[@nitujha4884]

Nepal 🇳🇵🇳🇵❤️❤️❤️

[@aniketdas1145]

Hello Sir , Can you create Java script video beginner to pro

[@sam-wj9lr]

Controversial hot take: There is no software product in the world that requires 10,000 engineers to work on it simultaneously. Just like we saw with twitter

[@mr.shroom4280]

Where's the podcast gone 😭😭

[@CrucialTech]

This is a problem I had just a few days ago

[@henshalb]

This is cinematic

[@infodsagar]

Fetch a day keeps big conflicts away 😅

[@jyotirmoybandyopadhayaya]

Is this video sponsored by Google? But yeah awsome content

[@_romerdev]

I dont understtand everything but I finished the video because I want to become one of programmers hehe :

[@FrizzyTrademark]

2B+ lines of code⁉️ What is this project😭

[@rahim.tn99]

commit, push and pray

[@notredeye3962]

Hello , i don't understand coding and tech. But i found a tool in github that i want to use. But the tool is just a file with codes (python). I want to know how can i run the tool as an app or install it . Please let me know

[@lokeswaranaruljothy8100]

How does one understand a legacy codebase with millions of even billions of lines??

[@nomadshiba]

why is it massive

[@vadym-beep1241]

Fifth comment? This video came out 4 mins ago!

[@pronoob249]

cool

[@MrDragos360]

If the project got sooo big that you have 10k people working on a repo you are doing something wrong. You need micro services. And for a front end app to be soo big there is again something very wrong. Even more, if a PM request a featurere based on another feature that is not ready to be done in paralel (the example from the video) he has a problem and should step down. Also, I never got why software devs love to over complicate stuff and not keep it as simple as posible. My take is that they want to both flex their own skills and rise the dificulty level so new people will have a harder time joining the industry so they improve their jobs security. All software can be done with 50% less people, 50% less hassle if people would care more about results, simplicity and not about flexing their own skills.

[@robertobokarev439]

No matter how many people are working on the project, the main thing is to not allow that one stomach cancer from StackOverflow to manage the process or be a part of it

[@SamuelViagus]

PLEASE HEART THIS LEWIS!!!

[@dhruvvaria]

Le me doing, ver1, ver1.1, ver1 4, ver 3, ver 4.32 😅😅😊l😂

[@allergictobs9751]

All programmers are working to eliminate all programmers and we wonder why our salaries are not increasing. We are fking idiots.

[@Louis-L186]

“You just begin a job in IT”, image : an apple laptop 😹😹😹 Feels like cognitive dissonance

[@Xiefux]

yes, people with jobs use apple laptops since they're the best tool for the job

[@Louis-L186]

@@Xiefux you mean managers ? 😂

[@Xiefux]

@@Louis-L186 among the many who use them, yes. they are the most successful in a company no doubt thanks to using apple devices day to day

[@s.m.abbasali9374]

Hi i am first to make comment here 😂

[@CodingWithLewis]

Second actually

[@s.m.abbasali9374]

Ohh , still it's top 3 😁 I am still happy getting recognized by you Lewis !

[@ImTheDot]

.

[@mazyr5908]

frist

[@marsguy2200]

frist off

[@arbitervildred8999]

I think your video is broken... it keeps zooming and swiping every 5 seconds 💀💀

[@legendrags]

Ok I changed my mind I will not join big corps

[@Eric-xh9ee]

This video is too long but you're just telling me they're using git and everyone probably isn't working on the same file so there shouldn't be a lot of merge conflicts

[@ConanDuke]

God I hate the future.

[@patryk4815]

master, not main

[@simonabunker]

Monolithic repositories are dumb. There really is no need to put everything in a single repo - especially if they are unrelated projects.

[@AdamPippert]

The point is that they’re unrelated now, but in the future they could be integrated together for some new product more easily than if it was in a separate repository. For bigger companies with lots of products this can be a great feature, though I work for one of these and we definitely do separate repositories (it is easier to do open source development with separate repositories since there are separate groups doing the development)