Monorepos - How the Pros Scale Huge Software Projects // Turborepo vs Nx

[@AzureFlash]

Finally, I can have a giga-scalable monorepo and build tools for my 100-line Discord bot written in Python

[@gh05tparkourfreerunning31]

I was actually thinking of using this for my Discord bot lol. It has a REST API to manage a roster of members in a DB and I wanted to make a website to interact with the database directly as well. Except in this case, the Discord bot would be a tool for the entire application

[@donquixoteupinhere]

Hahaha. Yeah context is VERY important as to whether this is the recommended route someone chooses, for highly practical reasons.

[@Mkrabs]

***written in a single Python file

[@andrey2001v]

And 40 of those lines are imports. This is literally most of my uni coding experience

[@arctic.wizard]

This is the way

[@patrik5123]

This is by far the best channel to get enough knowledge in order to to make it sound like you know what you're talking about.

[@patrik5123]

@@mikkelens Yea I've spent enough time in front of online courses during the various lockdowns to know how demanding it is. I've spent countless hours on React and I still feel like I don't _actually_ 'get it'.

[@reed6514]

I like these, more for entertainment than learning. My preferred learning style mainly involves diving in, writing code, and looking at documentation.

[@noskillzdad5504]

Fake it till you make it!

[@reed6514]

@@noskillzdad5504 i do this when I'm manic lol. Sometimes it's good. Sometimes not

[@heroe1486]

@@reed6514 It's good to discover things and know if you want to dive deep into them, like if a video showcases a new framework which has some particularities you can't live with, if you were able to spot them in the video it may save you hours of reading the docs. Same for this one, most people probably went to the conclusion that Monorepos weren't yet a fit for their projects. These videos are handy in a field with so much (sometimes useless) novelties

[@brennanfife]

Software teams then: ‘This codebase is too coupled together… let’s break it up’ Software teams now: ‘This codebase is too broken up… let’s couple it together’

[@overloader7900]

When the moores law died (as in single core SISD performance, cuz its too hard to make parallel code), code performance became important again

[@ronanharris8216]

That one guy on the team : "... this goes against everything that I know to be right and true ..."

[@muizzy]

At the risk of sounding like I don't get the joke; a little context for the uninitiated: We're still decoupling as much as possible, we're just doing it in a smarter way which allows us to make simpler use of a single source of truth. (1 monorepo, but still many micro-services)

[@MiniKodjo]

do then undo is work

[@dorktales254]

Google's codebase: moaaaaai

[@oussamasethoum1665]

You explained TurboRepo better than the creator himself, huge respect to Fireship 🎩

[@andresramos7965]

At this point, all software should feature a Fireship video in their mainpage

[@oussamasethoum1665]

@@andresramos7965 I couldn't agree more, they should pay this guy to make more understandable explanations.

[@hydra4370]

@@oussamasethoum1665 I already pay him to explain stuff to me 😭

[@magnusmarkling]

@@oussamasethoum1665 Maybe you mean WE should pay him

[@forrestmorrisey]

You're comparing apples to oranges here. Engineers are nutoriously bad at explaining their own work. Fireship is a seasoned technology communicator.

[@KeithGraves]

What people call best practices in software engineering amuse me. Backend: Everything is a microservice Frontend: Use micro frontends Git: Put everything in a giant monorepo

[@matheusvictor9629]

The thing is about performance and not everything going down at once, where you store the codebase doesn't matter

[@TheBillionDollarSaaS]

Infrastructure and code have different requirements

[@webentwicklungmitrobinspan6935]

no! its all about decoupling. microservices to enforce a api and maybe consider an integration layer. and you split your frontend because you dont want to compile for minutes at a time and have a 100gb node module installed for fixing a a hardcoded link in a selfservice tool which no one seems to use anymore but people manage to complain about somehow

[@dadudeme]

​@@matheusvictor9629as linus torvalds said: Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it

[@VarunGupta3009]

I scouted the entire internet a few months ago to get the exact same monorepo build setup as I'm used to at Google, but here you are, who explained it all so beautifully in a video under 10 minutes. Bless you.

[@webentwicklungmitrobinspan6935]

how long have u beeing coding before getting into google? is it still worth it?

[@bakedbeings]

Another fantastic video, just one nit pick: it wasn't super clear (to me) early on that this was about repositories for web tech specifically. When you started out with 2 billion lines of Google code I had their C++, Go etc code in mind - a major driver of Go's toolchain design was long C/C++ build times on huge code bases - but Turborepo is js/ts only.

[@omarkarim9298]

Using build tools like maven/gradle and scripting I found is the best way to go for multiple projects of different programming languages

[@tmthyrd]

Thanks for pointing this out, I completely missed this point. It's somewhat ironic that a tool written in Go to manage repos doesn't actually support Go. Does NX or Lerna support a larger set of languages?

[@IxMeTutorials]

@@tmthyrd Lerna is based on npm, so probably not.

[@nathanhedglin931]

There is only JavaScript 😂

[@Zokiio]

But Bazel that google use does actually handle multiple codebases like Java, C++, Go, Android, iOS

[@vinaymama]

Just few days back , i saw the Fireship comment in the TURBOREPO launch video. Now we have a complete tutorial on it... Thank You dude

[@akillersquirrel5880]

Having recently left a company that was built around a monorepo, here's my takes: Pros: * Visibility - it's easy to see where things are used and discover stuff. * Uniformity - no need to manage versions between things used internally. Cons: * Upgradability - you can only upgrade something once *all* of its users are upgraded. * Maintenance - if you don't invest enough talent into setting up and maintaining your monorepo tools, it's easy to get to a point where they are no longer usably fast. * Speed - if a simple `git status` command takes multiple seconds to complete, nothing in your project will feel snappy.

[@Flackon]

I've actually started using monorepos for a couple of solo projects, due to the ease of packaging and dependency sharing

[@krishgarg2806]

Thank you for covering this. Recently vercel acquired turbo repo and I was confused what was it as I use vercel a lot.

[@ultimatum97]

As somone who works at a certain large software company with proprietary source code manager, repository and build tool I can tell you it is no fun at all. It is a nightmare dealing with dependencies when you don't have Maven or Gradle.

[@JChen7]

Sounds like someone has a case of the NIH blues.

[@bobobo1673]

"Maven or Gradle" Look with Gradle I can't comment but with Maven? are you serious? Do you know how complicated it is to simply add a project and the problems it gives looking for its dependencies? Sometimes you have to solve the problems yourself by hand (looking at the dependency tree, and that I don't even talk about projects with Wildfly is a horror) and it makes me lose time from work when all I wanted was to install a library

[@not_nerp]

would this proprietary build tool happen to share a name with a certain south american country

[@ultimatum97]

@@bobobo1673 Be thankful that atleast it downloads the versions you mention in pom. The build system I work with needs a kind of dummy project just to deal with dependencies. It then does some black magic, pulls the wrong jars in a secret location and then shoves it into the cryptic buildpath directly during environment level global build 🤣🤣

[@ultimatum97]

@@not_nerp Maybe....not

[@akashkumar121213]

I have been setting up a typescript monorepo for one of my personal projects ... And lets just say there are some pains you are gonna feel that you wont forget.

[@webentwicklungmitrobinspan6935]

.. cant wait to get through these pains in my next vacation.

[@thepianist6425]

Would love to see a video covering Micro Frontends. Having said that, always a big fan of the quality of videos. 10/10 college degree material right here. I should start adding fireship io under the education section of my resume now.

[@DominicanRepublicInvestment]

Second this!

[@muizzy]

I work with a micro frontend every day (though admittedly on the backend). Do you have any specific questions? I may be able to answer them.

[@ThiagoVieira91]

@@muizzy Do you think micro frontends improved your development experience in you project? How did micro frontends improved your team workflow? Can you talk about some trade offs in comparison to currently traditional frontend architectures?

[@JChen7]

Watched "Java in 100 seconds". - Proficient in Java

[@essayemyoung4009]

Would check out Jack Herrington’s videos on this (e.g., kzbin.info/www/bejne/onyupJ2mfZVsmrc where he goes through a React example in detail)

[@0xedb]

turborepo is great. I just want a demo where everything is done from scratch instead of the boilerplate to better explain it.

[@AndersonSousa33]

me too

[@kevyyar]

That is my only problem with Fireship. Everything else is great.

[@floffah264]

their docs have a good step by step process of this and their examples help a lot

[@saurabhjainwal4604]

We use nx at our firm, it’s really cool and definitely helps maintain our mono repo. 😀

[@middlefloor]

🙌

[@skyhappy]

@@middlefloor Why are you raising your hands? Are you catching a basketball?

[@middlefloor]

@@skyhappy yes!

[@skyhappy]

@@middlefloor well this is a tech channel not a basketball one

[@johngc2010]

@@skyhappy Jeffrey Cross is one of the maintainers of Nx...

[@dmitrii_zolotuhin]

Man your channel is pure gold.

[@devbyemil5201]

You are really doing some great explanations! How about a video about how to plan software and what tools to use? Would love to see it👍

[@pablopoggio4899]

That’d be pretty cool actually

[@JonathanAdami]

"You might know him with packages like Formik" Telling me that after 3 weeks of trying to solve issues with fomik and eventually having to refactor my entire code base to use react-hook-forms (which is MUUUUCH better) makes me wanna ditch Turborepo even if I know nothing about it hahaha Cheers for the video!

[@jztsaurabh8111]

Formik has its own issue but it created the whole form library ecosytem

[@codevev]

That's literally what we've been doing in the last couple weeks haha

[@Filaxsan]

Our man Jeff becoming funnier and funnier (and informative!) every video. Way to go Jeff, love you

[@ShawnCZek]

Another issue with monorepos is the access management. Especially with the public available tools, you cannot easily define what directories one should have access to.

[@devnol]

Doesn't github's codeowners do that?

[@qwerty-or1yg]

@@devnol I know that they can limit access to specific repos, but not directories inside a repo. Don't think they can limit your access to specific directories inside a repo (unless I missed and not aware of something)

[@mudscuffer]

@@devnol Codeowners can limit write / merge / PR rights. But not read access.

[@muizzy]

Aside from fringe cases (like fraud rules), why would you want to manage read access?

[@mudscuffer]

@@muizzy For example if you use freelancers or outsourced development for the frontend or for integrations, but don't want them to see your proprietary backend algorithms.

[@clementbowe7594]

I'm probably really late on this but congratulations on reaching 1m subs!! I watch daily and always learn a new thing or two. I hope that you keep making amazing content!

[@williamchamberlain2263]

0:30 I like that the codebase is elvoving blindfolded _and_ being pulled side to side by different teams _and_ will irrevocably topple over with crushing force after just one pull too hard or in the wrong direction. Great metaphor.

[@andyhall7032]

I like how the narrator sounds like text-to-speech software...it's one of the many appealing aspects of this channel.

[@krishgarg2806]

Time to use turbo repo for my todo app.

[@nathanbrown19]

Worked in both mono and traditional repos. Monorepo removes so many dependency and versioning headaches.

[@archmad]

It can go to an opposite way. 1 app only supported old version while the other uses the new

[@jedcal]

i love this channel

[@kettenbach]

Frikin love you man! It's been 84 years. That was frikin hilarious bro. 💪❤️👍

[@alexIVMKD]

Your videos are spectacular, congratulations in advance for 1M

[@cornheadahh]

I'm loving the frequent uploads

[@anubhavbansal8922]

too much of information in just 10mins that's why I love you

[@KevinVandyTech]

I just converted most of my company's front end code to Nx. It's been a game changer

[@middlefloor]

Awesome to hear!

[@KerimWillem]

You have no idea how helpful this video is and how much time it saved me. THANK YOU.

[@dlysele]

Quality video that everyone should watch

[@PatricioHondagneuRoig]

Your videos are my non-guilty pleasure, I always leave having learned cool stuff and enjoying it, it's like having celery for dessert.

[@codeaperture]

Let's get fireship to the moon 🚀

[@isaac4386]

I feel like your tracking my Google search wow. Been researching monorepo since the weekend and now you post this 🤣 . Love it

[@wheytomuchforher]

Serious question. Why would I use this instead of NX's built in CLI tooling to run the "affected" aspects of my code base. I use NX for both of my side projects

[@syamjulio3685]

I'm interested in this as well.

[@middlefloor]

Turbo would probably be used instead of Nx, not alongside it. If you're already using Nx, turbo won't do anything for you that Nx doesn't do.

[@Viviko]

I usually only use Monorepo on things like Microservice based systems. I kind of feel like having one repository for ALL your products kinda is overkill. But I guess not, after watching this.

[@AhmedKachkach]

What's overkill is going through the effort of maintaining different repos for a small project. If your web app, your mobile app and your data processing jobs all use the same data representation, why not store everything in the same repo and keep everything in sync?

[@yurayurec6071]

Top quality as always 👌 waiting for your video on webpack module federation, another hyped tool on the market

[@code_react]

you saved my life by explaining how monorepos works. Thank you so much.

[@erkinkurt6799]

I was searching for this to decide if I should use this approach or not. OMG either you can hear me or you have amazing skills to access my search history

[@a.c.vermillion]

I was waiting for so long to finally see this uploaded. Thank you!

[@nmanikiran]

Please build a course around it (mono repo) with all the tools you have mentioned. take 1 big project and architect the same in all tools like nx, turborepo, lerna, yarn, blaze, webpack module federation etc

[@junzhengca]

Monorepo coupled with almost perfect IaC is a god send, it gives amazing visibility to everything. Yes it is complex, and the on-boarding learning curve will be steep, but once you are familiar with it, you don’t even need to leave your IDE to find anything.

[@theatypicaldeveloper]

you should always think big and leave an extra space in your app (if possible) so it's growth-friendly great vid, as always!

[@andreb.1352]

you should always think small, and iterate it to the better.. otherwise you end up in perfection and you need a lot of time instead of just shipping a lot of small parts

[@zyriab5797]

Keep it lean and modular

[@FilledStacks]

Awesome vid. Good idea to build something like this for flutter as well

[@chinmaykabi]

Melos?

[@FilledStacks]

@@chinmaykabi We use melos but it's not a build tool, it's more accurately defined as a "script coordinator". You're bundling commands for multiple packages under a single command. No caching, no smart rebuilds, etc. It will always execute all commands that you defined.

[@OptrixTV]

i knew this was coming when i saw you put "nice" on Vercel's stream on Turborepo :D

[@bruceaxelrod8291]

Monorepos are great but how to control visibility of the code & access to the code if everything is in one repository. Like some departments are not allowed to see specific directories/code & things like that

[@homeofmotivation2339]

Yes, I was also thinking about that. Everyone can see everything

[@MercyFromOverwatch2]

Jeff is still my favourite tech youtuber

[@michaeldausmann6066]

Just finished refactoring my big messy Nuxt + background processes app into a clean and efficient monorepo with turborepo. Should have done this from the start, so much better

[@IsaacAlcocer]

You just give me an idea for a personal project, kudos... this is a nice video to watch.

[@AbdulbakiDursun]

HOW. DO. YOU. KNOW. ALL. THIS.?

[@midas6659]

Best tech KZbinr on the planet yo

[@osagiee9553]

ive been feeling the need for a monorepo for quite some time. this is helpful.

[@maxencedc]

Google dev : changes one line of code Piper : here, deal with those 15131 merge conflicts

[@haxney]

You only deal with merge conflicts in those specific files you are modifying. There is (almost) no branching, so you don't have to deal with merge conflicts from other people's commits. If someone changed the API of a library you depend on between when you start writing a change and when you submit it, then you'd have to deal with that, but that's a reason to make frequent, small changes. Ideally, each commit to head should take under a week to write and be only a few hundred lines of code at most. Obviously, that's not always possible, but it's a good starting point. If I make a change to a common library which would break other people, then it's my responsibility to provide a migration path to the new behavior. Since it's a monorepo, I can easily find all of the callers of my library and either fix them myself or ask the owners of those files to fix them.

[@maxencedc]

@@haxney Very interesting, thanks

[@yodkwtf]

So much information in one video. Huge respect. ❤️

[@lixou]

Me finishing after 3 days my MonoRepo setup. 2 hours later, this video got uploaded….

[@abh1yan]

Great video Jeff, really wanted to see this.

[@badreddine8727]

Thank you for the video I was confused when vercel acquired turbo repo and what it was, this cleared up a lot of things

[@joerivde]

If you do want to switch to linux, I can recommend Zorin OS (pro version). Switched a year ago, never going back to windows for anything other than gaming or 3D related projects.

[@MikePeiman]

Thanks for sharing, I had a look and this might be my next OS!

[@WhatIsThis-zq4hk]

why can't we go back to the good old days when programming was actually fun. Programming in 2021 is just spending all your time trying to figure out how to use 15 thousand tools that are supposed to make your life easier but in reality it actually makes your life hell.

[@jesusmgw]

Having sub 10 second rebuild times is actually massive for productivity.

[@felixoghina553]

Setting up a monorepo for all my unfinished projects

[@wlockuz4467]

After this video, I can confidently add 5 years of experience to "Managing monorepos" on my resume and move my Tic Tac Toe app and Todo app which have nothing in common, under a single monorepo.

[@roberthall9452]

I was too distracted by Bob and Alice. They seem like great programmers.

[@MikePatterson8831]

The weirdestthing for me is seeing big companies modify distributed version control systems so that they operate more like centralized version control systems. All the problems with git scaling is from git's own setup. Really glad I no longer work at one of the big companies mentioned here. Central version control systems manage monorepos so much better in terms of speed and out of the box access management. And this particular company's proprietary build tool had such poor documentation that it was almost tribal knowledge when you needed someone to setup the build pipeline or fix a bug. Great video!

[@marccawood]

I love the fact that you can download the cache. Oxymoron in action.

[@sakthisanthosh0103]

You're videos are great. Congratulations for your 1M subscriber mark. 🎉🎉🎉

[@ZackHoherchak]

Great explanation, but efficiencies aside, I think I still prefer separate repositories for MOST code bases. For some reason it is easier for my brain to keep things separated.

[@ed_iz_ed]

pretty interesting, my mind works a lot better if everything is in one place

[@reed6514]

i'm a php dev with about 10 foss packages, many depending on eachother, and composer (php's dependency manager) works great for me. & I think gitlab has a built-in feature for mono-repos, though idk how feature rich it is.

[@reed6514]

Okay, i have quite a few more foss packages than 10 ... but i estimate only about 10 of them are useful ... i just open source most stuff i write.

[@DiegoBM]

So two questions mainly. How do those tools help solve the massive git stack of changes and vscode to keep up with them, and do nx and turborepo override lerna? Or is lerna supposed to work in tandem with any of those? From what I can read in turborepo's site it seems like it can do all that lerna does?

[@Balance-8]

you literally read my mind on the video I wanted to see. Incredible!

[@ThaChillz]

Thanks Jeff, this video was very cache

[@nabeelsherazi8860]

One problem I would’ve liked to see addressed is that at the Google level, it’s probably impossible for most devs to clone the entire monorepo. How do they clone down just the projects they’re working on and their dependencies?

[@Nitr4Oo]

As it was stated, they dont use git but their own versionning tool called blazed, so they're able to clone only some workspaces of the repo

[@ultimatum97]

Basically they can choose what project (it's called something different) you can "clone" (it's called something different too)

[@Fireship]

It's also worth noting that git has a "sparse checkout" command for big monorepos.

[@nathanhedglin931]

1. They use an online code editor that edit the code without cloning it. 2. Their version control is central so the commit history is never local.

[@Nitr4Oo]

But to be honnest it's scary to try to understand how they are coordinated with such number of developpers. Yes of course they are split into teams working on different workspaces and each workspace has different micro services / modules where only 1 team work on it, but still

[@GagandeepSingh1984]

So much to understand in this build flows .. thanks for sharing it..

[@conororeilly5492]

I've got 3 apps, possibly a 4th coming if i finish the current project soon all along the same theme, i might try creating a monorepo for them to share some of the common functionality. It's most likely overkill at this stage but each app was to learn something, so i may as well do it and see what i learn

[@racecarjonny8460]

I have worked with monorepos built with lerna. OMG are they difficult to maintain!

[@lambrohan]

I knew it coming, since you starred the repo 2 days ago!

[@uwuLegacy]

YES finally monorepos are getting more coverage

[@abhinavrobinson2310]

Amazing,

[@laudijksterhuis]

Video request: Firebase emulator suite 😃 Also, great video, as always 💯

[@MrMeszaros]

Cool - like Gradle! Except that build caching sharing - that sounds like fun

[@wakham11]

1mil on the way!

[@BlackdestinyXX]

Really interesting, thanks

[@nitroflap]

Funniest thing is that today I was thinking exactly about monorepos & multi repos. Fireship reads our minds.

[@ihateorangecat]

hey jeff i love your voice.

[@YonatanAvhar]

The one thing I didn't understand is this: when an engineer joins Google for example, do they need to clone the code for every Google app to their local machine? Do they get an extra 3TB hard drive just for code?

[@DanSoloha]

My guess is that they only download the projects they need, but I'm not an engineer at Google, so my guess is as good as yours

[@MyPhuckDub]

Google organizes its monolith in different directories known as workspaces, with each having a team and a responsible engineer.

[@albert21994]

Get the Linux DX with remote containers ;) you can even clone the repository to the docker instance to get the speeds

[@AlexEscalante]

I think I am trying Turborepo soon! Thanks for your videos!

[@NezzyLawd]

Been watching for ages, but not subscribed. I don't know how!

[@rayanez]

Awesome video, thanks for the info. Do you know if there's a similar tool for Java codebases? or, I don't know if it makes sense, heterogeneous code bases?

[@middlefloor]

Gradle's probably the closest Java-first tool. Most of the tools in the video can be used for multi-language codebases, but Bazel and Nx have plugins that make it easier to work with different technologies/languages.

[@VolodymyrDovbenko]

That's a good piece of self-criticism regarding using Windows and not using Linux :)

[@titzko987]

expected something about an opensource java logging library today

[@alexiscadorette6848]

One downside I could see with monorepos is access control. In certain cases I do not want all developers to have access to every projects because A: they don't need too and B: it may contain proprietary info about a client. Having a monorepo for all the pieces of a same project is fine, but again if I want to share something between those two I have to extract that package in a separate repo.

[@aliasgharkhoyee9501]

Should the code repo have clients' proprietary info? I would have thought that's stored separately.

[@vikrantshah1633]

Thanks for fulfilling the request

[@jonasbroms]

THANKS! Great video! I have to test this now on my monorepos. :D

[@arcrimeaball]

I'm not an expert in monorepos, but I think another drawback of them is security. With a single hacker breach all codebase of a company is leaked, which puts in danger all products at the same time 🤔

[@viperactual]

Just having a codebase isn't really a threat much. You'd need much more. It's normally the data produced by the code you want to secure. Obviously, elsewhere. This is why, let's say, you don't commit .env files.

[@dealloc]

Same could be said for any source control management. Of course there will always be the endless battle against vulnerabilities and attackers. That's it's a good idea to hire people to can deal with these threats and make sure the infrastructure is secure. In fact, it could be argued that it would be easier to secure a single repository than having to deal with thousands.

[@arcrimeaball]

@@viperactual Yep, considering that majority of the modern solutions are data-driven, that's a good point!

[@fancylad22]

@@viperactual tell that to your legal team at work.

[@j03Biz]

Bob and Alice are running Google's codebase. That explains it!

[@metaltyphoon]

This is crazy… this “mono repo” based of one language only can be done with a Solution file in Visual Studio. It even supports multiple languages 😂

[@ynokenty]

Isn't VS dead for modern webdev yet? :)

[@kusayfarhan9943]

@@ynokenty web dev isn't the only software type around...