Monorepos - How the Pros Scale Huge Software Projects // Turborepo vs Nx

[@AzureFlash]

Finally, I can have a giga-scalable monorepo and build tools for my 100-line Discord bot written in Python

[@gh05tparkourfreerunning31]

I was actually thinking of using this for my Discord bot lol. It has a REST API to manage a roster of members in a DB and I wanted to make a website to interact with the database directly as well. Except in this case, the Discord bot would be a tool for the entire application

[@donquixoteupinhere]

Hahaha. Yeah context is VERY important as to whether this is the recommended route someone chooses, for highly practical reasons.

[@Mkrabs]

***written in a single Python file

[@andrey2001v]

And 40 of those lines are imports. This is literally most of my uni coding experience

[@arctic.wizard]

This is the way

[@patrik5123]

This is by far the best channel to get enough knowledge in order to to make it sound like you know what you're talking about.

[@patrik5123]

@@mikkelens Yea I've spent enough time in front of online courses during the various lockdowns to know how demanding it is. I've spent countless hours on React and I still feel like I don't _actually_ 'get it'.

[@reed6514]

I like these, more for entertainment than learning. My preferred learning style mainly involves diving in, writing code, and looking at documentation.

[@noskillzdad5504]

Fake it till you make it!

[@reed6514]

@@noskillzdad5504 i do this when I'm manic lol. Sometimes it's good. Sometimes not

[@heroe1486]

@@reed6514 It's good to discover things and know if you want to dive deep into them, like if a video showcases a new framework which has some particularities you can't live with, if you were able to spot them in the video it may save you hours of reading the docs. Same for this one, most people probably went to the conclusion that Monorepos weren't yet a fit for their projects. These videos are handy in a field with so much (sometimes useless) novelties

[@brennanfife]

Software teams then: ‘This codebase is too coupled together… let’s break it up’ Software teams now: ‘This codebase is too broken up… let’s couple it together’

[@overloader7900]

When the moores law died (as in single core SISD performance, cuz its too hard to make parallel code), code performance became important again

[@karakunai_dev]

That one guy on the team : "... this goes against everything that I know to be right and true ..."

[@muizzy]

At the risk of sounding like I don't get the joke; a little context for the uninitiated: We're still decoupling as much as possible, we're just doing it in a smarter way which allows us to make simpler use of a single source of truth. (1 monorepo, but still many micro-services)

[@MiniKodjo]

do then undo is work

[@dorktales254]

Google's codebase: moaaaaai

[@oussamasethoum1665]

You explained TurboRepo better than the creator himself, huge respect to Fireship 🎩

[@andresramos7965]

At this point, all software should feature a Fireship video in their mainpage

[@oussamasethoum1665]

@@andresramos7965 I couldn't agree more, they should pay this guy to make more understandable explanations.

[@hydra4370]

@@oussamasethoum1665 I already pay him to explain stuff to me 😭

[@magnusmarkling]

@@oussamasethoum1665 Maybe you mean WE should pay him

[@forrestmorrisey]

You're comparing apples to oranges here. Engineers are nutoriously bad at explaining their own work. Fireship is a seasoned technology communicator.

[@KeithGraves]

What people call best practices in software engineering amuse me. Backend: Everything is a microservice Frontend: Use micro frontends Git: Put everything in a giant monorepo

[@matheusvictor9629]

The thing is about performance and not everything going down at once, where you store the codebase doesn't matter

[@TheBillionDollarSaaS]

Infrastructure and code have different requirements

[@webentwicklungmitrobinspan6935]

no! its all about decoupling. microservices to enforce a api and maybe consider an integration layer. and you split your frontend because you dont want to compile for minutes at a time and have a 100gb node module installed for fixing a a hardcoded link in a selfservice tool which no one seems to use anymore but people manage to complain about somehow

[@dadudeme]

​@@matheusvictor9629as linus torvalds said: Only wimps use tape backup: real men just upload their important stuff on ftp, and let the rest of the world mirror it

[@VarunGupta3009]

I scouted the entire internet a few months ago to get the exact same monorepo build setup as I'm used to at Google, but here you are, who explained it all so beautifully in a video under 10 minutes. Bless you.

[@webentwicklungmitrobinspan6935]

how long have u beeing coding before getting into google? is it still worth it?

[@vinaymama]

Just few days back , i saw the Fireship comment in the TURBOREPO launch video. Now we have a complete tutorial on it... Thank You dude

[@akillersquirrel5880]

Having recently left a company that was built around a monorepo, here's my takes: Pros: * Visibility - it's easy to see where things are used and discover stuff. * Uniformity - no need to manage versions between things used internally. Cons: * Upgradability - you can only upgrade something once *all* of its users are upgraded. * Maintenance - if you don't invest enough talent into setting up and maintaining your monorepo tools, it's easy to get to a point where they are no longer usably fast. * Speed - if a simple `git status` command takes multiple seconds to complete, nothing in your project will feel snappy.

[@Flackon]

I've actually started using monorepos for a couple of solo projects, due to the ease of packaging and dependency sharing

[@krishgarg2806]

Thank you for covering this. Recently vercel acquired turbo repo and I was confused what was it as I use vercel a lot.

[@bakedbeings]

Another fantastic video, just one nit pick: it wasn't super clear (to me) early on that this was about repositories for web tech specifically. When you started out with 2 billion lines of Google code I had their C++, Go etc code in mind - a major driver of Go's toolchain design was long C/C++ build times on huge code bases - but Turborepo is js/ts only.

[@omarkarim9298]

Using build tools like maven/gradle and scripting I found is the best way to go for multiple projects of different programming languages

[@tmthyrd]

Thanks for pointing this out, I completely missed this point. It's somewhat ironic that a tool written in Go to manage repos doesn't actually support Go. Does NX or Lerna support a larger set of languages?

[@IxMeTutorials]

@@tmthyrd Lerna is based on npm, so probably not.

[@nathanhedglin931]

There is only JavaScript 😂

[@Zokiio]

But Bazel that google use does actually handle multiple codebases like Java, C++, Go, Android, iOS

[@akashkumar121213]

I have been setting up a typescript monorepo for one of my personal projects ... And lets just say there are some pains you are gonna feel that you wont forget.

[@webentwicklungmitrobinspan6935]

.. cant wait to get through these pains in my next vacation.

[@williamchamberlain2263]

0:30 I like that the codebase is elvoving blindfolded _and_ being pulled side to side by different teams _and_ will irrevocably topple over with crushing force after just one pull too hard or in the wrong direction. Great metaphor.

[@ShawnCZek]

Another issue with monorepos is the access management. Especially with the public available tools, you cannot easily define what directories one should have access to.

[@devnol]

Doesn't github's codeowners do that?

[@qwerty-or1yg]

@@devnol I know that they can limit access to specific repos, but not directories inside a repo. Don't think they can limit your access to specific directories inside a repo (unless I missed and not aware of something)

[@mudscuffer]

@@devnol Codeowners can limit write / merge / PR rights. But not read access.

[@muizzy]

Aside from fringe cases (like fraud rules), why would you want to manage read access?

[@mudscuffer]

@@muizzy For example if you use freelancers or outsourced development for the frontend or for integrations, but don't want them to see your proprietary backend algorithms.

[@saurabhjainwal4604]

We use nx at our firm, it’s really cool and definitely helps maintain our mono repo. 😀

[@middlefloor]

🙌

[@skyhappy]

@@middlefloor Why are you raising your hands? Are you catching a basketball?

[@middlefloor]

@@skyhappy yes!

[@skyhappy]

@@middlefloor well this is a tech channel not a basketball one

[@johngc2010]

@@skyhappy Jeffrey Cross is one of the maintainers of Nx...

[@JonathanAdami]

"You might know him with packages like Formik" Telling me that after 3 weeks of trying to solve issues with fomik and eventually having to refactor my entire code base to use react-hook-forms (which is MUUUUCH better) makes me wanna ditch Turborepo even if I know nothing about it hahaha Cheers for the video!

[@jztsaurabh8111]

Formik has its own issue but it created the whole form library ecosytem

[@codevev]

That's literally what we've been doing in the last couple weeks haha

[@dmitrii_zolotuhin]

Man your channel is pure gold.

[@ultimatum97]

As somone who works at a certain large software company with proprietary source code manager, repository and build tool I can tell you it is no fun at all. It is a nightmare dealing with dependencies when you don't have Maven or Gradle.

[@JChen7]

Sounds like someone has a case of the NIH blues.

[@bobobo1673]

"Maven or Gradle" Look with Gradle I can't comment but with Maven? are you serious? Do you know how complicated it is to simply add a project and the problems it gives looking for its dependencies? Sometimes you have to solve the problems yourself by hand (looking at the dependency tree, and that I don't even talk about projects with Wildfly is a horror) and it makes me lose time from work when all I wanted was to install a library

[@not_nerp]

would this proprietary build tool happen to share a name with a certain south american country

[@ultimatum97]

@@bobobo1673 Be thankful that atleast it downloads the versions you mention in pom. The build system I work with needs a kind of dummy project just to deal with dependencies. It then does some black magic, pulls the wrong jars in a secret location and then shoves it into the cryptic buildpath directly during environment level global build 🤣🤣

[@ultimatum97]

@@not_nerp Maybe....not

[@0xedb]

turborepo is great. I just want a demo where everything is done from scratch instead of the boilerplate to better explain it.

[@AndersonSousa33]

me too

[@kevyyar]

That is my only problem with Fireship. Everything else is great.

[@floffah264]

their docs have a good step by step process of this and their examples help a lot

[@cornheadahh]

I'm loving the frequent uploads

[@thepianist6425]

Would love to see a video covering Micro Frontends. Having said that, always a big fan of the quality of videos. 10/10 college degree material right here. I should start adding fireship io under the education section of my resume now.

[@DominicanRepublicInvestment]

Second this!

[@muizzy]

I work with a micro frontend every day (though admittedly on the backend). Do you have any specific questions? I may be able to answer them.

[@ThiagoVieira91]

@@muizzy Do you think micro frontends improved your development experience in you project? How did micro frontends improved your team workflow? Can you talk about some trade offs in comparison to currently traditional frontend architectures?

[@JChen7]

Watched "Java in 100 seconds". - Proficient in Java

[@essayemyoung4009]

Would check out Jack Herrington’s videos on this (e.g., kzbin.info/www/bejne/onyupJ2mfZVsmrc where he goes through a React example in detail)

[@nathanbrown19]

Worked in both mono and traditional repos. Monorepo removes so many dependency and versioning headaches.

[@archmad]

It can go to an opposite way. 1 app only supported old version while the other uses the new

[@Badadodadoop]

You have no idea how helpful this video is and how much time it saved me. THANK YOU.

[@KevinVandyTech]

I just converted most of my company's front end code to Nx. It's been a game changer

[@middlefloor]

Awesome to hear!

[@andyhall7032]

I like how the narrator sounds like text-to-speech software...it's one of the many appealing aspects of this channel.

[@Viviko]

I usually only use Monorepo on things like Microservice based systems. I kind of feel like having one repository for ALL your products kinda is overkill. But I guess not, after watching this.

[@AhmedKachkach]

What's overkill is going through the effort of maintaining different repos for a small project. If your web app, your mobile app and your data processing jobs all use the same data representation, why not store everything in the same repo and keep everything in sync?

[@clementbowe7594]

I'm probably really late on this but congratulations on reaching 1m subs!! I watch daily and always learn a new thing or two. I hope that you keep making amazing content!

[@devbyemil5201]

You are really doing some great explanations! How about a video about how to plan software and what tools to use? Would love to see it👍

[@pablopoggio4899]

That’d be pretty cool actually

[@kettenbach]

Frikin love you man! It's been 84 years. That was frikin hilarious bro. 💪❤️👍

[@a.c.vermillion]

I was waiting for so long to finally see this uploaded. Thank you!

[@wheytomuchforher]

Serious question. Why would I use this instead of NX's built in CLI tooling to run the "affected" aspects of my code base. I use NX for both of my side projects

[@syamjulio3685]

I'm interested in this as well.

[@middlefloor]

Turbo would probably be used instead of Nx, not alongside it. If you're already using Nx, turbo won't do anything for you that Nx doesn't do.

[@code_react]

you saved my life by explaining how monorepos works. Thank you so much.

[@erkinkurt6799]

I was searching for this to decide if I should use this approach or not. OMG either you can hear me or you have amazing skills to access my search history

[@isaac4386]

I feel like your tracking my Google search wow. Been researching monorepo since the weekend and now you post this 🤣 . Love it

[@PatricioHondagneuRoig]

Your videos are my non-guilty pleasure, I always leave having learned cool stuff and enjoying it, it's like having celery for dessert.

[@alexIVMKD]

Your videos are spectacular, congratulations in advance for 1M

[@junzhengca]

Monorepo coupled with almost perfect IaC is a god send, it gives amazing visibility to everything. Yes it is complex, and the on-boarding learning curve will be steep, but once you are familiar with it, you don’t even need to leave your IDE to find anything.

[@IsaacAlcocer]

You just give me an idea for a personal project, kudos... this is a nice video to watch.

[@anubhavbansal8922]

too much of information in just 10mins that's why I love you

[@wlockuz4467]

After this video, I can confidently add 5 years of experience to "Managing monorepos" on my resume and move my Tic Tac Toe app and Todo app which have nothing in common, under a single monorepo.

[@Filaxsan]

Our man Jeff becoming funnier and funnier (and informative!) every video. Way to go Jeff, love you

[@osagiee9553]

ive been feeling the need for a monorepo for quite some time. this is helpful.

[@krishgarg2806]

Time to use turbo repo for my todo app.

[@nmanikiran]

Please build a course around it (mono repo) with all the tools you have mentioned. take 1 big project and architect the same in all tools like nx, turborepo, lerna, yarn, blaze, webpack module federation etc

[@theatypicaldeveloper]

you should always think big and leave an extra space in your app (if possible) so it's growth-friendly great vid, as always!

[@andreb.1352]

you should always think small, and iterate it to the better.. otherwise you end up in perfection and you need a lot of time instead of just shipping a lot of small parts

[@zyriab5797]

Keep it lean and modular

[@OptrixTV]

i knew this was coming when i saw you put "nice" on Vercel's stream on Turborepo :D

[@yurayurec6071]

Top quality as always 👌 waiting for your video on webpack module federation, another hyped tool on the market

[@dlysele]

Quality video that everyone should watch

[@FilledStacks]

Awesome vid. Good idea to build something like this for flutter as well

[@chinmaykabi]

Melos?

[@FilledStacks]

@@chinmaykabi We use melos but it's not a build tool, it's more accurately defined as a "script coordinator". You're bundling commands for multiple packages under a single command. No caching, no smart rebuilds, etc. It will always execute all commands that you defined.

[@lixou]

Me finishing after 3 days my MonoRepo setup. 2 hours later, this video got uploaded….

[@yodkwtf]

So much information in one video. Huge respect. ❤️

[@conororeilly5492]

I've got 3 apps, possibly a 4th coming if i finish the current project soon all along the same theme, i might try creating a monorepo for them to share some of the common functionality. It's most likely overkill at this stage but each app was to learn something, so i may as well do it and see what i learn

[@DiegoBM]

So two questions mainly. How do those tools help solve the massive git stack of changes and vscode to keep up with them, and do nx and turborepo override lerna? Or is lerna supposed to work in tandem with any of those? From what I can read in turborepo's site it seems like it can do all that lerna does?

[@Balance-8]

you literally read my mind on the video I wanted to see. Incredible!

[@abh1yan]

Great video Jeff, really wanted to see this.

[@badreddine8727]

Thank you for the video I was confused when vercel acquired turbo repo and what it was, this cleared up a lot of things

[@MikePatterson8831]

The weirdestthing for me is seeing big companies modify distributed version control systems so that they operate more like centralized version control systems. All the problems with git scaling is from git's own setup. Really glad I no longer work at one of the big companies mentioned here. Central version control systems manage monorepos so much better in terms of speed and out of the box access management. And this particular company's proprietary build tool had such poor documentation that it was almost tribal knowledge when you needed someone to setup the build pipeline or fix a bug. Great video!

[@sakthisanthosh0103]

You're videos are great. Congratulations for your 1M subscriber mark. 🎉🎉🎉

[@GagandeepSingh1984]

So much to understand in this build flows .. thanks for sharing it..

[@nitroflap]

Funniest thing is that today I was thinking exactly about monorepos & multi repos. Fireship reads our minds.

[@jedcal]

i love this channel

[@AlexEscalante]

I think I am trying Turborepo soon! Thanks for your videos!

[@michaeldausmann6066]

Just finished refactoring my big messy Nuxt + background processes app into a clean and efficient monorepo with turborepo. Should have done this from the start, so much better

[@reed6514]

i'm a php dev with about 10 foss packages, many depending on eachother, and composer (php's dependency manager) works great for me. & I think gitlab has a built-in feature for mono-repos, though idk how feature rich it is.

[@reed6514]

Okay, i have quite a few more foss packages than 10 ... but i estimate only about 10 of them are useful ... i just open source most stuff i write.

[@maxencedc]

Google dev : changes one line of code Piper : here, deal with those 15131 merge conflicts

[@haxney]

You only deal with merge conflicts in those specific files you are modifying. There is (almost) no branching, so you don't have to deal with merge conflicts from other people's commits. If someone changed the API of a library you depend on between when you start writing a change and when you submit it, then you'd have to deal with that, but that's a reason to make frequent, small changes. Ideally, each commit to head should take under a week to write and be only a few hundred lines of code at most. Obviously, that's not always possible, but it's a good starting point. If I make a change to a common library which would break other people, then it's my responsibility to provide a migration path to the new behavior. Since it's a monorepo, I can easily find all of the callers of my library and either fix them myself or ask the owners of those files to fix them.

[@maxencedc]

@@haxney Very interesting, thanks

[@codeaperture]

Let's get fireship to the moon 🚀

[@midas6659]

Best tech KZbinr on the planet yo

[@ThaChillz]

Thanks Jeff, this video was very cache

[@abhinav.robinson]

Amazing,

[@parzh]

2:45 Correction: in monorepos, 'npm run test' works exactly the same as in non-monorepos, i.e., it is not "test everything", but rather "run 'test' command of the root", - if there's no test command, it will fail. To run tests in all workspaces, use 'npm run test --workspaces' or (shorthand) 'npm t -ws'

[@uwuLegacy]

YES finally monorepos are getting more coverage

[@marccawood]

I love the fact that you can download the cache. Oxymoron in action.

[@NezzyLawd]

Been watching for ages, but not subscribed. I don't know how!

[@rayanez]

Awesome video, thanks for the info. Do you know if there's a similar tool for Java codebases? or, I don't know if it makes sense, heterogeneous code bases?

[@middlefloor]

Gradle's probably the closest Java-first tool. Most of the tools in the video can be used for multi-language codebases, but Bazel and Nx have plugins that make it easier to work with different technologies/languages.

[@joerivde]

If you do want to switch to linux, I can recommend Zorin OS (pro version). Switched a year ago, never going back to windows for anything other than gaming or 3D related projects.

[@MikePeiman]

Thanks for sharing, I had a look and this might be my next OS!

[@ZackHoherchak]

Great explanation, but efficiencies aside, I think I still prefer separate repositories for MOST code bases. For some reason it is easier for my brain to keep things separated.

[@ed_iz_ed]

pretty interesting, my mind works a lot better if everything is in one place

[@felixoghina553]

Setting up a monorepo for all my unfinished projects

[@laudijksterhuis]

Video request: Firebase emulator suite 😃 Also, great video, as always 💯

[@igortobert]

0:07 top right corner, Htop and ping. Like a haker in best indian movies))

[@lambrohan]

I knew it coming, since you starred the repo 2 days ago!

[@VolodymyrDovbenko]

That's a good piece of self-criticism regarding using Windows and not using Linux :)

[@albert21994]

Get the Linux DX with remote containers ;) you can even clone the repository to the docker instance to get the speeds

[@WhatIsThis-zq4hk]

why can't we go back to the good old days when programming was actually fun. Programming in 2021 is just spending all your time trying to figure out how to use 15 thousand tools that are supposed to make your life easier but in reality it actually makes your life hell.

[@jrapp654]

Let’s keep in mind it’s probably a better pattern to keep everything decoupled, these companies are just so deep in their ways that changing to something more scalable and decoupled isn’t worth the investmen

[@pateatlau]

t*

[@jonasbroms]

THANKS! Great video! I have to test this now on my monorepos. :D

[@Mmustafa-v4j]

Amazing & concise explanation.

[@sessionswithtemitope]

Thank you for this. It's what's I need right now

[@officialismailshah]

Sir you are just awesome pro pro level knowledge.....

[@MrMeszaros]

Cool - like Gradle! Except that build caching sharing - that sounds like fun

[@toilet_man_G]

Just need this！ Thanks so much for the explanation

[@racecarjonny8460]

I have worked with monorepos built with lerna. OMG are they difficult to maintain!

[@wxrld1]

perfect, now I just need to start a big project. maybe next weekend. yeah.

[@codewizme]

"I blame myself for not using Linux" Linux User excitement level goes up to 1000000% [I am a Linux user too] AS always #1 Content creator Fireship

[@jawshoeadan]

Do devcontainers in 100 secs! 🔥🔥

[@mylifegamer1]

Hey, I used that in creating a blockchain 😃 Thanks

[@wakham11]

1mil on the way!

[@agzapiola]

great video and tools! question: what about the cases where, say, front and backend are written on different languages? thanks!

[@middlefloor]

It's nice to have both backend and frontend in the same repo, so you can have atomic commits when you need to change both. I think all the tools mentioned in the video support building anything, but Bazel and Nx have specific plugins for managing and building .Net, Go, Java, Python, etc.

[@haxney]

Blaze/bazel has build rules for each different language. One mental shift is that with more traditional setups, you have some per-repo config file, so the repository become the unit of configuration. In a giant mono repo, there's basically nothing which is configured at the repo level, so all of your config applies to some specific directory. So you have one directory with a bunch of Java files and another with a bunch of Typescript files. There are different ways those get deployed to live servers, but at a simple level, you're just running "blaze build //path/to/java/project" or "blaze build //path/to/typescript/project", and the build rules handle which compiler to invoke.

[@MercyFromOverwatch2]

Jeff is still my favourite tech youtuber

[@Ultimacho]

I wonder what are the main advantages of using Turborepo against the standard yarn workspaces? We are using the latter, but we will move to a new tool if we see obvious benefits. We only run one app at a time during development, so the process doesn't seem to differ much from the one described in the video.

[@jesusmgw]

Having sub 10 second rebuild times is actually massive for productivity.