How Working At FAANG Makes Your Life Harder | Prime Reacts

[@boredSoloDev]

"I didn't passively suggest, I actively suggested vscode was bad" lmfao

[@natedavidoff668]

i love vs code 😅

[@TurtleKwitty]

Fun fact, the canaries don't actually die, that's why theyre such good bvirds for this. They're EXTREMELY sensitive to passing out but not so much to outright dying in gases so the miners would bring the bird out with them

[@ThePrimeTimeagen]

oh thats neet

[@daltonyon]

I saw this in Globin Slayer, they use the canary for this reason!

[@Kane0123]

I’ve definitely been the canary before

[@t-spark]

​@@Kane0123did you die

[@groupCAP]

the cpu that you said, at 21:49 is the dimensity 1200 by mediatek, and its bad because unlike qualcomm snapdragon mediatek does not provide the source code for any thing. that restricts opensource dev of phones with mediatek chips. you need to get a license to get access to that data, which costs a fukton, but this guy probably happily paid that, so nobody can root or fuck around, because the more you fuck around, the more you find out.....

[@NotAFanMan88]

I love the "production fuckup" stories. I remember when I first worked for a company, I was like "oh this is test, I can play around with it as I feel like". Turns out I ended up sending "test" emails (that weren't labeled as such) to the entire production list of users, because the emailing lists were basically mirrored from production, so people thought they got negative pay stubs. Bosses had a fun time explaining that.

[@mikeha]

yeah you can play around in dev, not test. you risk upsetting business analysts who view the test environment as their territory

[@patricknelson]

Not even at FAANG (maybe adjacent), but so much of this still rings true. So much of it is bespoke out of necessity, _especially_ “at scale”. That simple app you’re working on ain’t so simple once you need to handle 1M concurrent users, be fault tolerant, distributed, fast, highly available (and etc) and yet still be resilient to F-ups from a team of 50+ developers (lol).

[@patricknelson]

@@monad_tcp Indeed. At larger companies (particularly tech companies) you _will_ see much more of that; more time spent on deeper and more specialized challenges. More rounds of review, testing, process and etc will go into issues you might not typically see (or even really care about) vs. what would be on a smaller mom-and-pop style website. For example, I just finished implementing an OSS feature that took me about 3-4 days of dedicated work but, if I wanted to hack it out really quickly (i.e. just getting it to work) without carefully understanding it very thoroughly, smoke testing, validation, then writing unit tests (and in this case rewriting the other unit tests or rewriting some other foundational stuff), it would have taken me maybe only 1-2 days max, or easily half the time. But, yeah - much of that does boil down to good management, particularly time and resource management.

[@blasttrash]

which company are you at that they need 1M concurrent users but the company is not a FAANG? tiktok?

[@BilboSwagginsTheThird]

@@blasttrash So, so, many companies need to do that that aren't part of FAANG. Things like Reddit, Steam, a few games, eBay, Twitter, Snapchat, and more. 1M people is really not that much (compared to KZbin, fb, twitter, Amazon).

[@gerdokurt]

I work in a non software company and develop all kind of solutions for company workflows or specific tasks in huge projects ! My boss isnt a software engineer and when I have a new idea or project, he is always like "cool, thats what I thought too. Do it" 😄 I really love my job because for me, the interessting part in software is the problem solving part and thats what I do like 90% of the time. Im not paid bad and I wouldnt be happy in a big software company!

[@Rafael-vi4to]

Silicon Valley applies the Netflix model both for their users and workers. They want the users to stay in the platform as much as possible. They want the workers to stay at the company as much as possible. More than anything, they want your time.

[@tensor5113]

*Gets fired at 3am in the morning for no absolute reason, company hits record quarter profits, then drives out all senior engineers and cycles through juniors every year*

[@returncode0000]

Thank you for pointing out how hard it is to be a developer in a big company/enterprise. I'm convinced that even 99.9% of developers have no idea what's going on there. That's why I don't like the tech bubble, most of these influencers are the worst with their pseudo-knowledge.

[@Bokto1]

It's not pseudo knowledge, it's just 99% don't care about your "big tech" bubble.

[@neruneri]

@@Bokto1 Not to mention that this bubble has a reputation for carrying themselves in a certain (bad) way. It's kind of similar with Crypto bros and AI bros. Nobody *wants* to understand the bubble when the ambassadors of that bubble have a habit of being insufferable.

[@k98killer]

That story about the backends being cached bytecode sounds scary similar to JDSL.

[@bckends_]

every dystopian/cyberpunk piece of media ever: corporations are exploitative evil overlords of the technological world us: damn, working for amazon really is peak life success

[@dijoxx]

Those two are not really mutually exclusive.

[@ShadoFXPerino]

There's never a need to worry about the lsp not being able to parse the 300gb monorepo if you're free to make a new 'dev tooling' project on a whim and pitch it to management whenever you're bored. That's not to say I'm not guilty of doing that too lmao

[@beofonemind]

nice intro dude, didn't see that coming, nearly spewed coffee all over.

[@JeremyAndersonBoise]

Protip: every situation you encounter will be literally unique in some way. There is no universal solution.

[@code-for-me]

5:46 🤣 Being forced to use vscode and lsp not working is like being handed a Ferrari with bicycle pedals and told to race. made my day, thank you!

[@himanshupandey9303]

Post commit hook to turn phone silent 😂😂😂 Bruh these online folks are legend.

[@nahkh1]

One more thing that makes debugging user issues at big companies difficult: privacy. At a small company you'll probably have a relatively easy time getting access to your production logs or prod database to find out what happened to a particular request. Not so at big companies, where you'll need to jump through way more hoops to get access to that (and for good reason). Additionally the scale makes it hard to even collect and retain logs for that much traffic; many places take a statistical approach to logging to conserve space. It's great for analysing the broader behaviour of the system, but investigating a _specific_ user or request? Not so much.

[@fadhilinjagi1090]

Cloudflare seems to log every request

[@homelessrobot]

The benefit of monorepo is instant consolidation of data and control. If someone needs to gather information across several projects, or make changes across several projects there isn't so much leverpulling and knob twisting because all of the data is right there. Its like if they trained an AI to know how to collection and compose source code repositories that depend on each other, and then shot it in the back of the head and buried it outback because thats just a filesystem(tm).

[@yugioh8810]

so basically everybody owns all parts of the code? what about focused team ownership of the code?

[@homelessrobot]

@@yugioh8810 no, not necessarily. its just that the boundaries of ownership are not dependent on the structure of the repo. You could implement that in the CI/review process; commits outside of some scope require approval/review by someone trusted with the code in that scope

[@cheaterman49]

TBF that's also how knowledge gets lost - if people get sucked into all this abstracted mess right after school, they'll probably never ssh into a machine in their lifetime

[@TheAcademik]

Nah, at Big Tech, half of the day is devoted to ssh-ing into a machine since the laptops they provision for you aren't safe enough to store and ruin the code you write for them. Checkmate

[@cheaterman49]

@@TheAcademik lol IDK about faang but I can totally confirm big companies will provision absolute shite laptops, and give you a VPS you can't ssh into from the outside - or serve files to the public from etc... For real. Plus in my case those morons got hacked roughly a year after we joined, whole company was paralyzed... except us because we decided to roll out our own infra.

[@TheAcademik]

@@cheaterman49 oof that's a rough one. I was just pedantically memeing about your ssh assertion, but I definitely agree with your assertion. FAANG (or MANGA or whatever tf it is these days) also has its version of this. Their tooling tends to be extremely different and unique from what the rest of companies use such that much of the tech that their employees have to use and learn are only applicable within that particular company and, while the concepts remain generally solid, the specific tech is not applicable beyond the company's borders

[@cheaterman49]

@@TheAcademik That part is actually especially terrifying to me - again, think of the junior fresh out of school, getting hired by FAANG (or MANGA as you said, never heard that one hehe), and suffering literal vendor lock-in but applied to their whole career! It's worse than AWS, it's worse than Apple, it really has the potential to impact these people's livelihood long term...

[@davidkennedydev]

2:20 Keep mono repo is way easier to manage overall dependency issues like security updates and compatibility issues, and keeping track of API and data schema changes is also simpler.

[@Cromius771]

Yes but you'll break everyone else's code in the process

[@cornheadahh]

What's an LSP? I've seen it in multiple contexts including for Vim and I've tried searching it up and I still can't figure it out. It's like auto-completion?

[@Razman5]

Seems to be language server protocol? I'm also trying to figure it out and don't know for sure yet.

[@stefnotch]

LSP is indeed "language server protocol". Though it's often used to mean "language server", aka "the piece of compiler-like code that will read your source code, and then do all the smart IDE thingies". As in, a language server is responsible for autocomplete suggestions, tricky syntax highlighting, showing error messages before compiling, etc. And LSP is a somewhat standardized protocol for language servers. The cool part is that it makes it much easier for small IDEs, like Neovim or Helix to properly support all programming languages. They just have to understand the language server protocol, and then they can use all the nice open source language servers.

[@Brunoenribeiro]

Thanks for sharing your mistakes man. Maybe it doesn't seem to be much, but it kinda wrecks people's imposter syndrome. Shows that you can fail. Big time. It's normal, it happens. Knew recently about a engineer that dropped his whole career after a big fail...

[@davidkennedydev]

7:53 Using remote LSP or even a simple SSH remote connection to a loaded LSP with the entire mono repo on a large machine is a reality; however, it creates an entirely new set of inherent waste issues.

[@nouai]

I know that feeling bro. Some time ago i dealt with the system which has ~200 sub-services it was hell (BTW Netflix is one of them). I recently discovered your channel, and now it's background emotional therapy for me, because I lost my emotional intelligence somewhere in the code.

[@realbutters]

6:20 - the ultimate counterattack for being called passive aggressive.

[@MPDR]

I don't really get the LSPs not working at Meta thing. I worked on many projects in different languages and the LSP in the remote VM ran just fine.

[@MrEo89]

There aren’t enough tests you can write to account for all the edge cases that will eventually exist in the wild.

[@pif5023]

Complexity gets you because it’s always factorial in its quantitative dimension. It progresses through combinations.

[@be1tube]

12:15 Canary: bird you kill in an ancient engineering ritual to keep your server from going down. Hail Knuth. Hail Stroustrup. May the integration be continuous. So mote it be.

[@ucarn5246]

I like you! You are the first developer that I have ever agreed with, on so many different subjects that it's almost unreal. Based not only on this video. Who are you.- Old-school shout out!

[@EDToasty]

18:08 imagine using us-west-1 and not the chad us-west-2

[@yehudak1646]

12:42 I do use this pose but only when I'm looking for a critical bug in production that I wrote

[@iammikeDOTorg]

Just finished the final round at Netflix. The culture and your take on it (really just in support of what I’ve read and have been told) is so refreshing. Fingers crossed.

[@EbonySeraphim]

Canary at AWS: continuously tests services in production exactly as a real customer would use it. It essentially uses the public API every 5 minutes or so; and publishes metrics in success or failure, and maybe even latency. If/when it starts to fail, you examine it and understand if other customers are likely experiencing the same failure. The goal is to detect a production failure is happening not because a paying customer complained it was broken, but because the canary failed and you’re already looking. This is important posture for a cloud platform; the provider knows things are down, not calling them up and they’re like “what? It’s not working?”

[@dijoxx]

Canary is not continuous testing. It runs as a first step of a new rollout or configuration change to see if it will break anything.

[@EbonySeraphim]

@@dijoxx if you're an SDE of any level at AWS and you think that, you're not very good at your role, and/or your team isn't very mature from an operational standpoint. At best, what you are thinking of is that the CI/CD process pauses post-deployment for a period to ensure canaries still report success. If canaries start to fail shortly after a deployment, a rollback should happen. A canary doesn't fully verify a deployment because it doesn't know that a deployment is happening at all, and it cannot verify new features until the feature is fully rolled out -- otherwise it will fail. Canaries, unlike normal tests, run 24/7/365 because something a feature or functionality might be all well and good in the code, and pass initial tests, and later it may stop working in a production environment and you need to know it stopped working. That is what a canary is for. Sometimes a deployment is the cause of an outage so it's relevant during deployment as well, but doesn't serve the purpose you mentioned.

[@dijoxx]

@@EbonySeraphim I'm a Sr SRE at a faang company and I can assure you that our teams and processes are quite mature. The whole idea of canary testing is to detect regressions. They work by deploying the new version on a very small subset of production and comparing various metrics to the existing version over a given time period. The specifics of the test vs control environment, metrics, comparison, etc may vary but usually it runs as one of the first steps of a rollout. You really shouldn't opine so strongly on stuff you don't understand that well.

[@TatianaRacheva]

4 years ago, “LSPs” worked and there were no modules (in the web code base; maybe he’s talking about mobile?)

[@petersuvara]

Even on a smaller application, we use multiple repos and also use submodules where necessary.

[@copierofvideos2]

So funny about the posture but he’s absolutely right. I suffer from Neck pain from probably sitting how that guy was sitting for year’s. Now I know to roll my shoulders back and have my head over my body😅

[@jatinchhabra_21]

I used to work with a client, who had mono repos, they used to work on those virtual machines vs code, they are so slow! but faster than running the repo on local though! It used to take around 30 minutes to create a hot-reloadable build for frontend application, debugging was my worst nightmare even on remote.

[@killermenpl9619]

29:30 "Big stuff is hard" That's something a lot of people new to "enterprise" programming don't realize. Serving a video to 10 users is easy, a day of work from start to finish, maybe a week if you want a pretty UI. Serving the same video to 10 milion people requires a whole team working for weeks just to make sure the whole thing doesn't explode at that scale

[@RetroGenetic]

My soul breaking bug was letter that was supposed to be capitalized, took 3 days to find it in the end

[@pif5023]

I have been so tempted to write my own TS lsp with Rust but I don’t wanna revise it at every update. Perhaps I will do it as a learning exercise in my spare time

[@daltonyon]

This type of discussion show the world is more widely than do you think!

[@aghileslounis]

What about the huge program that runs on a rocket to space? Is it 100% well understood and bug free? It's a critical domain, you can't fail. I do think you can create Netflix with close to perfect code, BUT it will cost 100x more and will take way more time to build. It's hard and requires careful planning, rewrites..., so big tech companies take tradeoffs and sacrifice code quality It's just what I think, I really want some Senior engineer in a big tech company to correct me.

[@dijoxx]

As a faang engineer, yes, you are correct. We even consider it a problem when a product is too stable because it raises the users' expectations and causes a greater disappointment when it eventually breaks for whatever reason. You might also be spending more resources on it than is necessary.

[@aghileslounis]

@@dijoxx Holy! I wouldn't think about that waw!

[@tofaa3668]

I like the first point but you gotta give props to jetbrains ide's cause they load up 100's of projects for me just as fine without anything breaking :D

[@rothn2]

What's a GSD? Could not find this on Google

[@adambickford8720]

About a year ago I went from HA, 24/7, proprietary everything, 'change advisory board', etc. to a small app just used by internal users between 9-5 and no PHI/PII. I feel like I'm getting away with murder; similar pay and its an order of magnitude easier and 2 orders more pleasant.

[@eej1.07]

When he started talking about the stock image dev's posture I died

[@A2Kaid]

In FAANG for 5 years, it’s indeed intense having to maintain high performance for most of the year, year after year, but it’s been a life changing and rewarding experience.

[@gadgetboyplaysmc]

I really like the reality check at the end of the video. It always seems easy at first but gets really complex in the end. Looking at big companies at face value as an engineer who can "build stuff", it's probably so easy to say "I can probably build a better version of this". lol

[@jfftck]

It always amazes me that Microsoft is no longer part of the acronym of top tech companies, even though it is at the top of money making companies.

[@gauravsingh1963]

We have monorepos at Microsoft but LSPs work nicely and even linting

[@rob011]

“We go to Galactus” is this an episode of Krazam? 😂

[@jamiebrs1]

I work for one of the companies he mentioned, I've not seen any "monorepos"??

[@elijahachiri]

I learn so much just listening to @ThePrimeTime, how old would I have been before coming across the term “Canary” for the first time. I swear it Would’ve been years and I’m working my first job in Software Development right now

[@stanrock8015]

I work at a large company and a lot of differences. We're an older company that is going thru a big modernization. overall very much agree with this guy.

[@eat120mm]

Prime's killed netflix so many times, we need a compilation of all the stories

[@jfftck]

Time zones are one of the biggest pain in the asses to get right and adding rules based on them is asking for problems.

[@JohnVandivier]

Chronos and Galactis? What is this krazam?

[@DigitalNomadOnFIRE]

"Interesting tech" Yes playing with the shade of blue on a button in Google with a team of 50 devs for 3 years only to can the project before launch 🤡 Super important tech.

[@ahmadaljallad3701]

Does anyone know what a staff engineer does? I don't want to google it!

[@pieflies]

Staff engineers can have different focuses depending on the company and/or specific role. Basically it is just the next seniority level after senior engineer. Usually next, although some companies might have different level orders. It’s always above senior though.

[@epiicSpooky]

Go to metings, write/approve docs, make/update spreadsheets and answer questions (if you're lucky, sometimes the questions are technical)

[@darukutsu]

What means hand-rolled?

[@n4bb12]

"The Netflix UI is so complex very few people fully understand it." But when you discuss HTMX it's "simple things should be simple". UI just very often inherently is not simple. Server-only frameworks artificially limit you in how you can do things and and hence usually are not the options that create the best user experience.

[@SXsoft99]

the netflix UI is so advanced i can remake it in a week max with functionality, 1 day just styling

[@bobbycrosby9765]

@@SXsoft99 incidental complexity. The actual UI is simple, but it's Netflix, you gotta justify that salary.

[@Exilum]

​@@SXsoft99Sure, you'll have no issues on every TV, handle all device-specific issues, deal with hardware limitations and figure out the solution to each device scaling in a different manner? Of course, there will be no issues with platform-specific APIs too. There's what, a few thousands TVs to deal with? What's a few thousands TVs tested and working in a week? Each manufacturer has different platforms and different generations of these platforms. The world of TVs isn't as uniform as the mobile market or the web. And that's without specific UIs that need to be built for a single category, take a UWP app for Windows. You also need region-specific things. Inserting translations is nice, but different languages might require different scales and fonts. Even within the latin alphabet, you don't scale German like you scale English. And you can't just blindly autoscale text, you don't want uneven font sizes from one part of the screen to the next or from one page to another. Things always seem easy when you don't think about the problems.

[@JeremyAndersonBoise]

“I’m always hard.” Solved.

[@interceptorghost1149]

fun fact, nvim stands for netflix vim

[@mhedengran1989]

The comment about his head being too high is actually wrong, the correct ergonomic position, is actually having your eyes around the top of your monitor. Your eyes should look slightly downward, when looking at the center of the monitor

[@markeissler]

So now we know why these faang cos need so many engineers: monorepos. I’ve been doing this for over twenty years and there isn’t ever a good reason to use monorepos. The whole point is to split up the codebase so parts align discretely with the teams that develop the code in them and won’t interfere with every other teams’ code. Keeping dependencies sane is accomplished with contracts and explicit versioning. Don’t they teach this stuff anymore?

[@lakhan4164]

Obviously its not like every team in the company works with the same repo. Its more about aligning verticals in one big repo. Let's say connectors, microservices, utilities, common packages, automation scripts related to a project exist in one repo

[@metatronicx]

Really informative video

[@duanebuell7055]

Can you please randomly mention something about posture in the rest of your streams? I found myself sitting up straight as soon as you pointed it out on that guy!

[@mytechnotalent]

I live for Prime's reactions just sayin.

[@PandemicGameplay]

I just subbed to your channel I didn't realize how bad these tech companies CI/CD process is lol having monorepos sounds absolutely terrible.

[@lakhan4164]

Monorepos is there to ensure any updates to a code consumed by multiple sources are automatically handled with a single commit or merge and then execution of processes. Also helps to separate verticals but that is more about business

[@R0ckBOX]

Working in one of MANGA and crying from comment. And the one which noone mentioned that one team is in US, other in Europe and then smth is maintained by TW team, who have holiday today

[@siya.abc123]

I think that video of that Linux guy leaving vim really broke Prime 😢. He just can't accept that VSCode leads to more productivity everything else being equal

[@Reydriel]

it's not that serious bro lmao

[@siya.abc123]

@@Reydriel it is, buddy. It is that serious. Lmao 😂

[@vaibhav5783]

@@siya.abc123 linux guy is not using vs code, he is using different IDE

[@DouglasRenwick]

The codebase for all of facebook is about half the size of the codebase as the software for a car. (50million half of 100million). Google has 2 billion lines of code for everything they've made.

[@pysof]

No one cares

[@MarcelRobitaille]

To be fair, lsp for c++ is super slow in neovim in my works monorepo. I'm sure I configured something wrong, but I find lsp very obtuse compared to most things in vim.

[@mfc1190]

You sounded like Joe Rogan a lot during this video.

[@PaulZyCZ]

Good posture is important for health. :D

[@johnyepthomi892]

You know….Aviato? Yes, Aviato

[@stanrock8015]

Google has a monorepo but many of the others do not.

[@thefaisman8956]

I liked and I subscribed because you remembered

[@salvadorroibon]

Should we start saying MAANG instead of FAANG?

[@salvadorroibon]

Just heard Prime said MAANG... damm.

[@ZeroBl_]

MANGA

[@andru5054]

Actually it's MAANG now - Mr Beast

[@DarrenJohn10X]

MANGA?

[@andru5054]

even better @@DarrenJohn10X - Mr Beast

[@T1Oracle]

RIP Yohanes Kidane 😔

[@NotherPleb]

Large codebases is 50GB? I work with AOSP and it takes me 650GB or more, I should call it petarepo

[@xakzpc]

Woody Harrelson Crying Money Meme GIF

[@viraatchandra8498]

I heard MATANA (Microsoft, Apple, Tesla, Alphabet, Nvidia, Amazon) is the new FAANG now.

[@LouisDuran]

0:04 LOL

[@raenastra]

It's true that, at scale, things will always be hard. But I wonder about this mindset that no one is able to create something better. Our tools do get better, and it's because of people who are sufficiently frustrated with how things are + have the skills to build something nice. I do think writing services in Go is a huge step up from writing them in C++ (the original problem it was meant to solve), and not in a subjective, "it's more clear to the writer" way, but in a way that's demonstrated by the results people are achieving with it. Maybe we're talking about different things here though - no one's really recreating the whole system like you've said, but incrementally trying to improve the pieces that go into it. Developing tools may be a different consideration than developing large systems.

[@IARRCSim]

When George Hotz quit Twitter, he complained endlessly about how hard the code was. He talked like a developer who never worked on a very large code base before. Do you know if ThePrimeTime reviewed a George Hotz video yet?

[@raenastra]

@@IARRCSim I like Hotz. He's been at Facebook and Google as well, and has said that Facebook has the best code, followed by Google, and then Twitter - I'm not sure that his criticism of Twitter comes from a lack of experience with a large codebase. Anyways, I don't think Prime has reviewed any of his stuff.

[@monad_tcp]

" no one is able to create something better " Whatsapp had the proper mindset. Its not that no one is able to create something better, its just that companies have to have more employees so they appear serious and big. Software structure becomes a copy of organization structure that made it. If you have 400 engineers, that's how you get a complex ball of rubber bands. Want a simple system, keep it small, no more than 40 people. But that's hard to do. The way C-suite works is that they are incentivized to grow their teams, to grow their bonus.

[@MPXVM]

Watchig the video at 1.5x and Prime setting the video in the video at 1.25x, now I have 1.75x speed :D

[@EliteTester]

alright I'm rewatching silicon valley

[@metatronicx]

At this point I want to work at Netflix. Fantastic culture ❤

[@DV31415]

Gsd- German Shepherd dog

[@olafbaeyens8955]

Don't waste your time and energy in trying to work for big companies. Use that time and energy to create your own company. Tens of thousands talented people enough, just allow them to work remotely.

[@Kenbomp]

Banks for the world deal with scale by using cobol. Don't need $300k devs.

[@4ohf]

who is this? you turned off alerts AND you remembered to say like an subscribe? what are you and what have you done with the real Prime 😂

[@bitcode_]

the name ... thehardagen

[@DreanPetruza]

With much better options right there like FANGA or GANAF, I think FAANG is a poor choice.

[@shaymiantewhau3014]

You sound a little like gru when you raise your voice

[@f0rtuna-xy]

Facebook is now meta so it's 'maang' now

[@DarrenJohn10X]

manga

[@sK0pe-d9b]

Monorepo seems a bit strange.

[@Kunaltwts]

Chad level mysoginist 😂

[@LuicMarin]

Working at MAANG

[@velo1337]

100% reproducible bugs... never came across one of those :)