What I Wish I Had Known Before Scaling Uber to 1000 Services • Matt Ranney • GOTO 2016

This presentation was recorded at GOTO Chicago 2016. #gotocon #gotochgo
gotochgo.com
Matt Ranney - Chief Systems Architect at Uber, Co-founder of Voxer
ABSTRACT
To Keep up with Uber's growth, we've embraced microservices in a big way. This has led to an explosion of new services, crossing over 1,000 production services in early March 2016. Along the way we've learned a lot, and if we had to do it all over again [...]
Download slides and read the full abstract here:
gotocon.com/chicago-2016/pres...
RECOMMENDED BOOKS
Sam Newman • Monolith to Microservices • amzn.to/2Nml96E
Sam Newman • Building Microservices • amzn.to/3dMPbOs
Ronnie Mitra & Irakli Nadareishvili • Microservices: Up and Running• amzn.to/3c4HmmL Mitra, Nadareishvili, McLarty & Amundsen • Microservice Architecture • amzn.to/3fVNAb0
Chris Richardson • Microservices Patterns • amzn.to/2SOnQ7h
Adam Bellemare • Building Event-Driven Microservices • amzn.to/3yoa7TZ
Dave Farley • Continuous Delivery Pipelines • amzn.to/3hjiE51
/ gotochgo
/ gotoconference
#Microservices #ScalingSoftware #scaling
Looking for a unique learning experience?
Attend the next GOTO Conference near you! Get your ticket at gotocon.com
SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily.
kzbin.info...

[@OgnyanDimitrov]

Plus one for admitting a lot of hard truths in such a diplomatic and polite tone. Scaling from 200 to 2000 for a year sounds like a monstrosity to me! - Whoa! Great job! I want to greet the one who was in charge of this situation!

[@RobertWierzbowski]

On rationalizing "using your baby" vs switching to a better supported service, I like the quote on writing by William Faulkner: "You must kill your darlings". We get attached to complex, clever, difficult code, but the best code is dull, simple, unclever, or non-existant.

[@SenorQuichotte]

you've probably never wrote a killer perl one-liner

[@joshpetit]

​@@SenorQuichotte ahaha no one would doubt my 200 character awk one liners are a piece of work.

[@_stokyo_]

The takeaways for me were: Own your uptime Use a common profiling shape that works for all used programming languages Configure distributed tracing Use a common metric dashboard Consistent logging Performance IS important even if it's a small subset of a call chain Stop logging everything

[@cupofkoa]

Honesty is incredibly valuable.

[@dac514]

Step 1) Hire two thousand developers. Step 2) ??? Step 3) Profit.

[@webmastergrumpy]

Great story, great presentation.

[@leozilla]

one of the best presentations about the danger of (micro)services

[@Mike-rt2vp]

@@qm3ster ((((:-))))

[@csvegso]

I fully agree with all the statements. We faced all of these issues. One by one. In the painful ways.

[@revmusical]

Great content , pretty much captures every pitfall of using the microservices

[@jenesuispluslameme]

Great talk from experience with microservices - thanks

[@ChuckJHardy]

Great talk, well done.

[@MargaretLeber]

"Immutability changes everything."

[@PratyushAgarwal]

One of the key takeaways (that I didn’t see in other comments) was the point regarding tradeoffs. Everything that is being decided (small or big) has tradeoffs. Thus, decisions need to be taken deliberately instead of just accepting it as the way things are headed.

[@richardhp77]

Yeah that was cool. Make your trade offs tailored to your organisation so the bad stuff doesn't have such a big impact

[@globetrotter373]

Architecture 101

[@web3tel]

Great presentation

[@fnjordy]

Multiple languages for services are advantageous as it encourages a better separation of the transaction. The challenge is an environment with bias on one language that leads to 2nd tier broken support for the other languages as no one wishes to fix the dominant platform when it errs. Much like RFCs vs. an implementation of a protocol being the reference.

[@KangoV]

Very nice talk. Eclipse allows cross cutting changes. IntelliJ does not. I really miss workspaces.

[@dardanbekteshi3177]

Just realized we have the exact same problems like Uber, only thing that differs is we don't have Uber as a product 😂

[@transparentpolitics]

Good talk. Politics is an "obvious" (to me) of teams owning their *own* uptime -- not uber's uptime as a whole. Suggestion: I have this same problem : you were not looking at the audience and engaging with your eyes. Lift up your head and not down at the monitor all the time.

[@vanhoangtran1270]

thanks for sharing

ge looks so happy!

[@rickardoberg]

I think a majority of the issues brought up could be minimized by using EventSourcing rather than REST as the internal distributed system pattern. A service is responsible for writes, and letting other services consume those through e.g. feeds (using Atom, this would still be RESTful btw). Those dependent services can then project those events into whatever database they need locally, and do not depend on uptime or scalability of the other service. This also minimizes need and impact of migrations, as all databases are just projections of events, and can be rebuilt by simply deleting them (we do this in production all the time, in a rolling fashion). Blue/green deployments is also an option (and VASTLY simpler using EventSourcing) to constantly do major upgrades to services without downtime.

[@MartinMortensen]

I don't even know where to start. But eventaorucing and microservices is by no means less complexity.

[@richardhp77]

@@MartinMortensen event sourcing brings it own demons, but in this particular use-case, based on what uber are trying to achieve with rapid development of decouple features, you could make the argument that event sourcing would be a better trade off than distributed micro services

[@baptistepattyn7555]

@@richardhp77 Event sourcing does not exclude microservices.... on the contrary

[@AlexKavanagh29x]

Politics - succinctly - is the process of deciding who gets what, when and how. (Harold Lasswell) .. whenever there are scarce resources (including people), and people fight over them, politics is what happens.

[@thunderloong]

Politics is what happens when people agree not to kill each other over resources. Scarcity has no validity. Distribution has no validity.

[@richardwalker3760]

Strongly disagree. *Economics* is the process of deciding who gets what, when and how. Politics only matters when the government tries to control the economy.

[@anis.lounis]

Well said!

[@MattMcCormick1]

This is a much better definition. I've heard the Company > Team > Self before and always wonder a) where is the end customer in that equation because they are what really matters and b) there are plenty of cases where the self's decision could go against the company but it would be the right one (ie. illegal behaviour at the company)

[@tauruscolvin]

Insightful talk -- thank you. Regarding Matt's question: 'How can we safely insert the "ride with a puppy" step into the Uber architecture?'. Perhaps a dab of functional programming? That is, we assert that each microservice should be a composition of (pure) functions -- aka "pipe". These "micro-pipes", would give us the *synchronous* composability we want while preserving *asynchronicity* in the space between the services achieved through event sourcing. Google seems to call this pattern "functional microservices".

[@ArchimedesTrajano]

43:29 I go with the notion that politics specifically "playing politics" is the notion of utilizing people to achieve a goal but making it so that the people that are important to you are satisfied. Bad politics reduces the amount of people being satisfied, good politics conversely increase the number of people satisfied.

[@TheBookDoctor]

Great talk. Interesting how many of the problems you discuss really do boil down to the "politics" category of human problems: emotional sunk-costs arguments for sticking with the known-but-broken vs. taking the cost of switching to the new-but-not-broken. Or as Douglas Adams said, "People are a problem."

[@mennovanlavieren3885]

29:37 Ah, ORM induced performance problems.

[@rftluong]

Great talk ... the number of ads in it are ridiculous though.

[@MarshallSorenson]

tldr; like, we thought micro-services meant nano-services, so like, that's bad.

[@vincentmax4571]

Cool stuff

[@Skorps1811]

That's a COVID-like growth of microservices there

[@HoldOffHunger]

"Don't bring out the sticks. Carrots are the only way to go." Nailed it.

[@RayBrooks0]

Matt, such a great presentation! Also, you have tape on your face.

[@omarmoataz]

I watched 21 minutes of this talk and I still can't figure out if I learned anything.

[@_stokyo_]

Can someone explain what cross language context propagation means? He talks about passing context down to intermediary calls but I don't grasp what the alternative is?

[@HaithamElGhareeb]

One of the most interesting Microservices sessions I ever saw at all. Thanks a lot.

[@eugeneleroux6140]

Lots to think about, thanks.

[@jabberl.]

node, go, java, python - "Wow, that's like a lot of languages!" LMAO

[@reespozzi4334]

It is. Less the better

[@bigdlamz]

The is what happens when you embrace microservices with all its buzzwordy glory. Some decisions need to be centralized eg decomposition & language to use.

[@Pjblabla2]

Very informative It was unusual to see someone with the chief systems architect role in modern IT organisation It'll be good to get an understanding of different roles within Uber's IT organisation and how it is structured E.g. are these just his observations in retrospect or does he have the authority to influence future decisions based on these insights

[@Silmarieni1]

Isn"t a bunch of criticisms of microservices in fact a failure to choose the right size for a microservice ?

[@TheNiters]

10x growth in engineers in 1 year is probably too fast... but might be hard to avoid

[@pkeshava]

Pardon my ignorance. What is WIWIK?

[@dn5426]

"What I Wish I Knew"

[@boltonkim7090]

My thoughts after watching this video: 1. Are engineering teams at Uber comprised only of junior engineers and fresh grads? or 2. Maybe Uber doesn't have much of an engineering management culture? or 3. Maybe Uber's product organization wants things moving so fast that the engineering organization has to throw common sense away?

[@Divinii]

10:55 Wow! *"By having multiple languages, it can fragment the culture."*

[@rentsy3444]

don't do it

[@Divinii]

@@rentsy3444 Thanks!

[@tsingylu]

I can feel his frustration

[@n8style]

"It's so hard to get a count of how many services are running because it's changing so rapidly" Wait what, you don't have an automatic count you do it manually?!?!

[@waleedjubeh5955]

When I hear "Buying new computers (scaling) is cheaper than hiring software engineers." I have shocked about this fact. Bom!

[@chashdeveloper]

He is an excellent presenter. I have attended one of his talk in 2015

[@saidimuapale]

Subtext of this talk: the Uber Engineering mega-team is fairly dysfunctional.

[@JulienCROUZET]

Plot twist : After reading a comment on KZbin from a random guy with no real experience of high traffic they decided to change everything because I thinked that they're "fairly dysfunctional"

[@jerrygreenest]

Double Plot Twist: After they noticed another guy used the word “thinked” in his arguments, they reconsidered what the else random guy is saying, and indeed fired half of people

[@-Jason-L]

Approaching microservices like RPC is why you will have so many issues. It's a mistake made by folks rooted in older distributed system approaches.

[@tux1968]

like

[@Beebop121]

I really dodged a bullet staying away from Silicon Valley.

[@charlessharpe2121]

Why not use an event-driven, non-blocking I/O to speed up REPO synchronization, aka nodejs? Is the company focus dependent upon centralized or decentralized philosophies?

[@christopherstamp9716]

Charles B. Sharpe it's not that easy. They obviously use node js but node js is single threaded and can run into similar scaling problems as any language. The complication is concurrency/caching and cpu metrics/resource management. Very complicated environment

[@tscan89]

Seems like they need a new microservice to coordinate all their microservices. :)

[@sahild6584]

Tim Scanlin nice sarcasm

[@Calphool222]

Actually, Netflix has a presentation where they talk about that. Even when you're doing a microservice based architecture, you end up having some services that are far more critical than others. Aspects of monolithic systems begin to re-emerge. In monoliths we addressed these "critical components" with various high availability tricks. Netflix took a slightly different approach. They built Hystrix to provide systemic fallback and short-circuiting of cascade failures, which are an emergent property of a microservice based architecture.

[@hariveturi4193]

@@Calphool222 I happened to stumble onto this talk after the one Netflix one :) I know it has been a long time since this comment but still. Really loved the netflix talk. Wish they would've done QA later.

[@minyakonga8897]

best have so many defs, like best libs, best cuz i know it, or best ....

[@beedeeuniko]

the frequency of him saying "like", "kinda", "sort of", "totally" and "you know" is - like - staggering, you know?

[@Skorps1811]

is like kinda sort of totally staggering, you know?

[@georgesmith3022]

So somebody discovered distributed systems again, called them microservices, but nobody seems to know how to make them. Is it another fad that will go away?

[@electroman7396]

Uber does no profit and just pumped up and hyped. Why is everyone taking it as an example?

[@nickbarton3191]

I'm surprised that he's not a jibbering wreck of shredded nerves.

[@DodaGarcia]

He kind of is though

[@minyakonga8897]

everything is a tradeoff, everything is a choice

[@ahmedbadia1485]

this guy talks like Richard from silcion valley, guess the show was accurate after all

[@billmoney1]

sup

[@szym1]

like like like like like like I can't bear it over 13:25

[@dieterplaetinck4431]

me too. i gave up on this presentation. I couldn't stand the "street" talk. like like you know like

[@jake8217]

All of these sounds like Uber problems than microservices problems.

[@dotanoob466]

1. break laws 2. have a toxic culture 3. get a-holes to invest in an app made by ppl who end up doing all the work but getting no credit.

[@kobac8207]

Thanks for this presentation and for a completely unexpected reason. I thought that guys doing architecture at Uber actually know to do architecture. Who the fuck scales to 1k microservices in a year??? Do they even have a clue how and when to separate a microservice? Reason for a new microservice is a new bounded context. How in the hell did you manage to find 1k bounded context in the app that routes the vehicles for taxi rides? Uber is a distributed ball of mud. This guy is an antipattern himself.

[@olegzandrvondenmanoresoftw596]

How many "likes" can you say in a minute lulz

[@m424t]

This was a very disappointing video. Unfortunately it seems like Uber architecture was designed without much thought in a way junior devs would approach it. Given the very basic nature of problems they're having, I am surprised that Matt hasn't mentioned problems with transaction support and REST API versioning.

[@mensaswede4028]

So Uber is a great business idea, but as usual it was poorly implemented. But it succeed in spite of its poor implementation because it was a great business idea. The world is not a meritocracy.

[@Teslawaverunner]

Interesting how people have one personality during structured presentations and another when asked questions: the way the presenter dealt with the questions at the end was rude and condescending and probably gives an insight into the culture present at Uber

[@anujpandey8753]

too high level and too abstract. Touching so many things at a time and not diving into details. This presentation could have been better.

[@nelsonc5339]

Wow...! Are the *investors* watching...?! this is _like_ a *Technical Debt Time Bomb* of a "stack"! Uber is going to *implode* so spectacularly... 😧

[@DavinStewart]

Like, please stop saying like, like all the time.

[@cantwaittowatch]

rambling so many things..yikes!

[@hoopssheaffer]

This guy doesn't seem like a very good engineer.

[@stoneshou]

previous CTO of Voxer

[@Calphool222]

Huh? He's pulling back the curtain and describing the difficult problems Uber is facing, most of which were emergent properties of being engineering purists. The ignorance of Conway's Law and a hundred other systems thinking phenomena in engineering circles is epidemic. Code slinging is an important baseline skill that everyone needs to have, but it's *not* what it takes to lead armies of technical people, and having intentionally ignorant HR managers is *not* a solution.

[@tj8870]

It's the complete opposite. This is how a great engineer thinks. Seeing the trade-off between all the different types of methodology, technology and architectures. It is impossible to design a fully working system at a large scale. Every layer you add increase the complexity of the project exponential not matter what architecture you use or how great your starting UML was. Trade-offs are constantly added to the ones you were already facing. On top of that you are building off the previous architecture. Any flaw in the system or outdated methods or technologies only get propagated with growth. Small scale architecture is ridiculously simple relative to large scale architecture. Personally, I often find that maintaining an MVC project with 1 or 2 teammates is more time consuming and produces worse results then just coding it by myself and they are handling 2000 coders... it is crazy...

[@MrHics]

regular ads? downvoted

[@suckerformimi]

This conference is crap

[@SC1240]

This dude can't present well at all. Decent info though.

[@ralfoeldi]

After 15 minutes... why isn't this guy sooo embarrassed that he crawls under a rock and hides? THIS guy is responsible for anything? Sounds like a junior engineer after having completed his first project (which might be close to the mark).

[@temaz3334]

Sounds like Matt proud a lot from 3:25 . Probably, that chaos + a big pressure as a result of that become the reason of Uber engineer suicide..

[@SagdiyevOfficial]

like

[@mayo2001]

like a like

[@atulmy]

like a like a like

[@Sick_Pencil]

like a like a like a like

[@neerajshukla]

I like you, do you like me?