Modern Banking in 1500 Microservices

InfoQ Dev Summit Boston, a two-day conference of actionable advice from senior software developers hosted by InfoQ, will take place on June 24-25, 2024 Boston, Massachusetts.
Deep-dive into 20+ talks from senior software developers over 2 days with parallel breakout sessions. Clarify your immediate dev priorities and get practical advice to make development decisions easier and less risky.
Register now: bit.ly/47tNEWv
----------------------------------------------------------------------------------------------------------------
Video with transcript included: bit.ly/3n2BSMX
Matt Heath and Suhail Patel explain how Monzo team builds, operates, observes and maintains the banking infrastructure. They talk about how they compose microservices to add new functionality, Monzo’s culture, deployment and incident tooling, monitoring practices and how they share knowledge effectively.
This presentation was recorded at QCon London 2020: bit.ly/2VfRldq
#Microservices #Banking #Monzo

[@preetham524]

Dev : declares a string in code Manager : create a microservice for this. Dev : This is useless Manager : Just do it Dev : StringService

[@AlwynSchoeman]

I used to hate my monolith online banking platform, but it was due to ignored technical debt. I see a place for microservices, but I would gladly take my monolith over 1600 of them any day.

[@AgreementMaker]

Having worked with monolith applications before, operation (wrt availability) and scalability was hard. That's what microservices hope to fix, but to fix the other problems they introduce you have to build what the presenters point out: container orchestration, distributed tracing, aggregated logging, uniform and ubiquitous application metrics, and a monorepo codebase... The cost of these things start to pay off at a certain scale, but definitely not worth paying in every case.

[@DustinDeus]

1600 services and almost zero words about: - deployment - service-discovery - versioning - scaling - documentation - dependency management - chaos engineering 😭

[@bennewton]

and yet stability and performance is going pretty darn well.

[@eniton]

the number of microservices should not be a proud thing to exaggerate

[@marcelsolera6878]

Yup, it's very stupid

[@pateastwood]

There are 2 things that are easy to do. Create a system with one service and create a system with an infinite number of services. Finding the balance between the two is the tricky part.

[@JamieRossUK]

I hope you created a micro service to post that comment

[@_programming_]

How do we reached 1600 microservice? - We overrides the new operator, so every time you create a new Object, we spawn a new microservice to store the Object information in our nice Cassandra Database and expose a nice RPC API to handle the functions. We call the concept: "Microservice Oriented Programming" MOP.

[@fancifulconcept]

calm down Satan

[@SaifUlIslam-di5xv]

This makes me feel so horrible just by reading it.

[@aaronmachin3524]

Working for Monzo be like: This code is over 10 lines long... it should be a microservice!

[@CharlesGodfrey]

Comment section, in a large microservice ecosystem, how are you guys handling: Authentication? - do large number of microservices all share the same security roles? Does each microservice verify roles or do they make assumptions that they are internal only? Deployments and testing? - if 10% of your microservices change on a weekend, are we testing just those microservices? Or are we also testing all products which use those microservices? Versioning? realistically speaking are these microservices versioned or do they really just make lots of changes that are backwards compatible? Products? If microservice C creates a new version which now takes full date of birth instead of age, just as an example. How are we troubleshooting issues when microservice A still calls V1, and microservice B calls V2 of microservice C? Total transaction time? What's the average number of microservice calls or network calls that occur for your requests? Repos? - are people using mono repos or does each microservice have its own? With the right tooling, does it even matter these days?

[@brutesque]

I thought this video would be interesting. But actually the comments are most interesting.

[@rotimibest8530]

hahahah, exactly. I haven't even watched the video but I have gotten full entertainment from the comments

[@littlewildtube]

I think this implementation example is one of those "microservice architecture" where the persistent data store is shared between the 1600 microservices.

[@abdulkaderjeelani]

Yeah, essentially microservices demand for own independent schema not strictly always a own database.

[@supertren]

KISS methodology , right? XD

[@NickWareing]

So that's 1 microservice for every 2,500 customers...

[@ancbi]

Gotta be flexible and ready to scale, man. Now, if only there would be more customers...

[@orlovskyconsultinggbr2849]

This is madness

[@williamdavies1977]

This . . . is . . . SPARTA!!!

[@orlovskyconsultinggbr2849]

@@williamdavies1977 well played man , well played ......

[@alex_chugaev]

IT industry needs architects to make simple things really complex and call it "architecture", but in fact everyone tries to get piece of big pie while it's warm.

[@unicodefox]

I think we figured out why the Monzo app is very slow loading.

[@parsapourali7315]

Nano-Services is the right term.

[@kamalhm-dev]

Number of microservices aside, I really think they have a solid engineering practice there

[@AmithBelur]

Having such granular services, how do they manage distributed transactions. I thought transactions usually define service boundaries. Secondly, what is the cost of network latencies for time-sensitive operations? Making so many network calls, how reliable is the application?

[@landonbaker3951]

For microservices, distributed transactions should be avoided at all costs. They typically use what's called "Sagas" instead. Basically, it's a series of transactions that can be rolled back. Network calls also should be kept to a minimum. For example, when doing a GET, if the data being requested is sourced from multiple services, there would usually be a service that acts as a VIEW to that data, and it would be updated based on subscribing to the sourced services events.

[@AmithBelur]

landon baker the distributed saga I guess is similar to a 2 PC where you have an intermediate state before commit right. If there is a way to deal with it, what is the case against it?

[@landonbaker3951]

@@AmithBelur There are a couple of reasons to choose a saga over 2-phase commit. 1.) Each microservice should have it's own data store, and each microservice should have the ability to choose different data stores. For example, one microservice may use MongoDB while another uses SQL, but if 2 services use MongoDB, then it should be different database instances for each service. 2.) 2 phase commit locks the objects for the entire commit while a saga only locks the object during its short local transaction. This is mostly a problem because the services exist over the network, so each additional resource would add some overhead. So a 2PC could become a bottleneck pretty quickly when needing to lock resources from multiple services many times per second. Additionally, 2 phase commits can potentially lead to deadlocks, while Sagas won't. With that being said, with everything in the Microservices vs Monoliths debate, it adds complexity that must be planned for. if it is not done well, you'll wish you had the 2PC/monolith, but if it's done well, it'll be better.

[@gigiduru125]

@@landonbaker3951 as far as I understand, the problem with sagas is that the dbs can still be in an inconsistent state for a short period of time between when all the data is commited or rolled back, so there seem to be some tradeoffs. 2pc transaction coordinator should be more consistent. Why not use something like cockroachdb, spanner, faunadb, yugabytedb, tidb instead, as they keep the db at serializable isolation and you don't have to think about it as long as it keeps working ofc 😂

[@agustinbs]

Perhaps they did not clarify the flows or verticals boundaries between those services, because it seems something went horrible wrong with that design.

[@peco3221]

I thank you so much for this video. This gives me a realistic perception of how much code and time it takes to build something of this scale. It was also very informational and a HUGE guidance on how to start building these evolving massive companies. Thank you very much!!!

[@bkjenner]

I reviewed this in quite a bit of detail. When you watch the video you are left with the impression that they have created this incredibly flexible architecture that is extremely efficient. When you dig into the comments you find they have 1551 employees in of which half are developers. That is an amazing number of developers and does not imply efficiency at all. When you check their financial statement you see they lost 130 million in 2021 and are heavily dependent on investor money and are at risk of failing completely. (check out video "Monzo in trouble") I am mostly interested in it from a technology perspective though. It makes me wonder when you create a banking system based on Cassandra which is not ACID compliant whether this contributes to the issues. As a result of not being ACID compliant, it forces developers to make up for it in code and do things like transaction processing (lock, rollback and commit). On a side note, I am confused by the response when the presenter is asked about Cassandra not being ACID compliant, and he says there is a common misconception that banks must be ACID compliant. (44:00)

[@abhaykatiyar3539]

Yaa keen observation

[@hhellohhello]

lol.. did every developer in the company created their own service?

[@rkalla]

😂🤣😂

[@dmitrit5488]

It's onboarding task for newcomers

[@miketyson7274]

According to wikipedia, Monzo have 1,551 employees... so yeah 1 micro service per employee lol

[@EduardCB]

@@miketyson7274 1551 employees doesn't mean 1551 developers. I would say that at most half of them are devs. So that gives you at least two microservices per developer 😂

[@miketyson7274]

@@EduardCB Thinking about it over night, if the 2 micro services I have responsibility for are ~2000 lines of code this doesn't sound as bad as it looks. Reality might be more complex tho.

[@vetiarvind]

This sounds crazy to me tbh. 1600 services?! Is it skynet?

[@solitudebronx59]

where can I see that speech he mentioned "Support at the speed of thought" by Sophie ?

[@MCroppered]

Upward inflection overload.

[@defco89]

That's it !!! I couldn't figure out why there was a strong urge to tune him out. I guess it's hard for his colleagues to critique that. Hopefully he reads the comments. 😏

[@claudeb.3473]

Just when you thought the Valley Girl up talk was annoying, in comes this guy adding a British accent to it.

[@brianevans4]

I wanna know if each of their microservices has its own instance of cassandra, or if they all share one massive database??

[@privettoli]

So basically they made autonomous components as separate deployments and call it micro-services.

[@miketyson7274]

What worry me the most is the shitstorm that must be their network... 1600 services all doing api calls to each other. but I guess the network is provided by amazon in this case ?

[@sbylk99]

Hey I am interested in how the microservices calling graph is generated? Are there any tools ?

[@rogierdikkes]

Only tool I know is called Vizceral, I doubt they used it for this overview.

[@GSazheniuk]

You should have information about which service calls which in the logs. And then you can use visualization tools or libraries like d3js to make simple Graph chart - observablehq.com/@bstaats/graph-visualization-introduction

[@dm2060]

AppDynamics and DynaTrace

[@littlewildtube]

Weave Scope if you have k8s, but watch out for the recent vulnerability.

[@PeterZeller1]

The combination of locking and cassandra sounds like trouble.

[@tes11vcsw]

Excellent!!!

[@gigiduru125]

1000 lines per service... so basically a microservice for each class?

[@BradleyWeston92]

That 1,000 lines doesn't include the infrastructure/dependencies they use for communicating with each other, security etc. Also they use Go, the good thing about Go is you just write the business logic and change as you need to, not writing code that needs to be abstract, needing big factories or complex just straight to the point.

[@gigiduru125]

@@BradleyWeston92 yeah, but if you just write a monolith you don't need the communication libraries. Also it is faster. We are in year 2021, I want to be abstract, I want the computer to work for me, not write C like boomers.

[@gigiduru125]

@Adithya R yeah well guess what linus torvalds builds monoliths not shitty 1000 lines microservices

[@dsmyify]

@@gigiduru125 ~ 'not write C like boomers'... how disrespectful to those that paved the path you're now walking.

[@gigiduru125]

@Marc Green the same thing that happens when you have to change code in 100 microservices to implement 1 feature. You just write your monolith with well designed interfaces between the modules, like you have well defined api definitions for your microservices. The separation between the microservices is not real, they are still just procedure/method calls or more generally messages, but remote in this case. Ofc I am not advocating for always using the same language or programming model if it doesn't make sense, just saying that writing a microservice for 1-3 classes is stupid and everyone who does this needs to stop or the whole software industry will turn to shit. Only separate them on bounded contexts/transaction boundaries and if you need to have separate databases for each microservice. Distributed transactions are nasty, you are basically building a broken database with your microservices as stored procedures.

[@parthabiswas7771]

I have only one word to say 'Madness'.

[@CharlesRay87]

More doesn’t mean better

[@CharlesRay87]

@Marc Green complexity does not disappear, it is shifted somewhere. You have to maintain hundreds of services with hundreds of teams which have to communicate etc. Maintaining SLA and security is much harder when number of services grows.

[@ordinarygg]

@Jonah Meijers security inside app management itself can't be abstracted from any service that uses any permission model

[@ordinarygg]

@Jonah Meijers it must use API not "imported", when you reuse code, you create coupling and it called distributed monolith

[@vetiarvind]

@@mg00 i get micro services as a concept but it should be a higher level abstraction, not abstract every little thing into it's own service.

[@mikeha]

What I want to know is how do you go from knowing nothing about an industry to being able to build a system for that industry?

[@juliarekamie]

Great speech , tnx 🔥🌞

[@sofyanhadiahmad]

I'm still thinking about this, are they really need that many services? I believe no

[@KangJangkrik]

I don't think it is a good idea to prioritize quantity over quality in this case

[@TheUmnez]

If you improve one microservice with one purpose, how does that replace other microservices (decrease quantity) with different purposes?

[@KangJangkrik]

@@TheUmnez depends, if it's modify the API schema, then it's going to be a big chore

[@FaizalHeesyam]

its like addiction now...

[@carveratutube]

Imho, if you have such a complexed architecture your domain must be quite simple.

[@danielschmider5069]

wanna change the system? just add a service.

[@user-ss3ej3xk8g]

Open-Closed principle

[@martinmogusu]

Wow. That's impressive!!!

[@troooooper100]

LOL here is why this is bad idea... what's a microservice (ms). basically think of functions but on their own servers and to call them you need to call a url. so every function is behind a http server theoretically making it very scalable but coding is 1. Most code is synchronous - once transaction is verified then transfer funds) any point in making them a separate ms if 2nd part is stuck 1-to-1. 2. Conditional code - if this happens, do that, if not then this, but finally do this. A logic that could fit in 10 lines of code, but if they are in ms you are really in the world of hurt....you can emulate if this then that... by having service A call service B, but what about finally..Service C needs to be called once data from both A and B have been returned. To do that you call them in reverse. Service D would call A and C then wait for them. Complexity increases and if A calls B then it has error, now you are going propagate that error all the way to D. 3. Debugging - do you like your debugger have control over entire program's state and step through it one by one.. lol. debugger can't do that if all of your code is on different server. To debug, you have to provide mock state for every possible state that your microservices upstream can give to a particular service and if you dont find an error here, keep moving. now do that for every function. Oh you want actual data, there is no stack trace.. go read over 1000000 lines of logs 4. Global state, often functions have access to global state, that is shared by many functions, there is none of that, you either do read operation on DB or pass along bloated objects. 5. Traffic / http - before unless cpu crashed you could expect abc() to always run. Now you have to worry about http request failing, or timing out due to severs starting or dying, load balancing and plain traffic congestion. So you have to implement retries, throttling, queues, events systems. before you could have done abc(). 6. Code / Business requirements change - as a result now you have either 30 versions of same function purely because other code relies on it and to modify it you'd have to modify everything. It's classic problem in all code bases, but here refactoring is impossible as it'd take 10x longer. Often microservices are under different teams so good lucking opening requests tickets and telling them EXACTLY what you want, never mind that development is trial and error process, but surely you wont waste other time so tell them what you want, wait a week, and then cry if you need to change it again.

[@VivekYadav-ds8oz]

Also using microservices is basically using stringly-typed data. Atleast in monoliths a lot of errors of using the wrong API, sending only year instead of FullDateFormat would be caught by the static type checker.

[@troooooper100]

@@VivekYadav-ds8oz honestly I think a compiler should exist that takes CPU heavy code paths and automatically spawn that code onto new machines and then instead of using running code locally complier could make rpc calls

[@yajnasteju]

Amazing !! That's every developers dream to build such systems... Certainly mine too ❤️

[@bennewton]

so brilliant. All these years later and their *still* ahead of many banks.

[@shivam-cc5ts]

how tf do you even think of 1500 different microservices to make?

[@VictorMartinez-zf6dt]

Having multiple teams and problems

[@sizwesokopo281]

Homie in the back nodding away is suuuuuper distracting

[@zoomkl9136]

What nonsense it is

[@Y2Kvids]

what's a Microservice ?

[@adityak2988]

1600 microservices doing what? What bank needs so many services? Example of overuse if Microservices

[@mx2000]

I'd be more interested in their use of Go, than their microservice architecture. 1000 lines per service is basically not possible with Java due to the enormous amount of boilerplate code that Java requires. It feels like they are successful despite their big ball of mud architecture, because of their use of Go.

[@Smaylik03]

Use Kotlin, no boilerplate. Less than in Go.

[@paulfriedrichschulz1964]

Their big ball of mud won't be a problem for a few years. If they want to change things, they will have big problems.

[@jeremydone1982]

MICROSERVICE ORIENTED PROGRAMMING 🤣

[@convitcrazy]

Too much microservice .i think 20-30 microservice is enough cover business of banking.

[@Ryan-xq3kl]

Banking? Lol more like scamming

[@brokula1312]

Go? I want Javascript to handle my money! :D

[@pas.]

Go and Cassandra. Error handling is optional. Default read/write consistency level is ONE (or at least used to be).

[@edenassos]

Those slides are worse than what an elementary school student can come up with.

[@xychromosome6510]

micro micro micro CRAP