Stock Exchange Systems : Distributed Transactions, Financial System

Join the discord community to share more learnings on system design and distributed systems:
/ discord

[@superrin]

the story telling is great. Easy to understand. Thanks a lot!

[@tonyyang8424]

I think those queues design is problematic, if the queue is the normal queue. Instead it should be the priority queue which are sorted by price and timestamp.

[@architguleria989]

That is actually the task of matching engine as per my understanding. It can use redis to maintain sorted sets and do the matching. Queues are primarily introduced to achieve async behaviour.

[@ankitpersie1]

@@architguleria989 agreed.i think the queues should only be used for maintaining the chronological order of placed orders for fairness and downstream system should do the logic of matching.

[@weichli]

Trying to make things clear here - each stock owns its normal queues for storing the incoming order request. In matching engine we should maintain another priority queue based on the price. Each node of the priority queue can further be a sub-queue to maintain the orders in chronological order. The matching engine will pull from the priority queue to see if it can perform a trade or not.

[@luv-nnd]

@@architguleria989 We should use Kafka to store the incoming orders. either buy or sell, so each order will be processed FIFO.

[@tasinpoyraz6679]

As a programmer Love to watch this kind of video excellent work.

[@nikhilbesra]

Great Video actually. I was looking for Stock Exchange design and believe me i have gone through the other videos which have higher views and I can guarantee this has the best content.

[@bhishmaacharya7535]

Wow.. somehow this amazing content found on my feed.

[@sangarvinayak]

Glad you enjoy it!

[@Rafael-yn7fq]

Keep this up! Great videos

[@sangarvinayak]

Thanks!

[@techinbanking6177]

Good effort but lot of things missing like estimation of storage facilities, if the market time is 10 hrs then how come 86400 (24 hrs) for calculation? Order request missing type like Market/Limit Order.

[@vex123]

This is great stuff! Would really love to see how you would translate this system diagram into an Object-oriented software architecture. Can you please make a vide on that?

[@sangamsahai9823]

Bro this is gold !! Thanks sooooo much for making these and please keep making more :)

[@mr_world_wide]

I think the matcher can be designed with using queues, we can simply keep 2 tables for sell and buy orders, each time customer makes a sell or buy request just put the request in a queue, which is then picked up by the matcher. If it is a buy request matcher queries the sell table for the unmatched sell orders, if appropriate order is found, it marks both the records as match in their respective tables then passes the request to executor queue. if it is a sell request matcher queries the buy table for the unmatched buy order and does the same. Adding a the unmatched orders in the queues again will prohibit us from giving fifo service.

[@varshard0]

That's my exact design for an interview and from the feedback, it's not optimal enough with the db overhead. It's better to have the matcher also store trees of buy/sell and match them in memory instead. The buy and sell tables can exist for record keeping and restoring the tree if the matcher die.

[@somghos4363]

Great insightful video!! one design considration. we can use one kafka cluster here . with different topic for each stock list. this kafka can also help in notification and analtyyics. purpose

[@TheKshitijagrawal]

I have my doubts about choosing order request db as NOSQL. 1. I don't believe order request data is going to be in hundreds of TB because this will only store one day worth data - this should less than 1 TB data. Matched orders is where we will maintain the history of orders but then there again we can split that into data for current day and history database. 2. Order request database does require high consistency. Often clients would like to modify their orders which are yet to be matched or cancel these. 3. Lastly, I have my doubt on order_id being the primary query key because trading mostly happen via brokers like icici direct or zerodha so will zerodha poll status for each order_id.. I guess that, they might poll based on their broker id or something which gives info in bulk.

[@meditationdanny701]

Hey Kshitij will try to answer your doubts based on my experience and understanding in the order you have asked 1. Keeping both ongoing order request and past order data in order request table make sense as we aggregate all order related info int o its own collection which makes our data denormalized that avoid unnecessary joins and can have a flag to make active order this will help us in our future API's hence no-sql storage 2. So based on our initial assumption we are letting user update their ongoing order hence data is consistent by nature 3. Yes in matched order there should a broker_id by here the order status user is requesting is abstracted wrt our order request status and yes internally we need to expose a webhook endpoint to get the realtime status and according updating our order request table service coulmn

[@gulhanturk3004]

Interesting video learn more about from this video.

[@goyalpankaj237]

Really Nice Content! One point - There should be one more service that will consume the Matched order Topic messages to debit/credit the actual stocks from the Demat of the Customers.

[@ravipradeep007]

Nice video , just one comment @37.58 when you are discussing about limit orders , imagine there are 1 million limit orders for a stock which is going through a nose dive the matcher system might get overwhelmed , second i am assuming queue is not ordered and say a stocks price is 100 and somebody has placed a limit buy order at 105 at 11.00 am and someone else at 102 11.02 am , but since the matcher is taking them out sequentially it might end up matching 11.00 order since it would be fetched first , to a sell order , if the matcher has not pulled the whole queue , This can be a big bottleneck or more so incorrect behaviour

[@Oyunlog_2011]

Thanks for sharing information and education video.

[@Emir_bs182]

Great concept videos you made.

[@thebks1]

This is amazing and unique content

[@auroshisray9140]

top quality ! thanks bro!!'

[@jaykumarjaviya1941]

what if once the matcher service finds a match and processes it for the executor service? and during that time client cancel his buy/sell order?

[@joelfoit]

Excellent, well thoughout course. Would you have time to do a follow up video on the executor service to expand on how you'd design that part. I know you mentioned the airline reservation ticket but would be nice if you actually went in detail on the executor service here. Thanks again though !

[@AdityaGupta-fz8mr]

Superb

[@sumitraut6987]

Fantastic ☺️

[@sangarvinayak]

Thanks 🤗

[@SunnyKumar-ud9gp]

Loved the content

[@ashutosh_verma_fullstack]

good design, but i think you could have optimized matcher service a bit using, cna u help me with app you are using for this presentation.

[@vardaansangar3568]

Great design

[@ishallwin24]

Won't we need ACID properties for order request table? Given it would be used for order cancelling, money blocking and money refunding.

[@digitalmarketer9675]

You cover more information on your. 43 minutes video this is great to watch

[@Anton-z8l]

Why matching engine runs once in 5 seconds? Why this system can handle 60k rps for every stock, but match them only once in 5 secs? I don't understand. HFT will drop this exchange (imagine if its real) at 6 second and never come back )

[@amriteshsingh2952]

Thanks for the video, very insightful. I am just wondering about the matching service which takes the events from buy and sell queues and matches them. Since the queue maintains the messages in order, and a buy event can match with a sell order which is present in later part of the queue, how do we match the orders in that case here?

[@timo9madrid7]

same question for me. did you figure it out?

[@yaldagh5157]

Thank you

[@marcioduarte2906]

How do you manage consistency between when you get a message from the queue (SQS), have a positive match, send it to the Executor service and then delete the message from SQS? Do you have an intermediate step in the Matcher service in which you synchronously check and update the state of those orders (maybe in the db) before deleting the message from SQS? My concern is: If you have orders Buy A with Sell B matching, you process them, send another message to the queue for the Executor Service and right after you fail to delete those messages from SQS (in the Matcher service) Then retry happens and you get more messages from both Buy and Sell queue. A and B but now also Sell C, this time the algorithm matches A and C, sends it to the Executor service successfully and deletes A and C from SQS successfully. We are not in an inconsistent state, The executor service cannot de-dup on Buy A and we've effectively lost a Sell C order. Thanks

[@perfectalgos9641]

Great questions. I think we should use a persistent storage with 2PC capability. When we successfully notified the Queue then the storage will be settled and fire the event.

[@rabindrapatra7151]

what is the time complexity of matcher service.

[@jacintduduka9137]

The order request db could have been a queue as well, instead of a nosql db? Why did you prefer a nosql over a queue?

[@ankitpersie1]

i think for persistence.Where will the customer see the status and history of their orders.

[@ayushtiwari4686]

What if the matcher service pulls out some data from the queue and goes down, how are we going to update that order

[@ayushtiwari4686]

Queue locks the entry, ignore 😅

[@bhaskarsharan4280]

NSE today has approx 2000 stocks, so what you are proposing 2000 SQS in place?

[@piyush_chomal]

It depends on the buy/sell traffic a stock or "bucket" of stock receives. Idea is to normalise the load on consumers.

[@kl-1183]

greeting's sir, how can i connect you regarding a critical project.

[@DimagKiDahii]

👏👏👏👏👏👏👏👏👏👏👏

[@banjodeiyowun]

what software is the drawing board?

[@namduong6028]

what software do you use, sir?

[@rabindrapatra7151]

why someone will buy from more than sell value. this looks absurd to me.

[@vinayakmajgaonkar6168]

Run video on 1.5 speed thank me later ...🎉

[@piyush_chomal]

I'd say 2x because 2.5x isn't available like VLC.

[@rabindrapatra7151]

10 minutes video becoming 40 minutes.

[@huynhtrung6338]

As a software engineer who are developing systems using Nasdaq exchange stack, i can this video is crap! Nobody nowadays will design an exchange like this.

[@itisyerdad]

Care to go into detail about how it is crap? It's also meant to be a system design interview question so being the most "up-to-date" iteration is not necessarily the goal (especially since not everyone will actually understand how a stock exchange even works). The goal is to have a conversation and see how the interviewee responds.

[@sampatkalyan3103]

Great video.