System design basics: Real-time data processing

#realtimedataprocessing #streamprocessing #dataprocessing
#systemdesigntips #systemdesign #computerscience #learnsystemdesign #interviewpreperation #amazoninterview #googleinterview #uberinterview #micrsoftinterview
In this video learn how to process messages or data in real-time.

[@VV-nw4cz]

That is hilarious. "We cannot collect data on one machine" (why?), then immediately we need some queueing, then immediately let's say "kafka". That is so bad that it made me laugh.

[@reelinvestor]

Isn't kafka a pull based system? Meaning , the subscribers need to pull the data rather than a driver pushing data to subscribers

[@anadinum]

Great video. It explains concept of distributing systems without going into technical branding.

[@gsenthilkumar8139]

00:02 Stream processing involves real-time processing of a continuous stream of messages. 01:40 Real-time data processing requires a queue data structure for scalability and reliability. 03:13 Driver orchestrates job execution and task distribution. 04:44 Real-time data processing using micro batching in Spark 06:11 Real-time data processing involves scaling computing power and distributing jobs. 07:42 Implement real-time data processing with message receiving and processing logic 09:03 Real-time data processing flow through a queue structure 10:31 Scaling real-time data processing using partitions in the queue data structures. Crafted by Merlin AI.

[@ramesh4joylife]

Live use case example and a context surrounding messaging queue, logs etc would have made this video much better.. nice try

[@ThePradeep2010]

If I am correct, in Kafka producer need to decide on partition. How can consumer scale if producer is not in our control (we can't control number of partitions from producer)?

[@_ipublic]

5:23 Kafka also have this feature for batching of messages.

[@rishabhjaishwal1247]

I have a scenerio where i have to analyze data on 50-100gb data as of now. And i also have to apply filter on whole data and need a real time filtered result then which approach i have to use. And data also require some calculation in between.

[@shrutikamboj4607]

Going with elastic search/elk stack is the best bet.

[@awesome_arnab]

Hi, I really love your vdos. I have a question. Can you make a vdo which explains about capacity management calculation for vms in a distributed env. Would be very helpful. I mean how do I calculate what should be the server cpu and ram of each server in a distributed env.

[@jerryocrow1]

There are rules to prohibit defining something in terms of itself. That is, you can’t define stream processing as a stream that is processed by processing a stream, in which a stream is used which in turn is then processed.

[@shagunbhardwaj84]

😂 this needs more likes

[@iitgupta2010]

You forget to shuffle and sort part, writing directly from executor to db would cause concurrency issue. Spark do shuffle and sort, to accumulate the data in single piece like collect all 200 as count 200->500 [as single or time based which will be multiple] , then write

[@jokerjussi8562]

nice teaching

[@ajaypatidar]

Thank you for the nice informative videos. i have a request to you, if you can make a system design video on Forex Trading platform

[@ranjithsudhakar9304]

Why do we need a layer before the executors, can't the executors (subscribe) get msg from stream and process it?

[@TechDummiesNarendraL]

It is possible and spark does that too.

[@sp_liberated]

Good job but I think it might better if you don’t go with a solution in mind. In the next videos, try to evolve the solution over few minutes to provide reasons for scale. This way the students learn about the tradeoffs.

[@anadinum]

Do you have any blog expalining these problems and concepts?

[@mayikx]

Thanks for your time and patience teaching it, all your videos are amazing, I hope that soon increase the number of likes or subscriptions

[@hasithaphantom6558]

Which means this somewhat similar to concerrency

[@shivyadav3486]

excellent work narendra

[@nikhilkr123456]

Great video as usual :) Can you please do a video on Google Map design

[@shipra0805]

Thanks a lot Narendra .. Love watching your videos .. Your hard work is our short work :P

[@kevinkamau]

PubSub + Cloud Dataflow + Big table + BigQuery

[@souravsardar]

@techDummies Can you recommend me courses on Kafka with pyspark?

[@dylanwang6818]

can the job processers directly grab tasks from the queue?

[@Jegan977]

Please do a video on backpressure and other type of failures in this architecture

[@theyoutuber273]

You are awsome sir! I like the old school pen and whiteboard explanation

[@srinivastadinada4846]

Best place to ace system design!!

[@jyotis425]

Very nice explanation Naren Sir

[@kanaiyapatel5691]

Narendra, I dont miss yr videos. they are very informative. I have request. can you please make video for system Integration keeping in mind secure integration and data exchange. u may take some different scenarios for integration and talk about solution. I searched a lot but difficult to find good article about integration.

[@puneetpatwari]

Hi @narendra, What is the logic by which a driver forwards a message to one of the many executors? Thanks for the informative video as always.

[@iitgupta2010]

It basically partition the data based on given size. Let say you have 1000 item to distributed and you have 10 executers to distribute. Then in the chunk of 10, assigned to each.

[@puneetpatwari]

@@iitgupta2010 you mean chunk of 100?

[@iitgupta2010]

@@puneetpatwari Yes, sorry for typo

[@ShabnamKhan-cj4zc]

Thanks for explaining the topic in laymen terms..

[@nayaleezy]

6:58 🐈

[@大盗江南]

Thank you Narendra :D

[@jinalkumarpatel5138]

Does any message queue support ordering based on some key ?

[@buddha11585]

Yes exactly ordering semantics is possible on Kafka out of the box.

[@singh07neeraj]

Sir Make video for IRCTC system

[@rahulsinghai3033]

Please talk about getting job abroad

[@drakezen]

Likely the best thing to do is to get into an international company locally from which you can eventually transfer over.

[@rahulsinghai3033]

@@drakezen i am already in a multinational company , I want to know how he has got direct job offer from abroad...

[@drakezen]

@@rahulsinghai3033 Ah, that's another question. :)

[@rahulsinghai3033]

@@drakezen I just want to know about it.. which website to follow and use

[@user-ki9kz6sy3s]

Kafka is a PubSub and not a queue.

[@ujraman]

in a layman terms, Queue is correct. Any Queing system does the same....whether you call it pub/sub or whatever....