Google Docs System design | Part 1| Operational transformation | differentail synchronisation

Google docs system design part 1
Includes explanation for operational transformation and differential synchronization.
system design diagram: imgur.com/a/4w...

[@isaaczhu9223]

IMHO, this video is less smooth than others and not as clear, either. Anw, it contains some complex topics where each may take a whole video itself to explain. It'd be great to explain OT and DS in more details separately.

[@djcool994]

He copied all other videos from Grokking System Design course. This is one of those design question that is not present there, hence the quality. :)

[@athenanouhi]

@@djcool994 disagree with you! I have completed Grokking System Design course and don't believe Narendra's videos are copy of that course. His contents are very informative and is a different source than Grokking System Design course. He does a lot of studies from companies blogs and reliable resources to make these videos.

[@kaustabpaul4514]

why dont use timestamp of the operation and send it over a priorityqueue or sorted queue sorted based on timestamp?

[@WoodyLPG]

I do not 100% agree with the statement that 'lock can ever be a solution to collaborative editing', it eventually comes down to the use-case / requirement you are designing against. Quip is a perfect example of using lock for concurrent editing, it locks down a row / code block when someone is editing. Others can only acquire the lock by explicitly requesting a control from the lock holder. It works in quip because there is probably a really rare case where there will be 5+ people editing the same row at the same time in an enterprise. Maybe quip uses a different algo in a different pricing tier, but it works perfectly at my company (10k+ people worldwide). Gdoc might be designed for a more general case where there can be many more editor at same time, so they use some different algorithm

[@BhanukaSiriwardhana]

exactly. A lock per raw/line could work I guess

[@dipendupaul]

Good point

[@Mahesh-js6hp]

How do you know what is the operation that corresponds to a` from a?

[@baothai9715]

From what I understood at a high level thinking. The idea is really boils down to a' (a prime) and b' (b prime) in mathematics are represented as CHANGES of these operations. Like how you calculate for velocity/prime in math, it is the change from the old state to the new state. With his example - AT right? If there was an a with Deletion(T, 1), and b with Insertion(H, 0) -> a' should calculate for the difference of what is AT deletion is (T deletion after A is a' let's say), and b' should calculate with the insertion of H into AT is (H -> AT is an insertion before A, let's say is b'). And at the end a would combine with b' operation, and b would combine with a' operation hence resulting in a HA instead of HA|HT. The exactly logic behind it, I am not for sure, but I would suspect some kind of edit distance algorithm with some wildcard for transformation. But that is the high level overview.

[@anujvishwakarma5158]

Please try to clear the output voice. Sometimes I just miss some important words.

[@vishakhasangtani3354]

Love your content! It's really helpful for the people prepping for interviews, information in such a concise way is not available on many channels/platforms.

[@Kdsiitd]

Can you please also make one system design video on creating a RDMS e.g. MySQL and Excel Sheet or Google Sheet?

[@anthonyc8579]

It would be great if you used a better mic/camera, because the audio quality is a bit hard to hear.

[@WittyGeek]

Great video Narendra! I have question in terms of synchronisation of document for users across the region. For example: A document is created by user in India and he is connected to a server in India. If he is collaborating with users in US and Europe and the master copy of the document is in India server, then when they try to perform any operation on the document it will be slow to communicate with the server in India and the experience will laggy due to Round Trip Time. And we cannot distribute the same document across servers as that will cause issues with which data operation needs to be performed. How can we solve this?

[@rahulsharma5030]

lots of confusion after 18 mins.OT part is not clear.Also in differential sync , once client applies patch to server, and get diff back, then client will apply that diff but client might have made some other changes, again it will get diff and patch diff and send to server?

[@gauravsachan5810]

In the example where you give delete(T,1) can be replaced with delete(1) The delete can work with only one argument position, need to send the char at that position.

[@kumarc4853]

The longer the beard, the wiser the Guru. Thank you Guru Narendra, very informational!

[@cantwaittowatch]

Thanks for the wonderful video Narendra. I heard you ask for feedback, hence sharing it here. Minor constructive feedback - I heard you repeat OT so many times and explained this in various flavors still leaving the outcome as OT. I think what you are trying to say is that users do their CRUD operations and this is sent to the server, wherein the server keeps the latest state and versions, and does the right logical manipulation to get meaningful and consistent information in real-time.

[@test59431]

@Tech Dummies....in the video at 25:36 mins, shouldnt it be D(T,2)??...and that's how you get "HA" for user 2........PLease confirm

[@lmxqlmxq]

to me, Operational transformation explanation was not clear

[@TechDummiesNarendraL]

Its a difficult concept, i guess it needs a separate video, i will try to do that

[@robertmartinez7938]

@@TechDummiesNarendraL This is the best explaination of operational transformation that I have found. Hopefully you can use it to inform your next video. conclave-team.github.io/conclave-site/

[@ishaangupta4125]

Just from looking at the video, it seems that this transform function is some Blackbox utility that has all the rules precomputed. This utility is just invoked as a library at runtime when the changes are made to the server copy and the final version is broadcasted to users

[@ishaangupta4125]

Unless you're really interested in knowing the granular details of this algorithm, I'd say think of it as a rule engine which already knows all the possible combinations that could occur and transforms the data accordingly

[@actondon6573]

waste of time, these content is at most a 10mins video

[@sumonmal009]

THIS COMMENT IS FOR MY PERSONAL REFERENCE. TO UNDERSTAND PROPERLY WATCH THE FULL VIDEO -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- why not pessimistic lock 3:13 optimistic lock 4:28 sync idea(event passing also known as OT, Diff sync) 6:00 Operational Transformation 8:50 14:35 20:05 OT example 20:54 Diff Sync (Preferred) 26:16 27:30

[@johnnytan2972]

vid would be 100x better if your audio output is fixed.

[@indrajitbanerjee5131]

You have put a lot of effort, BUT you repeat the same things again and again which makes the video elongated. Please try to make the videos short and crisp :) .. Kudos to your hard work for the community!!

[@manishbassi]

I really like your other videos, but this is the most confusing one. You really went into mathematical side instead of sticking to core system design concepts.

[@acobster]

That's because the mathematics are really the core design concept in a system that uses Operational Transformations. You really can't design an OT system without understanding the underlying math. I know; I've tried. :)

[@suhagraval8774]

Differential Synchronization Research Paper: neil.fraser.name/writing/sync/

[@InfinteMotivation]

Love your content! It's really helpful for the people prepping for interviews, information in such a concise way is not available on many channels/platforms.

[@bigyanrokaya842]

Hello sir I lack the basics .. Where & how should I start ?? I very much interested to learn .. Needed guidance..

[@microdocs8989]

Bigyan Rokaya ...Basic of what?

[@HabibAlSaki]

Check this github.com/donnemartin/system-design-primer/blob/master/README.md

[@MrSaransh]

@@HabibAlSaki Thank you very much from my side.

[@jaydeeppatwardhan9120]

@@HabibAlSaki This is one of the best reading materials to learn overall system design basics!

[@ameetmonty]

Narendra, great content. One minor feedback, the audio quality can be improved

[@az8134]

Thanks for the great video. I still don't get why we need a copy in the differential synchronisation. Can you please help?

[@Aavez2301]

the content is awesome, however the audio quality very bad

[@rahulsharma5030]

if git diff cannot resolve conflicts, then how can differential approach? It does not have enough info how to merge.

[@AbhishekSharma-si8ui]

AWESOME

[@adithyabhat4770]

Thank you for all the system design videos, this really helped me!

[@logicboard7746]

Great session. Btw, where did you get the T-Shirt from?

[@jalanubha]

Sir, MS Office 365 has all collaborative editing feature much before the release of this video. You may want to update that :)

[@BrianSmith-cw7mr]

Great video! Where did you learn the concepts and terminology from?

[@vinitbabariya7294]

Hello Sir, Can you make video on how stock exchange server, algorithm and application works? I am very curious to know.

[@abcd12272]

Why do we need two copies on the client and the server side? Why not just one on each side?

[@AnilKumar-ig8kk]

Hi Narendra, A big thanks to you for making the videos and explaining in detail. I am having interview's with faang and I am completely relaying your channel.

[@Engineer_With_A_Life]

How did it go bro? You got the opportunity?

[@alanwong68]

Would an operational transformation face the challenge of a chatty browser-server interaction?

[@devdev19]

Thank you for this explanation. I had a question similar to this and your explanation helped me to understand parts I was missing!

[@pk-bn1hc]

Can u pls explain how do to that chunking part??

[@pk-bn1hc]

@Brennan Abdiel how does this answer my question ?

[@anastasianaumko923]

🙏Thank you

[@RodrigoLopesBrazil]

Well, there is a good article about 4:30. en.wikipedia.org/wiki/Optimistic_concurrency_control But, optimistic concurrency control is not absolute lock-free. ;)

[@viditmathur8437]

If the client always talks to the server, wouldn't that cause too much lag to be real-time? Would P2P communication make more sense?

[@adithyaks8584]

It wont scale when you share docs between users then you need to handle n:n mapping / communications. If you choose a server client model then it is reduced to maintaining sync betwenn 1:n

[@zacklight]

Truly amazing!

[@cutegeetika7132]

So awesome

[@amber.k]

16:11

[@tedthebed7877]

Nice explanation

[@IbnIbrahem]

The last bit about waiting for ack from server is a probably a bad advice, you can't hold the UI waiting for server ack, instead let the user edit and merge or force the changes once the server responds

[@ishaangupta4125]

Aptly put

[@10clicks-freesoftwareengin51]

Good One!

[@KristopherWindsor]

Buzz buzz

[@pavanr92iitm]

Topic suggestion: Web sockets Thank you so much for the system design collection. One thing I really like about your videos is the simplistic way of conveying the point. It's sure going to be a popular channel over due course. Keep making, keep rocking :)

[@TechDummiesNarendraL]

Thanks and suggestions considered

[@stevenlawson9553]

really cool, thanks a lot

[@TarunPrasadKottary]

Do you have GIT link for all this ? Super helpful. Thank you

[@TechDummiesNarendraL]

Git link of code? Nope

[@TarunPrasadKottary]

@@TechDummiesNarendraL not code. git instead of imgur

[@michalsokolowski5728]

Very confusing explanation, the drawings don't reflect what is being told.

[@vk-p]

Not really clean explanation

[@saiprajeeth]

I learned something new because of you

[@MohitYadav-nu4fu]

can you speak in your regular accent ?

[@DanOhCaptainDaniel]

Man.... so good. This guy is a great teacher! MIT? Harvard? UW? Lol

[@TechDummiesNarendraL]

😂😂

[@DanOhCaptainDaniel]

@@TechDummiesNarendraL so my only gripe is I usually watch lecture vidoes in 1.5 x speed. sometimes difficult to understand then. I use caption though. Anyways, thank you so much! Really is a big help in understanding different systems design questions!

[@gauravlodhi5137]

content is good but sometimes your accent sucks.

[@xxbighotshotxx]

This is a great video, thank you so much for sharing it! For next time, can you please remember to silence your phone? It's quite distracting