Design file-sharing system like Google Drive / Dropbox (System design interview with EM)

Рет қаралды 30,524

Күн бұрын

Today's system design mock interview: "Design Google Drive."
Candidate: Alex, engineering manager (ex-Shopify) and now a coach on our platform.
Book a session with Alex here: igotanoffer.com/?Y...
Chapters:
00:00 Intro
00:36 Question "Design a file-sharing system like Dropbox, Google Drive, etc"
00:55 1. Clarifications and requirements
09:51 2. High-level design (components)
13:08 2. High-level design (APIs)
19:20 3. Drill-down (client responsibilities)
22:09 3. Drill-down (schema)
29:41 3. Drill-down (upload flow)
33:53 3. Drill-down (download flow)
36:24 4. Refinements (regionality)
37:45 4. Refinements (S3)
39:22 4. Refinements (CDN)
40:13 4. Refinements (versioning)
41:00 4. Refinements (encryption)
41:45 4. Refinements (database)
42:42 5. Follow-up (read vs write)
44:07 5. Follow-up (folders)
47:35 6. Outro
About us:
IGotAnOffer is the leading career coaching marketplace ambitious professionals turn to for help at high-stakes moments in their career. Get a job, negotiate your salary, get a promotion, plan your next career steps - we've got you covered whenever you need us.
Come and find us: igotanoffer.com/?Y...

Пікірлер: 58

@IGotAnOffer-Engineering 5 ай бұрын

Get 1-on-1 coaching to ace your system design interview: igotanoffer.com/en/interview-coaching/type/system-design-interview?KZbin&

@evgenirusev818 5 ай бұрын

I think that this is the best IGotAnOffer video so far. Please bring in Alex for another one - perhaps to design Google maps? Thanks.

@IGotAnOffer-Engineering 5 ай бұрын

Wow thanks yes I've already asked him to do another one, so watch this space!

@scottlim5597 2 ай бұрын

This candidate has real life experience and it shows in the interview. He starts out simple and build on top of it. I love it.

@arghyaDance 4 ай бұрын

Very useful - Simple, Clear, no hurry, flow is really good

@lakshminarayanannandakumar1286 4 ай бұрын

@31:00 I believe that the rationale for adopting SQL to maintain consistency and reference the CAP theorem is misleading. It's important to note that the concept of consistency in the CAP theorem differs significantly from the consistency defined in ACID. Consistency in CAP means "every read should see the latest write" as opposed to consistency in ACID which ensures that your DB always transits from one valid state to another valid state.

@dmitriyobidin6049 Ай бұрын

Yea, you are right. Consistent follower/read replica DB in terms of ACID can be in non-consistent state in terms of CAP, just because it wasn't yet update from the main/write replica.

@user-jz1lx1rv1n Ай бұрын

He meant consistency in the CAP theorem.

@naseredinwesleti300 5 ай бұрын

im genuinely happy to discover this beautiful channel this was very insightful. thank you and keep sharing.

@franssjostrom719 4 ай бұрын

Indeed a great video everything from rough calculations to being communicative with the customer was great 🎉

@DevendraLattu 3 ай бұрын

Great question asked about compression at 20:55 with a well-structured answer.

@vivekengi01 4 ай бұрын

Its great video but I think we missed 2 important things: 1) The file permissions were missing while considering schema design which is "must have" for any file sharing system 2) For very large files how the upload and download can be optimized to save network bandwidth instead of just redirecting to S3. Please take these inputs positively and keep sharing such videos.🙂

@AkritiBhat Ай бұрын

Great video. The way he approaches depth shows that he is very strong

@IGotAnOffer-Engineering Ай бұрын

Glad you think so!

@almirdavletov535 5 ай бұрын

Great video, thanks. One thing I'm really missing is some sort of a judgement, what went well, what was not ideal. I see that DB design wasn't really well though out, or maybe it's just me. Sorting such things out as a conclusion to video would be a great value to those who watch these videos!

@Rajjj7853 5 ай бұрын

Good Video! I think there is a mis-calculation, the total storage use for 100 million users is around 1,500 Pb, not 1.5pb.

@moneychutney 3 ай бұрын

not so important

@dimitryzusman6711 2 ай бұрын

Oh, no. It is uber important. 1.5PB - 1 large storage account. 1.5 EB is totally different scale of algorythms and data storage - @@moneychutney

@dwivedys 3 күн бұрын

Yes I too noticed that: 100 Mil users and 15 GB per user = 100 * 10^6 * 15* 10^9 = 1500 * 10^15 or 1500 PB

@rsbgarg 10 күн бұрын

Well, from how I see it metadata is more suitable in a no-SQL setup Because then the file structure can be stored in metadata as lightweight storage for sync ups amongst new devices of the user and the JSON structure gives the capability of nested folder structure being stored. So we have the tables - user_details-personal details+meta, user_login, user_folders, and file_meta-with versions. also, NoSQL is fine as we do not have large volumes of concurrent requests updating on the same keys. So, locking and transaction atomicity is not an issue. Also, reads are easily handled.

@prasadhraju 3 ай бұрын

For 100M users, each user has 15 GB storage space, shouldn't the total storage be 1.5 Exa bytes? Explanation: 100,000,000 * 15 / 1000 * 1000 = 1500 PB = 1.5 EB.

@31737 Ай бұрын

yea calculated the same thing, that is why the back of napkin/envelop is dangerous, I skip it all together as if you get it incorrect it is a big fail.. whats obvious is for this system we must scale horizontally and distribute the load, who cares how many servers is required or not its not even the scope of the interview.

@lalitkargutkar595 2 ай бұрын

Great video!!!

@aju126 5 ай бұрын

I think the main requirement of a file-sharing system is how the edits are handled, something like every edit on a file does not sync the whole file across devices and just the data chunk that was edited, without this requirement, its same as any other design with models and data floating around. Overall it was a great design interview, But one question i have across all the design interviews is the math performed in the beginning wrt to no of users, traffic, QPS etc, how is it even used?

@mrchimrchi1867 4 ай бұрын

Alex is fantastic in this video. The interviewer looks like he wants no part of being in this video though.

@nobreath181818 4 ай бұрын

Maybe the best system design tutorial I've ever seen.

@IGotAnOffer-Engineering 3 ай бұрын

Wow, thanks!

@brabebhin368 4 ай бұрын

It is not ok for the client to tell the api server that the upload is done. The client may be unable to tell the api server about the upload being finished due to network outage, so your metadata database will now be out of sync. You will want to keep that logic on the server.

@KNukzzz 3 ай бұрын

Agreed x1000

@adityasanthosh702 2 ай бұрын

And how exactly would that be implemented? I am curious

@RenegadePawn 4 ай бұрын

Seemed legit to me for the most part, but why would you use a CDN? Unless there are lots of users with certain big files that are the exact same, what benefit would a CDN provide?

@MarcVouve 2 ай бұрын

Since the video mainly used AWS, I'll use cloudfront as an example so I can be specific. In addition to some security benefits, CloudFront operates over AWS's private network globally and which is typically faster/more reliable than public backbone internet.

@user-jz1lx1rv1n Ай бұрын

I think CDN can also be location optimized. So, a person living in Brazil can fetch data from a CDN located near to brazil instead of the one located in US, making it more efficient.

@jockeycheng8183 17 күн бұрын

don't quite understand the workflow, shouldn't the server itself interacts with S3 to put the data in?

@yacovskiv4369 2 ай бұрын

Is that bit about partitions in S3 accurate? S3 uses a key-based structure where each object is stored with a unique key. The key can include slashes ("/") to create a hierarchy, effectively mimicking a folder structure but there aren't any actual folders in S3; it's all based on the keys you assign to your objects. So, what does he mean by splitting into more folders when they become too large?

@tameribrahim6869 Ай бұрын

AWS S3 (and other cloud blob storage services) are flat storage, the console UI shows folders structures but it's just extracted from the file path (/{folder}/..). The only think that may suggests adding a DB field/table, is to store & track the available folders per user to improve performance, as listing the folders directly from SDK/API means you fetch all the blobs then extract the folder structure from them!

@drawingbook-mq6hx 26 күн бұрын

If the data is compressed in client side , wondering who will divide the data in blocks and send the data in block. Sending block is having advantage to deduplicate . Thought? May be make sense chunking in client side it self. Once the compressed chunks are uploaded to storage, meta data DB can be updated.

@yiannig7347 4 ай бұрын

Some data inconsistency issues between DB and Queue.

@R0hanThakur 2 ай бұрын

I think adding the Loging Auth on the client side is not recommended for security reasons. One of the points which my Interviewer didn't seem to be happy about....and since that was for the security team I think that cost me losing the offer

@oleksandrbaranov9649 2 ай бұрын

always have an impression that the interviewer is trying really hard not to fall asleep 😂

@PetarPetrovic-qw9zj 3 ай бұрын

where is the cache on client side?

@gouravgarg6756 2 ай бұрын

It is not safe to store AWS credentials on the frontend(client/Browser) for direct S3 uploads. We can use multipart s3 API but it has to be done through API server. Correct?

@naveenn7935 Ай бұрын

You will not be storing AWS creds on client machine. He mentioned he will get the signed URL from S3 via API server which will have TTL.

@AzharAhmedAbubacker 5 ай бұрын

Which app is that to used to draw the diagram?

@massaad 5 ай бұрын

It's free version of Figma, its a called a "fig jam" I think. Very fun to use!

@oleksandrkhivrych6349 5 күн бұрын

Can you send the Figma template, please?

@sinchanamalage7249 3 ай бұрын

What’s the drawing tool used in this video?

@tamarserok3164 3 ай бұрын

it's Figma

@webdevinterview 2 ай бұрын

Couple of notes: - back of the envelope calculations were not utilised at all - notifications part was not covered at all - the interviewee is pretty much into AWS stuff and goes too much about its specifics - upload endpoint should return file id plus some additional metadata about file, probably also 201 response - also API design didn't really correspond to what he told in the end. If we upload directly to S3 the flow should be following: a) client calls API server to get an upload link b) client uploads file directly to S3 using link c) once upload is finished, client gets some file id which it sends to another API server endpoint to record that file was actually uploaded. Or maybe there is a way how S3 can itself notify API server, idk about that. - there should be an endpoint get info about all the files in the cloud - compression on web or mobile is probably not a great idea, compressing 10gb file will eat ton of battery Overall, i'd say this system design lack quite some depths.

@adityasanthosh702 2 ай бұрын

10GB won't be compressed as an entirety. It will be split into chunks and each client process/thread parallely compresses it. Regarding notifying the server when a client has finished downloading/uploading, I do not think that info needs to be saved by the server. The server's job is to store the latest files and provide links when clients request them. If the file download from client is unsuccessful, the server can assume that its the responsbility of clients to request new changes. Same with conflict resolution. Instead of server resolving them, it can ask clients "hey some other client changed the same file. Which one is the latest?"

@user-jz1lx1rv1n Ай бұрын

For 100 million users, shoul it not be 1.5 exabytes?

@pixusru Ай бұрын

There is no database on the high-level diagram. Then, in drill-down, Alex jumps to designing DB schema.

@31737 Ай бұрын

I believe the math is incorrect you must take 100M users * 15 GB to get to the total which is 1,500 PB

@RajdeepBiswasInd 3 ай бұрын

I think you missed out on synchronization.

@naveenn7935 Ай бұрын

I think notification part does the sync part?

@kaiwu191 Ай бұрын

This guy is definitely an experienced engineer, but he didn’t prepare for this kind of interview very well, maybe he is a little bit nervous during the interview. He is trying to say a lot terms like s3 to make him sounds professional but lost many details on how to design the solution from scratch like handling big files. This is a question about designing Dropbox, not use case of s3. The performance would be rejected for any senior positions.