Google Dremel: 1 TRILLION FILE READS in 10 seconds

[@gkcs]

Thank you for watching! Here is the system design judge link: interviewready.io/practice/system-design-judge/

[@ArunRamakrishnan]

Wonderful effort and quite rare to see discourses on deep system design. We used to do this when we were working on I/O on our large NUMA systems but it was internal. Keep up the amazing effort.

[@gkcs]

Thank you! A new video is coming this Saturday 😁

[@ashokansivapragasam2762]

I have been following you since 6 years for Data Structures, Algorithms and System Design. Always informative and detailed. Love to see more videos! Good work!

[@gkcs]

Thank you!

[@plashless3406]

Wo wo wo, wait a minute. I just want to say "You are absolutely awesome"

[@gkcs]

Thank you!

[@shubhamjagtap108]

Please do a research paper walkthrough for Hadoop Distributed File System too... Thank you for this, I work with parquet daily, I will go in details of compression.

[@gkcs]

Great suggestion, thank you!

[@blasttrash]

@shubhamjagtap108 when will you make video on parquet?

[@shubhamjagtap108]

@@blasttrash Hey, I mean I work as SW engineer (2 years experience) at an MNC, I daily work on distributed systems and AWS services (DevOps).

[@baibhabmondal1740]

Did not watch the video fully yet, but just felt happy watching your vids again. I used to watch your videos back in college, we even chatted once over linkedin. Things got busy, Now I actually get to do what you used to explain on your older videos about design. Serve scale of Billions, as an EM. And its just feel nice that I started from here (your YT channel). Times were simpler then, maybe for both of us. I was in college. And you were still at Uber, unmarried, without a kid. Now you got a company of your own, married and a kid. Time flies. I am too getting married next year, lets see how that goes. 2017 feels 2 days ago man.

[@gkcs]

I am glad to connect with you again. Hope you have a great life ahead, and congratulations on the wedding!

[@YashpriyadeepKatta]

That was lovely to read mate, what's your name on Linkdin?

[@harishrg777]

I come from biomedical background, initially I was like why was this suggested randomly. But this explains skeletal concepts, since now we are all dealing with data and analytics irrespective of field and most part of queries is monopolized through google. This explanation gives a good understanding, thank you for making this explanation. Liked and subbed.

[@gkcs]

Thank you!

[@ankushtagore4299]

following you since i was in college and i can tell this without thinking that you are one of the greatest teacher of all time. your lecturers are great, whether viewer is a noob or a veteran developer.

[@gkcs]

Thank you Ankush!

[@gradientO]

Thanks for this!! I was just searching papers for Dremel, Bigtable and Aurora

[@gkcs]

Cheers 😁 More on the way!

[@shubhamwaingade4144]

I think I have found a goldmine here! New subscriber added!

[@gkcs]

Thank you!

[@madhuiitb-cse]

Thanks a lot for putting efforts for the engineers. It is really very helpful.

[@gkcs]

Thank you!

[@Tdavis1911]

Fantastic work, great way to organize and present the subject material

[@gkcs]

Thank you, cheers!

[@Md_sadiq_Md]

Pushing the algorithm ❤

[@gkcs]

Thank you!

[@jiteshpanda09]

I think Oracle Exadata is also inspired by Google Sharding and Columnar Storage.

[@ArunRamakrishnan]

I am not sure if Cockroach DB is based on the spanner design enhancements or some parts of dremel as well.

[@imhiteshgarg]

Hi Gaurav, Nice video. We are also in the process of using some tools for queries related to aggregations at our end but we have finalised with Apache Pinot. After going through this video, I think Dremel is a lot similar to Pinot. What are your thoughts on it?

[@raaz9662]

Good one Gaurav

[@gkcs]

Thank you!

[@shreyaspatange8653]

13:28 this is similar to AOS and SOA

[@gkcs]

Yes 😁

[@shafiq_ramli]

Where can I learn this like you did? Should I search a systematic syllable?

[@gkcs]

I use these resources: interviewready.io/resources

[@ankk98]

Aggregation on column based databases are also faster because the chances of predicting the next index are much higher.

[@gkcs]

Interesting point, thanks for sharing!

[@kartikkaushik4743]

Thanks for sharing it Gaurav from where we can study the reasearch for the same article can u pls share about it?

[@gkcs]

It's in the description 😁

[@sachinkumar-jc3ub]

Yes I seen system design judge on cup ,i think lot of missed it😅

[@gkcs]

Hahahha :p

[@sarveshsawant7232]

Does amazon athena has similar architecture?

[@BudsintheAir]

Thank you sir ❤

[@gkcs]

Cheers!

[@thilaksalian5419]

Thank you

[@gkcs]

You're welcome :D

[@Vikrant19912]

I couldn't help but compare spark's architecture with this!

[@gkcs]

It does have some ideas from here :) Spark also uses Map Reduce, and real-time mini batches.