How to NOT Fail a System Design Interview (By a Data Engineer)

This is what a systems design interview at Google, Amazon, Facebook or any other big tech company looks like. Systems design interviews are vague and tough to crack, but with the right approach, you can nail it! Sharing my experience and strategy to do just that here!
Resources:
cloud.google.com/architecture
acloudguru.com/course/google-...
www.udemy.com/course/aws-cert...
www.lapabooks.com/book/design...
airflow.apache.org/docs/apach...
systemsexpert.io/
Follow me here:
LinkedIn: / jashradia
Instagram: / jash.radia
Google interview experience: • Google Data Engineer I...
Sharing my resume and tips: • The Resume that got me...
Thanks for watching the video! What did you think about it? Drop your comments! Don't forget to like and subscribe :)
0:00 Pre Intro
1:41 Intro and Disclaimer!
3:49 Application design questions
6:25 Application design solution 1
9:22 Application design solution 2
12:00 Data pipeline design questions
14:21 Data pipeline design solution
17:29 Resources and preparation
19:13 What next?
#interview #google #technology #cloud #bigdata

[@JashRadia]

We have just reached 1000 subs! And it's all thanks to you super awesome subscribers and viewers! ❤️ If you have not subscribed already, what are you waiting for? Drop your questions or suggestions below! Let's talk :)

[@arnavkashyap822]

awesome video jash, generally nobody talks about piepline designing and sytem design for the DE domain. Great sharing of resources, great applaud to you

[@JashRadia]

Thanks a lot, Arnav! And yes, there is a lack of videos on such topics in DE, that was my main reason to create one! Really glad you liked it 😊

[@salmansayyad4522]

hey, really cleared my concepts. thanks bro, keep posting such videos.

[@1godfrey]

this is amazing please do more system design vids like this for data engineers where you go in depth like this

[@priyankapandey9122]

Thanks jash for creating the video. Really helpful.

[@JashRadia]

Aweosme to know! Thanks for watching 😊

[@jaladhithakur7567]

Great video. This really explains how to solve system design questions. Also, thanks for sharing the resources 😃

[@JashRadia]

Really glad you liked it, this one took a lot of effort :D

[@snaekboi]

That was excellent! Well structured, and well explained. Even my undergraduate self was able to understand quite a bit of it :)

[@JashRadia]

Thank you! Really glad you liked it 😊

[@larissaarreola2448]

this was so helpful and somehow reassuring, I really appreciate all your work and effort! Keep it up (:

[@JashRadia]

Thank you! 😊

[@yahoo5728]

Awesome video, really helpful. I found these questions related to system design which are usually asked in the interviews. If you could share some insights on these would be really helpful. 🌊 Data Pipeline Design: How would you design a data pipeline to handle large volumes of streaming data? (e.g., IoT devices or website clickstreams) ⚖ Batch vs. Stream Processing: Explain the differences between batch and stream processing and when to use each in data systems. 🏢 Data Warehousing: Design a data warehousing system for e-commerce. Discuss storage tech, data modeling, and querying methods. 🧩 Data Partitioning & Sharding: How to improve performance and scalability by partitioning and sharding a large database? Discuss trade-offs. 📦 Data Serialization Formats: Compare JSON, Avro, Parquet, and ORC. When to use each in data processing? 📜 Data Compression: Discuss data compression techniques in big data systems and choosing the right algorithm. 🌐 Distributed Data Processing: Explain distributed data processing with Hadoop, Spark, or Flink, emphasizing fault tolerance and data locality. 🔄 Data ETL: Design an ETL process to migrate data from a relational DB to a data lake. Discuss tools and frameworks. ⚙ Resource Configuration: Handling 100GBs of data per spark-submit - How to configure the cluster? 🧐 Data Quality & Validation: Ensuring data quality and validation in data pipelines, handling missing or erroneous data. 🔒 Data Security: Best practices for securing sensitive data in big data environments - encryption, access control, auditing. 📈 Scalability: Scaling data systems horizontally and vertically to meet growing data volumes and workloads. 📊 Monitoring & Logging: Importance of monitoring and logging in data systems, tools, and metrics for system health. 🗃 Data Archiving & Retention: Data archiving and retention strategy for a data warehouse, handling historical data. 💰 Cost Optimization: Strategies to optimize data storage and processing costs in cloud-based data architectures (AWS, Azure, GCP). 📜 Data Governance: Role of data governance in data engineering - ensuring compliance with data regulations.

[@artofheart2891]

Excellent Information. Thank you for sharing..

[@JashRadia]

Thanks for watching! 😀

[@financial_cycle]

the intro is so accurate

[@hamidomar350]

Appreciate the effort, the content is very clear and presented in a way that it is easy to absorb. I would love to know if beginners can actually configure and test these systems, even if to a limited extent, without having to pay for any cloud service? If not, what would be the next best thing once can do to learn these concepts practically?

[@JashRadia]

Thank you so much for watching and liking the video! 😊 For the hands on practice, every cloud has some free tier usage. For example, snowflake gives 30 days trial and AWS is free always for certain services. Just ensure you are checking if the service is free or not before spinning up.

[@vrohan07]

Thank you

[@SidharthanPV]

Thank you for your video.

[@JashRadia]

You're welcome!

[@abhinavpandey5306]

Can you share the detailed list of topics from basic to advanced for system design for data engineers

[@paragradia]

Cool 😎 Awsome information 👌

[@JashRadia]

Thanks 😁

[@joyjitpal]

Excellent explanation

[@JashRadia]

Really glad it was helpful, thanks for watching 😁

[@joyjitpal]

Please make more of these kind of videos

[@JashRadia]

@@joyjitpal will definitely do!

[@sirajansari2848]

Excellent Video !! Although, I have a question. What would have changed in the data pipeline if the source would have been a streaming source ? Where should we put the Kafka/PubSub in the data pipeline ?

[@JashRadia]

2 options. Between source and landing or between landing and processing. Also can be put in both places. And thanks btw 😊

[@ajourney179]

any good book to learn system design for data engineering & not SDE.

[@nobodyinparticula100]

Crisp and concise. Super helpful. Is this entry level question? Does L5 level have same depth? And is mostly for India based companies or US based? Will Data Engineer have more of the second example type System Design? Or is it same as SDE? What is a good site to practice DE specific system designs? Thank you!

[@JashRadia]

Questions are similar no matter the level. It's just that you are expected to go in more depth in L5 in your answer. India or US doesn't matter, it's common. DEs will have 2nd type of system design mostly unless it's a startup.

[@pradhyumansinghmandloi8240]

Is software engineer system design are same as data engineering system design?

[@puneetnaik8719]

Great video on system design. Just wanted know to what tool/software do you use for creating data flow diagram during interview.

[@JashRadia]

Draw.io is my go to tool. In some cases, I also just actually draw it on the screen using one note etc.

[@puneetnaik8719]

@@JashRadia thank you so much

[@thedailyepochs338]

Awesome

[@angelnadar6451]

Hello Jash , The podcast with Shashank was helpful. Just can you please let me know what resources you used for dsa using python.

[@JashRadia]

Hi Angel, thanks for watching it. I am sure you'll find the content you like on this channel, too. As for the DSA preperation, I would recommend hackerrank. There is a specific python path and a dsa path that you can solve using python. It starts from basic and goes till advanced

[@angelnadar6451]

@@JashRadia thank you so much Jash !!! Have a great day !!!😊

[@Nick-du9ss]

Where can we find projects for data engineering for targeting product based companies ?

[@JashRadia]

Go to this link for GCP and pick projects you're interested in. They also let you create a qwiklabs environment for hands on. cloud.google.com/architecture For AWS, you'll find some projects here in the projects section. thirdeyedata.ai/projects/data-engineering/?_gl=1*72axvl*_ga*MTU0Mjg4MzYxLjE2NjI1MjQ1NjY.*_ga_DPYTFQ0MMC*MTY2MjUyNDU2Ni4xLjEuMTY2MjUyNDU3MS4wLjAuMA..&_ga=2.184481186.54541546.1662524567-154288361.1662524566 And I have also mentioned a data engineering course from acloudguru in the description. It has a lot of hands on for GCP

[@Nick-du9ss]

@@JashRadia thanks

[@asktostranger8296]

Can you tell Difference between system design round For software engineer vs data engineer If I cover the lld and hld for software engineer Will i able to answer the system design question for data engineering Plsss clarify 🙏🙏🙏

[@JashRadia]

I would suggest you to check out system design video on this channel. I am explaining this very thing by 2 examples.

[@hritikapal683]

@@JashRadia video other than this one? If yes can you provide me the link? thankyou!

[@meetpatel9690]

Could you tell us that what will be the job of data engineering in web 3.0 ?

[@JashRadia]

Interesting question.. I had a job offer before joining google in a web3/defi company as a data engineer.. The job of data engineering is going to remain pretty much what it has already been. The difference being, the data is going to be captured from web3 services and products rather than traditional way. Web3 also has a ton of information already available especially on smart contracts. This will be the starting point.

[@meetpatel9690]

@@JashRadia means data engineering job is going to be remained in web 3.0 space ?

[@JashRadia]

@@meetpatel9690 100%. Any firm needs to handle data and learn it to grow. Web2 or web3.

[@meetpatel9690]

@@JashRadia thanks jash

[@jacksmith7160]

How much time needed to become data Engineer if we start learning data engineering from scratch ?

[@JashRadia]

Anywhere between 4-6 months. You can refer to this post I made on LinkedIn. www.linkedin.com/posts/jashradia_cloud-bigdata-technology-activity-6960846154028728320-e_Dc?

[@jacksmith7160]

@@JashRadia it will help a lot to get into data engineering

[@ganeshtaware9883]

Hi Jash....I am regular follower of your vedios. I would like to connect with you if possible ...Is Data Engineer opening still in Google ? How to prepare for the same ... I have 12 years of experience mainly in ETL and datawherhousing...what is path that I should prepare that will help to get into Goole or like companies ... Sorry many questions are in single comments but it will be great if we have one to one ...

[@JashRadia]

Currently hiring freeze is there for most positions in Google. So I wouldn't recommend applying now. And for getting selected, it's about getting your basics right. Prepare on SQL, Data modeling, DSA, pipeline design, distributed systems and cloud in the same order I mentioned here. You will be good.

[@ganeshtaware9883]

@@JashRadia Thanks.

[@sathyamanikantabk4483]

do a unboxing video of the perks that you got from GOOGLE may be it will inspire others and All the best brother hope to see you Bigdata content on your channel and I have just subscribed your channel and Congratulations and wish me the same luck for Google

[@JashRadia]

While I would love to do that, I have already unboxed everything out of excitement 😂 but yes, I can create a video separately showing all the things we get. Thanks for suggestion! And yes, best of luck to you!

[@sathyamanikantabk4483]

@@JashRadia lol...I understand

[@video-hs5no]

Isn't it datalab deprecated?

[@JashRadia]

Yes, should be replaced by vertex ai

[@video-hs5no]

@@JashRadia vertex AI workspace

[@WelcomeToDataverse]

Do data engineers are asked system design?

[@JashRadia]

Yes, we are. More often than not it's the 2nd type of question that I have covered rather than 1st one.

[@WelcomeToDataverse]

@@JashRadia Yes I went through the whole video. I posted this comment while the intro.

[@tusharhatwar]

Hey Jash! Just gone through your video on Shashank's Channel Landed up here and subscribed to your channel. I would request to bring more content on this channel related to Big Data technologies. By the way, I Loved your calm and composed way of Explanation. Sent you a LinkedIn connection request, It would be great if you can accept it :)

[@JashRadia]

Thanks Tushar! I'll also keep your suggestion in mind about the content. Glad to know you like it so far 😊