Data Engineer Mock Interview | ADF | Medallion Architecture | BRONZE, SILVER & GOLD Layer| ADLS GEN2

𝐓𝐨 𝐞𝐧𝐡𝐚𝐧𝐜𝐞 𝐲𝐨𝐮𝐫 𝐜𝐚𝐫𝐞𝐞𝐫 𝐚𝐬 𝐚 𝐂𝐥𝐨𝐮𝐝 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫, 𝐂𝐡𝐞𝐜𝐤 trendytech.in/?src=youtube&su... for curated courses developed by me.
I have trained over 20,000+ professionals in the field of Data Engineering in the last 5 years.
𝐖𝐚𝐧𝐭 𝐭𝐨 𝐌𝐚𝐬𝐭𝐞𝐫 𝐒𝐐𝐋? 𝐋𝐞𝐚𝐫𝐧 𝐒𝐐𝐋 𝐭𝐡𝐞 𝐫𝐢𝐠𝐡𝐭 𝐰𝐚𝐲 𝐭𝐡𝐫𝐨𝐮𝐠𝐡 𝐭𝐡𝐞 𝐦𝐨𝐬𝐭 𝐬𝐨𝐮𝐠𝐡𝐭 𝐚𝐟𝐭𝐞𝐫 𝐜𝐨𝐮𝐫𝐬𝐞 - 𝐒𝐐𝐋 𝐂𝐡𝐚𝐦𝐩𝐢𝐨𝐧𝐬 𝐏𝐫𝐨𝐠𝐫𝐚𝐦!
"𝐀 8 𝐰𝐞𝐞𝐤 𝐏𝐫𝐨𝐠𝐫𝐚𝐦 𝐝𝐞𝐬𝐢𝐠𝐧𝐞𝐝 𝐭𝐨 𝐡𝐞𝐥𝐩 𝐲𝐨𝐮 𝐜𝐫𝐚𝐜𝐤 𝐭𝐡𝐞 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰𝐬 𝐨𝐟 𝐭𝐨𝐩 𝐩𝐫𝐨𝐝𝐮𝐜𝐭 𝐛𝐚𝐬𝐞𝐝 𝐜𝐨𝐦𝐩𝐚𝐧𝐢𝐞𝐬 𝐛𝐲 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐢𝐧𝐠 𝐚 𝐭𝐡𝐨𝐮𝐠𝐡𝐭 𝐩𝐫𝐨𝐜𝐞𝐬𝐬 𝐚𝐧𝐝 𝐚𝐧 𝐚𝐩𝐩𝐫𝐨𝐚𝐜𝐡 𝐭𝐨 𝐬𝐨𝐥𝐯𝐞 𝐚𝐧 𝐮𝐧𝐬𝐞𝐞𝐧 𝐏𝐫𝐨𝐛𝐥𝐞𝐦."
𝐇𝐞𝐫𝐞 𝐢𝐬 𝐡𝐨𝐰 𝐲𝐨𝐮 𝐜𝐚𝐧 𝐫𝐞𝐠𝐢𝐬𝐭𝐞𝐫 𝐟𝐨𝐫 𝐭𝐡𝐞 𝐏𝐫𝐨𝐠𝐫𝐚𝐦 -
𝐑𝐞𝐠𝐢𝐬𝐭𝐫𝐚𝐭𝐢𝐨𝐧 𝐋𝐢𝐧𝐤 (𝐂𝐨𝐮𝐫𝐬𝐞 𝐀𝐜𝐜𝐞𝐬𝐬 𝐟𝐫𝐨𝐦 𝐈𝐧𝐝𝐢𝐚) : rzp.io/l/SQLINR
𝐑𝐞𝐠𝐢𝐬𝐭𝐫𝐚𝐭𝐢𝐨𝐧 𝐋𝐢𝐧𝐤 (𝐂𝐨𝐮𝐫𝐬𝐞 𝐀𝐜𝐜𝐞𝐬𝐬 𝐟𝐫𝐨𝐦 𝐨𝐮𝐭𝐬𝐢𝐝𝐞 𝐈𝐧𝐝𝐢𝐚) : rzp.io/l/SQLUSD
30 INTERVIEWS IN 30 DAYS- BIG DATA INTERVIEW SERIES
This mock interview series is launched as a community initiative under Data Engineers Club aimed at aiding the community's growth and development
Our highly experienced guest interviewer, Ankur Bhattacharya, / ankur-bhattacharya-100... shares invaluable insights and practical advice coming from his extensive experience, catering to aspiring data engineers and seasoned professionals alike.
Our talented guest interviewee, Sasiram Kolisetty / kolisetty-sasiram-0285... has a remarkable approach to answering the interview questions in a very well articulated manner. Where he talks about the complete Medallion Architecture followed in the projects that he worked on belonging to Insurance and Green Energy Domain. He describes how the raw data was initially added to the bronze layer tables. Then performed cleaning and transformations on this data which was then added to the Silver layer tables. Finally, based on the business requirements, data was aggregated and the results were added to the Gold Layer.
Link of Free SQL & Python series developed by me are given below -
SQL Playlist - • SQL tutorial for every...
Python Playlist - • Complete Python By Sum...
Don't miss out - Subscribe to the channel for more such informative interviews and unlock the secrets to success in this thriving field!
Social Media Links :
LinkedIn - / bigdatabysumit
Twitter - / bigdatasumit
Instagram - / bigdatabysumit
Student Testimonials - trendytech.in/#testimonials
Discussed Questions with Timestamp
3:51 Brief discussion about the project.
9:06 What business problem are you solving in your project?
9:56 What work have you done to secure your data?
11:20 Do you use Unity Catalog for role-based policy?
11:33 How do you resolve bottlenecks in terms of latency in your pipeline?
15:14 If you have 2 large tables, what kind of join strategy will you follow?
16:12 Difference between bucketing and Z-ordering.
16:48 How do you perform data ingestion?
18:20 How would you create a pipeline for real-time data updates, aggregation, and live dashboard creation?
20:13 What triggers are available in ADF?
21:18 How does lazy evaluation help in Spark optimization?
22:45 What approach would you take for cluster configuration
24:53 SQL coding question
Tags
#mockinterview #bigdata #career #dataengineering #data #datascience #dataanalysis #productbasedcompanies #interviewquestions #apachespark #google #interview #faang #companies #amazon #walmart #flipkart #microsoft #azure #databricks #jobs

[@satyanarayanabadam9764]

This is the best interview till now. He worked on security , CDC concepts etc. Thanks sumit sir for conducting these interviews. Really helpful who is preparing for the interviews.

[@joerokcz]

Ram is one of the best candidates of the series. All the best Ram.

[@sumitmittal07]

yes definitely, he has a great exposure

[@sonurohini6764]

Definitely. It is a great help sir.

[@ramprasadh5322]

This serires is great and gives me confidence to and sit in the interview.

[@asaisvlogs7258]

It just wonderful task... Its very much helpful ❤❤❤❤❤

[@NaveenKumar-ln9vg]

I am getting prepared for the interview next week. As you said failed interview better then not attending interview and these mock interviews giving me the confidence to face the interview. Great initiative, learning and help. Thanks!

[@skillhorizon]

You are doing great work. Lot of respect

[@sandhyasandy9072]

This is one of the best interviews and got good insights Sumit sir.Learnt the new concept of thin executor and fat executor and his optimisation techniques are also unique like join re order Adq etc…

[@sandipansaha6847]

Very helpful video.... gr8 help ...

[@SiddharthVerma-lm3sj]

These videos have been very helpful. I have gone over many and this would be beneficial for a long time. Great work, initiative and thanks for the efforts.

[@Sudeep-ow4pe]

The interview series is really helpful, Thank you

[@niridha23]

Thanks for conducting these mock interviews Sumit sir. It is really helpful😊

[@pradumanyadav493]

this is very helpful series

[@AnandPatil-eu1tl]

Thank you sir , this videos are very helpful

[@harshpatel3384]

Very helpful

[@user-jg2tn1wb3d]

Thank you sumit sir and team, your hard work really means a lot

[@prasadrajupericharla5545]

Great one 🙌

[@bhagirathpandey3560]

Gr8 effort 🌟

[@nsinha27]

Sumit - these mock interviews are worth it's weight in gold.

[@user-dj4ht7rg2f]

Good one!!

[@anikethdeshpande8336]

Thanks for sharing. Very helpful

[@sumitmittal07]

Glad it was helpful! Keep watching for more such interesting content

[@swapnildande4706]

Really helpful for all Data engineer

[@sumitmittal07]

Happy to know that you are finding the interview series helpful

[@FF-bp9bj]

Sumit sir you are a legend in BIGDATA... You are the best trainer I've seen so far... Thanks for making the bigdata enthusiastics life easier....

[@sumitmittal07]

Always happy to help!

[@ravikanth6178]

The interviews are very helpful… Thank you Sumit and Team🎉🎉

[@sumitmittal07]

Glad to hear that you found the mock interviews insightful. More such sessions are scheduled for release in the upcoming days.

[@pranavadhav597]

What an amazing interview. This series is so helpful. Thanks sumit sir

[@sumitmittal07]

Keep watching for more such engaging interviews

[@RohitSharma-ny1oq]

Best

[@ManaviVideos]

Thanks for the interview series!!

[@sumitmittal07]

glad that you liked it

[@chinmayamahapatra6797]

It was really effective to us very very helpfull

[@sumitmittal07]

Happy to know that you found the interview helpful

[@BooksWala]

best interview till now

[@sumitmittal07]

absolutely

[@VishalKumar-ie3gr]

Very informative.

[@sumitmittal07]

thank you

[@DataJourneyHuub]

Thank you as always 🙏🏻

[@sumitmittal07]

Always happy to share good content for all the data enthusiasts

[@ameygoesgaming8793]

The series is amazing , I am learning a lot. Please keep making it

[@sumitmittal07]

Happy to hear that!

[@rajdeepreddycharla]

Been watching the interviews everyday. Definitely helpful. IMO based on the performance of the interviewee feedback at the end would be great.

[@sumitmittal07]

we will add it in upcoming ones

[@boreddymanohar8339]

It was awesome

[@sumitmittal07]

Glad that you are finding the mock interviews informative.

[@gouthambheema1779]

Very helpful ❤

[@sumitmittal07]

Glad it was helpful!

[@bhushanthoke858]

Realistic interview ❤

[@sumitmittal07]

glad that you enjoyed it

[@salilagarwal7915]

Best interview, every interviewer should give feedback at the end. it can be more helpful.

[@sumitmittal07]

noted the suggestion, will incorporate it from the upcoming interviews

[@abhirambandi6091]

Too technical man🔥

[@Visionpro33]

What is the common salary range for Big data developer (4yrs) ?

[@PurbaragPChoudhury]

Most of the interviews seem to focus on junior and mid-level DE. I’d also appreciate some video for Senior DE role.

[@sumitmittal07]

let me know if you would like to attend?

[@vijaykumarvc3037]

Sumit Sir, Please share any mock interview which will cover Azure cloud alone if possible

[@sumitmittal07]

As soon as possible, noted

[@chandranshuyadav3515]

Very nice interview sir can we have an interview for someone who is having more than 5 years of experience and is looking to join as an Architect level or Team Lead, Manager in Data Engineering world.

[@sumitmittal07]

Definitely, there would be many more such videos released as part of this interview series. Will try to consider all possible scenarios

[@ThanmayiTan]

I would like to know the answer for the question, "why in spark why we will get duplicate of the column on which join is performed, whereas in sql we do not get that duplicate ?" From my limited knowledge, I know that in both spark and sql joins we will get the duplicate columns when join is performed on two tables. I would really appreciate it if you could please let me know the correct answer.

[@sumitmittal07]

The reason behind this lies in the fundamental differences between Spark and traditional SQL databases. Spark performs parallel processing to handle partitioned data distributed across different nodes in the cluster. It is inevitable that there would be duplicates while performing joins on such datasets in this context. In contrast, traditional SQL databases usually operates on a single machine or a tightly coupled cluster and ensures uniqueness by preventing the duplicated from being generated in the final results

[@ThanmayiTan]

@@sumitmittal07Thankyou so much ! That really makes sense now.

[@ameygoesgaming8793]

I would like to attend interview, let me know what is the process?

[@sumitmittal07]

Please fill this form - forms.gle/UMpNCZvAHgoLvvuJ6

[@sandyreddy3937]

He is looking somewhere for answers😮😅..

[@sandyreddy3937]

It is scripted interview