Azure Data Engineer Mock Interview | PySpark | Delta Live Tables| Managerial

𝐓𝐨 𝐞𝐧𝐡𝐚𝐧𝐜𝐞 𝐲𝐨𝐮𝐫 𝐜𝐚𝐫𝐞𝐞𝐫 𝐚𝐬 𝐚 𝐂𝐥𝐨𝐮𝐝 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫, 𝐂𝐡𝐞𝐜𝐤 trendytech.in/?src=youtube&su... for curated courses developed by me.
I have trained over 20,000+ professionals in the field of Data Engineering in the last 5 years.
𝐖𝐚𝐧𝐭 𝐭𝐨 𝐌𝐚𝐬𝐭𝐞𝐫 𝐒𝐐𝐋? 𝐋𝐞𝐚𝐫𝐧 𝐒𝐐𝐋 𝐭𝐡𝐞 𝐫𝐢𝐠𝐡𝐭 𝐰𝐚𝐲 𝐭𝐡𝐫𝐨𝐮𝐠𝐡 𝐭𝐡𝐞 𝐦𝐨𝐬𝐭 𝐬𝐨𝐮𝐠𝐡𝐭 𝐚𝐟𝐭𝐞𝐫 𝐜𝐨𝐮𝐫𝐬𝐞 - 𝐒𝐐𝐋 𝐂𝐡𝐚𝐦𝐩𝐢𝐨𝐧𝐬 𝐏𝐫𝐨𝐠𝐫𝐚𝐦!
"𝐀 8 𝐰𝐞𝐞𝐤 𝐏𝐫𝐨𝐠𝐫𝐚𝐦 𝐝𝐞𝐬𝐢𝐠𝐧𝐞𝐝 𝐭𝐨 𝐡𝐞𝐥𝐩 𝐲𝐨𝐮 𝐜𝐫𝐚𝐜𝐤 𝐭𝐡𝐞 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰𝐬 𝐨𝐟 𝐭𝐨𝐩 𝐩𝐫𝐨𝐝𝐮𝐜𝐭 𝐛𝐚𝐬𝐞𝐝 𝐜𝐨𝐦𝐩𝐚𝐧𝐢𝐞𝐬 𝐛𝐲 𝐝𝐞𝐯𝐞𝐥𝐨𝐩𝐢𝐧𝐠 𝐚 𝐭𝐡𝐨𝐮𝐠𝐡𝐭 𝐩𝐫𝐨𝐜𝐞𝐬𝐬 𝐚𝐧𝐝 𝐚𝐧 𝐚𝐩𝐩𝐫𝐨𝐚𝐜𝐡 𝐭𝐨 𝐬𝐨𝐥𝐯𝐞 𝐚𝐧 𝐮𝐧𝐬𝐞𝐞𝐧 𝐏𝐫𝐨𝐛𝐥𝐞𝐦."
𝐇𝐞𝐫𝐞 𝐢𝐬 𝐡𝐨𝐰 𝐲𝐨𝐮 𝐜𝐚𝐧 𝐫𝐞𝐠𝐢𝐬𝐭𝐞𝐫 𝐟𝐨𝐫 𝐭𝐡𝐞 𝐏𝐫𝐨𝐠𝐫𝐚𝐦 -
𝐑𝐞𝐠𝐢𝐬𝐭𝐫𝐚𝐭𝐢𝐨𝐧 𝐋𝐢𝐧𝐤 (𝐂𝐨𝐮𝐫𝐬𝐞 𝐀𝐜𝐜𝐞𝐬𝐬 𝐟𝐫𝐨𝐦 𝐈𝐧𝐝𝐢𝐚) : rzp.io/l/SQLINR
𝐑𝐞𝐠𝐢𝐬𝐭𝐫𝐚𝐭𝐢𝐨𝐧 𝐋𝐢𝐧𝐤 (𝐂𝐨𝐮𝐫𝐬𝐞 𝐀𝐜𝐜𝐞𝐬𝐬 𝐟𝐫𝐨𝐦 𝐨𝐮𝐭𝐬𝐢𝐝𝐞 𝐈𝐧𝐝𝐢𝐚) : rzp.io/l/SQLUSD
30 INTERVIEWS IN 30 DAYS- BIG DATA INTERVIEW SERIES
This mock interview series is launched as a community initiative under Data Engineers Club aimed at aiding the community's growth and development
Our highly experienced guest interviewer, Ankur Bhattacharya, / ankur-bhattacharya-100... shares invaluable insights and practical advice coming from his extensive experience in the Big Data Domain
Our expert guest interviewee, Srinivaasan AK, / srinivaasan-ak-609b8a195 has a remarkable approach to answering the coding round interview questions on Azure Cloud Services.
Link of Free SQL & Python series developed by me are given below -
SQL Playlist - • SQL tutorial for every...
Python Playlist - • Complete Python By Sum...
Don't miss out - Subscribe to the channel for more such informative interviews and unlock the secrets to success in this thriving field!
Social Media Links :
LinkedIn - / bigdatabysumit
Twitter - / bigdatasumit
Instagram - / bigdatabysumit
Student Testimonials - trendytech.in/#testimonials
Discussed Questions with Timestamp
1:16 Data engineering experience, current project, tech stack?
3:11 Explain Data pipeline architecture of current project?
5:45 What are your Roles and responsibilities?
7:14 How do you manage lately arriving records in Delta Live tables?
7:58 Explain cluster configuration for handling 1000 GB of data?
9:47 Experience with latency bottlenecks and resolution?
11:16 How much data do you handle?
11:33 How do you handle slowness or out of memory issues due to increased load?
13:52 What is an optimized command in PySpark and how does it work?
14:47 What are the join strategies in PySpark?
16:00 How would you join two large tables to avoid shuffling?
17:53 What components would you use to design a pipeline for on-premise CDC and live dashboard updates?
21:27 How would you implement CDC from on-prem to streaming data?
24:10 SQL Coding question
Tags
#mockinterview #bigdata #career #dataengineering #data #datascience #dataanalysis #productbasedcompanies #interviewquestions #apachespark #google #interview #faang #companies #amazon #walmart #flipkart #microsoft #azure #databricks #jobs

[@imranhossain1660]

Optimize is a perfomance optimization technique available in delta lake table. Whenever we perform any kind of DML operations in Delta table each and every time it generates a new records. Over a period of time it generates a huge number of small files and it is kind of overhead for the delta engine to effectively perform the execution of our query as it eventually increases our resource usage such as i/o read/write and the computaion. Optimize command helps us to combine these small files to a larger file which eventually improves the performance of the delta table. As after this optimize operation, delta table refers to this latest snapshot file in order to retrieve results whever we query our table. And we can delete the obselete small files and free up the space with the help of vaccum command.

[@user-jg2tn1wb3d]

Thank u sumit sir

[@prakashtripathi270]

Thank you Sumit sir for arranging such a insightful session..

[@sumitmittal07]

Always happy to help the community!

[@MsMohanj]

Is it the join is correct or we can go for left join

[@maheshtiwari2297]

Hello sir, i have interview for big data/Etl developer at amazon please guide me for that.

[@codinggeek9992]

Less questions from Azure Synapse....

[@user-jg2tn1wb3d]

Sir, need one video to know how bussiness requirement is, and how data engg gets the bussiness requirement and working strategy

[@sumitmittal07]

Noted. Will have a session around this aspect!

[@tahiliani22]

I would like to add to this. If I am understanding correctly, @user_j% is talking about questions like Design Yelp but from a Database perspective.

[@zaffer2024]

Tough sql question, 😭

[@user-nv6ho7uk8b]

Check this. create table oldest_youngest(person varchar(10),type varchar(20),age int); insert into oldest_youngest values ('A1','ADULT',54), ('A2','ADULT',53), ('A3','ADULT',52), ('A4','ADULT',58), ('A5','ADULT',54), ('C1','CHILD',20), ('C2','CHILD',19), ('C3','CHILD',22), ('C4','CHILD',15); WITH ranked_adult AS ( SELECT person as adult, ROW_NUMBER() OVER(ORDER BY age desc) as r_a FROM oldest_youngest where type = 'ADULT' ), ranked_child as ( SELECT person as child, ROW_NUMBER() OVER(ORDER BY age asc) as r_c FROM oldest_youngest where type = 'child' ) SELECT adult,child FROM ranked_adult a left join ranked_child c on a.r_a = c.r_c

[@user-nv6ho7uk8b]

create table oldest_youngest(person varchar(10),type varchar(20),age int); insert into oldest_youngest values ('A1','ADULT',54), ('A2','ADULT',53), ('A3','ADULT',52), ('A4','ADULT',58), ('A5','ADULT',54), ('C1','CHILD',20), ('C2','CHILD',19), ('C3','CHILD',22), ('C4','CHILD',15); WITH ranked_adult AS ( SELECT person as adult, ROW_NUMBER() OVER(ORDER BY age desc) as r_a FROM oldest_youngest where type = 'ADULT' ), ranked_child as ( SELECT person as child, ROW_NUMBER() OVER(ORDER BY age asc) as r_c FROM oldest_youngest where type = 'child' ) SELECT adult,child FROM ranked_adult a left join ranked_child c on a.r_a = c.r_c