10 PySpark Product Based Interview Questions

10 PySpark Product-Based Interview Questions
Code - github.com/sbgowtham/pyspark/...
PySpark Interview Coding PlayList - bit.ly/48H0Okg
Question 1 -00:00:00
Question 2 -00:04:24
Question 3 -00:07:36
Question 4 -00:13:35
Question 5 -00:21:29
Question 6 -00:24:28
Question 7 -00:26:36
Question 8 -00:29:12
Question 9 -00:33:13
Question 10 -00:37:01
17 Hours Big Data Engineering Full Course
Part 1 - • Big Data Engineering F...
Part 2 - • Big Data Engineering F...
KZbin- / @thedatatech
Instagram - bigdata.in
LinkedIn - linkedin.com/in/sbgowtham
Hash Tags
#bigdata #dataengineering #pyspark

[@abhishekn786]

Thanks for the video, Please continue this pyspark interview videos. Thanks Again.

[@Sandeep-bl9ji]

Really a nice explanation with a clear shot... Thanks a lot... Please keep more videos on pyspark

[@ashishlimaye2408]

Great questions!

[@vishnuvardhan9082]

Loving your channel more day by day

[@sudarshanthota3369]

awesome video

[@businessskills98]

New subscriber added

[@Ameem-rw4ir]

Bro, thanks for your effort for sharing interview based real time questions and answers. can you please share realtime streaming (kafka and pyspark) based interview based questions and answers??.

[@boreddykesavareddy669]

Instead of left anti we can use except

[@user-nl8gq3cc8t]

in interview if we solve the problems in SQL using sparksql will it be okay?

[@user-nv6ho7uk8b]

It depends, sometimes, interviewer specially asks you not to use the sql and rather use the dataframe apis.

[@selva30989]

Few guys will ask you to share code in both, spark sql and pyspark This way they can assess your pyspark and sql knowledge in single scenario based questions

[@hyderali-wl3yi]

bro, thanks for your inputs. below data is in a file. can you please help me how to handle this?. I got bit trouble on your one line string data with what i have in multiple rows with multiple delimiter. empid,fname|lname@sal#deptid 1,mohan|kumar@5000#100 2,karna|varadan@3489#101 3,kavitha|gandan@6000#102 Expected output empid,fname,lname,sal,deptid 1,mohan,kumar,5000,100 2,karan,varadan,3489,101 3,kavitha,gandan,6000,102

[@hyderali-wl3yi]

bro, thanks for your inputs. below data is a file.can you please help me how to handle this in pyspark? empid,fname|lname@sal#deptid 1,mohan|kumar@5000#100 2,karna|varadan@3489#101 3,kavitha|gandan@6000#102 Expected output empid,fname,lname,sal,deptid 1,mohan,kumar,5000,100 2,karan,varadan,3489,101 3,kavitha,gandan,6000,102

[@piyushramkar9404]

from pyspark.sql import SparkSession from pyspark.sql.functions import split, col spark = SparkSession.builder.appName("MyApp").getOrCreate() df = spark.read.format("csv").option("header", "true").option("inferSchema", "true").option("delimiter", ",").load("D:/data/employees.csv") exp_op = df.withColumn("fname", split(col("fname|lname@sal#deptid"), "\\|").getItem(0)) \ .withColumn("lname", split(split(col("fname|lname@sal#deptid"), "\\|").getItem(1),"@").getItem(0)) \ .withColumn("sal", split(split(col("fname|lname@sal#deptid"), "@").getItem(1), "#").getItem(0)) \ .withColumn("deptid", split(col("fname|lname@sal#deptid"), "#").getItem(1)) \ .select("empid","fname", "lname", "sal", "deptid") exp_op.show() #ouput +-----+-------+-------+----+------+ |empid| fname| lname| sal|deptid| +-----+-------+-------+----+------+ | 1| mohan| kumar|5000| 100| | 2| karna|varadan|3489| 101| | 3|kavitha| gandan|6000| 102| +-----+-------+-------+----+------+