3. pyspark interview questions and answers for experienced | databricks interview question & answer

#Databricks #PysparkInterviewQuestions #deltalake
Azure Databricks #spark #pyspark #azuredatabricks #azure
In this video, I discussed PySpark scenario based interview questions and answers.
PySpark advanced interview questions answers?
databricks interview questions and answers?
Create dataframe:
======================================================
data1=[(1,"Steve"),(2,"David"),(3,"John"),(4,"Shree"),(5,"Helen")]
data2=[(1,"SQL",90),(1,"PySpark",100),(2,"SQL",70),(2,"PySpark",60),(3,"SQL",30),(3,"PySpark",20),(4,"SQL",50),(4,"PySpark",50),(5,"SQL",45),(5,"PySpark",45)]
schema1=["Id","Name"]
schema2=["Id","Subject","Mark"]
df1=spark.createDataFrame(data1,schema1)
df2=spark.createDataFrame(data2,schema2)
display(df1)
display(df2)
-----------------------------------------------------------------------------------------------------------------------
df_join=df1.join(df2,df1.Id==df2.Id).drop(df2.Id)
display(df_join)
---------------------------------------------------------------------------------------------------------------------
from pyspark.sql.functions import *
df_per=df_join.groupBy('Id','Name').agg(
(sum('Mark')/count('*')).alias('Percentage')
)
display(df_per)
-------------------------------------------------------------------------------------------------------------------
============================================================
37. schema comparison in pyspark | How to Compare Two DataFrames in PySpark | pyspark interview:
• 37. schema comparison ...
Learn PySpark, an interface for Apache Spark in Python. PySpark is often used for large-scale data processing and machine learning.
Azure Databricks Tutorial Platlist:
• Azure Databricks Tutorial
Azure data factory tutorial playlist:
• Azure Data factory (adf)
ADF interview question & answer:
• adf interview question...
1. pyspark introduction | pyspark tutorial for beginners | pyspark tutorial for data engineers:
• 1. pyspark introductio...
2. what is dataframe in pyspark | dataframe in azure databricks | pyspark tutorial for data engineer:
• 2. what is dataframe i...
3. How to read write csv file in PySpark | Databricks Tutorial | pyspark tutorial for data engineer:
• 3. How to read write c...
4. Different types of write modes in Dataframe using PySpark | pyspark tutorial for data engineers:
• 4. Different types of ...
5. read data from parquet file in pyspark | write data to parquet file in pyspark:
• 5. read data from parq...
6. datatypes in PySpark | pyspark data types | pyspark tutorial for beginners:
• 6. datatypes in PySpar...
7. how to define the schema in pyspark | structtype & structfield in pyspark | Pyspark tutorial:
• 7. how to define the s...
8. how to read CSV file using PySpark | How to read csv file with schema option in pyspark:
• 8. how to read CSV fil...
9. read json file in pyspark | read nested json file in pyspark | read multiline json file:
• 9. read json file in p...
10. add, modify, rename and drop columns in dataframe | withcolumn and withcolumnrename in pyspark:
• 10. add, modify, renam...
11. filter in pyspark | how to filter dataframe using like operator | like in pyspark:
• 11. filter in pyspark ...
12. startswith in pyspark | endswith in pyspark | contains in pyspark | pyspark tutorial:
• 12. startswith in pysp...
13. isin in pyspark and not isin in pyspark | in and not in in pyspark | pyspark tutorial:
• 13. isin in pyspark an...
14. select in PySpark | alias in pyspark | azure Databricks #spark #pyspark #azuredatabricks #azure
• 14. select in PySpark ...
15. when in pyspark | otherwise in pyspark | alias in pyspark | case statement in pyspark:
• 15. when in pyspark | ...
16. Null handling in pySpark DataFrame | isNull function in pyspark | isNotNull function in pyspark:
• 16. Null handling in p...
17. fill() & fillna() functions in PySpark | how to replace null values in pyspark | Azure Databrick:
• 17. fill() & fillna() ...
18. GroupBy function in PySpark | agg function in pyspark | aggregate function in pyspark:
• 18. GroupBy function i...
19. count function in pyspark | countDistinct function in pyspark | pyspark tutorial for beginners:
• 19. count function in ...
20. orderBy in pyspark | sort in pyspark | difference between orderby and sort in pyspark:
• 20. orderBy in pyspark...
21. distinct and dropduplicates in pyspark | how to remove duplicate in pyspark | pyspark tutorial:
• 21. distinct and dropd...

[@ssunitech6890]

df_final=df_per.select('*', (when(df_per.Percentage>=70,'Distintion') .when((df_per.Percentage=60),'First Class') .when((df_per.Percentage=50),'Second Class') .when((df_per.Percentage=40),'Third Class') .when((df_per.Percentage

[@himangibari6591]

I didn't got how calculate perc by dividing count *

[@satishgs5355]

why we have to divide count('*')?

[@premanandramasamy]

formula is sum / number of subjects. So count used.

[@ssunitech6890]

Correct