40. UDF(user defined function) in PySpark | Azure Databricks

[@jerryyang7270]

You are dong a great job. Please keep p the good work. I have done all your modules in a hands on manner

[@polakigowtam183]

Thanks Maheer .. Excellent Vedio. Very Good Explanation.

[@WafaStudies]

Thank you 👍😊

[@nagatrivikramreddy]

Hi Maheer.. I have been following your pyspark videos from a while. The content is very good. Thank you for making such videos. I have a doubt in udf : Why do we need to create a user defined function? Why can't we simply create normal python functions (using def ) and use them in df.select or df.withColumn ? I was also able to register this normal python function( using def) in spark.udf.register() and use in sql statements as well. Can you explain what is the main difference between normal python function and spark udf ?

[@sahityamamillapalli6735]

User-defined functions (UDFs) can be useful when you need to perform custom operations on your data that are not already provided by the Spark SQL functions. UDFs allow you to define your own functions to apply to the data within the Spark SQL environment. Normal Python functions are not able to take advantage of the distributed computing capabilities that Spark provides and are not optimized for performance. Spark UDFs are optimized for performance and can run in parallel across multiple nodes. Spark UDFs can also be used in SQL statements, making them more versatile for data analysis.

[@tadojuvishwa2509]

also can u do videos on broad cast variable and broadcast joins,coalensce and repartititon,cache and parsist,accumulators

[@mdashfaqueali2853]

Hi, what is the scope of the UDF, like it is restricted to one session only or can be used in multiple sessions once registered.

[@hussamcheema]

Hi, After running SQL command, we get the result but can we get is as a spark dataframe in a variable?

[@excelwithsunil]

Hi, Do I need python knowledge to learn pyspark??

[@varunsingh545]

SIR JI PAYMENT IS SO MIDDLE CLASS....REMUNERATION BOLO :P

[@WafaStudies]

Haha okay 😅

[@manu77564]

Hi bhaii.. I was mailed you.... Would you please replay on that.....