Master Databricks and Apache Spark Step by Step: Lesson 27 - PySpark: Coding pandas UDFs

[@shriramsudrik7568]

I was looking for Pandas UDF and I am glad that I found your videos. 10/10 to you Bryan!

[@BryanCafferky]

Great! Glad it helps. Please let others know about my channel.

[@JoaoOliveira-rk8gv]

You are awesome Bryan. Thank you so much for all this quality content for f***** free. So much respect

[@BryanCafferky]

YW. Thanks for watching and please let others know about my channel.

[@IvanPerez-vk6dj]

Hi Bryan. Thanks a lot for your time and effort doing these series. All of your content is pure gold. Not only for the level of detail in the explanations, but also for how well structured they are. You have a great talent explaining things. I really enjoy your channel, congratulations! A question ... In cell 15 of this notebook, the type hints of the UDF shouldn't be Iterator[int] ? I think we are passing a pd.series right? which in this case is a column of ints, so what the function receives is an iterator if ints ... Not sure if I'm right. Live longer and prosper dear Bryan! 🖖🏼

[@BryanCafferky]

Thanks for that. I'll have to go back and look at that.

[@mohamedalryah287]

thanks a lot, Mr. Bryan for these videos, they are very informative and detailed! thanks for putting in time and effort

[@tzett0011]

your videos are really awesome!

[@BryanCafferky]

Thanks.

[@ryanjadidi8622]

Dont you think the first way of calling the panda udf is faster than iterator because its using vectorization?

[@BryanCafferky]

Good question. Please try both and get the timings. I'd love to hear what you discovered. Thanks

[@haneulkim4902]

Amazing tutorial! so we can not do more processing in between function and its return only when its `series -> series`? So I can't initialize model with broadcasted weights inside function when using pandas_udf that receives series and return series?

[@cssensei610]

The modeling example was hard to follow:-- Can you show me a pyspark groupBy and K-Means scikit model inside pandas_udf?

[@BryanCafferky]

Yeah. There are very few examples of the Spark pandas UDF api. See here for a blog about it databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html

[@cssensei610]

I’ve read the docs already would be much appreciated if you could solve the said example

[@BryanCafferky]

@@cssensei610 That's very specific use case. I like to do videos that have a board appeal but thanks for the suggestion.

[@Gerald-iz7mv]

hi nice video - do you have another video which covers Vectorized UDF?

[@dchandrateja]

Hi Bryan, it was a great explanation. Is it possible to write functions with spark context, like writing spark code in a fucntion which has a bunch of transformation fucntions to calculate a value. That would really solve my problem. I tried writing but I get this error “It appears that you are attempting to reference sparkcontext from a broadcast variable, action or transformation. SC can only be used on the driver, not in code that run on workers) Thank you in advance

[@ditalish]

thanks

[@severalpens]

Hi Bryan, I'm loading a bunch of JSON files with nested objects and arrays using Autoloader. This part works well but I was looking to create a scalar UDF that could parse and extract values from the resulting 'struct' cells. eg getTimeStamp(json_field) where json_field = {Id: 23, name: "foo", timestamp: 123413}. I know I can query within struct field but I've got complex requirements that I'd like to encase in a UDF.

[@BryanCafferky]

Cool. That's a bit beyond what can be written in a comment.

[@severalpens]

@@BryanCafferky No problem! I found your Patreon I might try there. Alternatively if you're taking requests I think this would be a good youtube video topic? Thanks for the videos. I'm suddenly neck deep in Databricks world and these are helping.

[@BryanCafferky]

@@severalpens I am starting a bi-weekly Data Lakehouse Support group using Meetup.com for Patreon supporters. Maybe that would help you?