PySpark Tutorial 6: PySpark DataFrame Functions | PySpark with Python

PySpark Tutorial 6: PySpark DataFrame Functions | PySpark with Python
GitHub JupyterNotebook: github.com/sid...
GitHub Data: github.com/sid...
About this video: In this video, you will learn how to about PySpark DataFrame Functions in pyspark
Large Language Model (LLM) - LangChain
LangChain: • LangChain Tutorial for...
Large Language Model (LLM) - LlamaIndex
LlamaIndex: • LlamaIndex Tutorial fo...
Machine Learning Model Deployment
ML Model Deployment: • ML Model Deployment us...
Spark with Python (PySpark)
PySpark: https: • PySpark with Python
Data Preprocessing (scikit-learn)
Data Preprocessing Python: • Data Preprocessing Python
Social Media Links
KZbin: / statswire
Twitter (X) : / statswire
#llamaindex #openai #llm #ai #huggingface #api #genai #generativeai #statswire #spark #pyspark #python #pythonprogramming #pythontutorial

[@r.ritika2963]

Do we have any function to find mode of particular column?

[@StatsWire]

We have to write custom function for that.

[@pritishbanerjee5517]

Hi Amir, , Great content. One doubt: can you please share how to do the sorting with desc order. Thanks.

[@StatsWire]

Hi Pritish, thank you for liking the content. Here is an example or sorting by desc order df.sort(df.department.asc() ,df.state.desc()).show(truncate=False) df.sort(col("department").asc(), col("state").desc()).show(truncate=False) df.orderBy(col("department").asc(), col("state").desc()).show(truncate=False)

[@StatsWire]

Hi Pritish, here is the link for the dataset emp_data.csv for windows function Github: github.com/siddiquiamir/Data/blob/master/emp_data.csv

[@pritishbanerjee5517]

@@StatsWire thanks a lot Amir.

[@StatsWire]

@@pritishbanerjee5517 You're welcome Pritish

[@statisticalseminarsdcmeetu8491]

❤❤

[@StatsWire]

Thank you

[@mazharalamsiddiqui6904]

Nice

[@StatsWire]

Thank you

[@bhaswatibaishya251]

How to do partitioning in csv dataset ?

[@StatsWire]

Define partitioning criteria: Determine the criteria for partitioning your dataset, such as separating training and testing data. Split the dataset: Apply the partitioning criteria to split the dataset into subsets. You can use the DataFrame API's randomSplit function to achieve this: train_ratio = 0.8 # proportion of data for training test_ratio = 1 - train_ratio # proportion of data for testing train_df, test_df = df.randomSplit([train_ratio, test_ratio], seed=42) train_df.write.format("csv").option("header", "true").mode("overwrite").save("train_dataset.csv") test_df.write.format("csv").option("header", "true").mode("overwrite").save("test_dataset.csv")

[@RangaSwamyleela]

How to get acces for those

[@StatsWire]

You will get access to all the videos. Those are scheduled videos they will go public one by one then you can access all the videos.

[@RangaSwamyleela]

Some are private

[@StatsWire]

You will get access to all the videos. Those are scheduled videos they will go public one by one then you can access all the videos.