How to write dataframe to disk in spark | Lec-8

[@jai3863]

The problem arose because you used .option("mode", "overwrite"), which is meant for reading data. For writing data, like in your case, use .mode("overwrite"). I used this and it worked fine - write_df = read_df.repartition(3).write.format("csv")\ .option("header", "True")\ .mode("overwrite")\ # Using .mode() instead of .option() for overwrite mode .option("path", "/FileStore/tables/Write_Data/")\ .save() Ran dbutils.fs.ls("/FileStore/tables/Write_Data/") and it showed the entries too, post-repartitioning of the data.

[@manish_kumar_1]

Yes we will have to use .mode function. I did face that again while I was shooting video for projects and then I found that

[@manish_kumar_1]

Directly connect with me on:- topmate.io/manish_kumar25

[@kulyashdahiya2529]

Best Teacher, You will have millions of subscribers soon :)

[@theprachidhiman]

Best course, best teacher

[@shubne]

loving this series. Eagerly waiting for the next video on Bucketing and partitioning. Please make video on Optimization and skewness.

[@easypeasy5523]

final_transformation.repartition(4).write.format("csv")\ .option("header", True)\ .mode("overwrite")\ .save("/FileStore/tables/Transformed_data_12_08_2024") Write code syntax to overwrite the current data in spark

[@Abhishek_Dahariya]

I never find this much information and easiest explanation. Thank you

[@QuaidKhan1]

real teacher🥰

[@abrarsyed9680]

Hi @MANISH KUMAR Sir, At 10:28 why there is 3 extra files _success, _committed and _started apart from this three repartition CSVs ? this success committed and started file are not .csv files why they are there is new folder ? Could you please explain this in reply, for what purpose this files are there?

[@younevano]

It's throwing an error because, .option("mode", "overwrite") though can work sometimes to overwrite files, is less reliable and might not be consistently applied across all formats or Spark versions. Using .mode("overwrite") is more standard and explicit, reducing the chances of errors or unexpected behavior. It is the standard and recommended way to specify the write mode in Spark when you want to overwrite existing data at the specified path. It’s part of the .mode() method, which accepts 4 write modes/options like "overwrite", "append", "ignore", and "error".

[@rishav144]

Very nice explanation .

[@sauravroy9889]

Nice❤❤❤

[@NahushAgrawal]

correct code that will work df.repartition(3).write.format("csv")\ .option("header", "true")\ .mode("overwrite")\ .save("/FileStore/tables/csv_write/")

[@pavitersingh4698]

great

[@girishdepu4148]

.mode("overwrite") worked for me. it replaced the file in the folder.

[@akashprabhakar6353]

AWESOME

[@isharkpraveen]

i Didnt understood that why we used header option in write? Normally we use in read right?

[@vaibhavdimri7419]

Hello sir, Great lecture. I am facing one problem, in the end part where you were partitioning, I am not getting 3 files. Just getting one file with this output [FileInfo(path='dbfs:/FileStore/tables/csv_write_repartition/*/', name='*/', size=0, modificationTime=0)]. Kindly help me.

[@rampal4570]

should we enroll any courses other site or bootcamp for data engineer or not please reply bhaiya

[@manish_kumar_1]

No need. Whatever you need to become DE is available for free. In roadmap wala video you can find all the resources and technology that is required to be a DE

[@vsbnr5992]

How much lectures are remaining for completing spark playlist

[@rishav144]

12-15 more

[@manish_kumar_1]

Yes it will be around 20-25 lecture

[@vsbnr5992]

@@manish_kumar_1 sir can u please complete the playlist in upcoming month..

[@raviyadav-dt1tb]

If we are using error mode but our file path not is available thek it will save file or not ?

[@younevano]

It will save, if the path does not exist ,or is empty and doesn't have data, Spark will save the file as usual, creating the path if necessary. Spark will throw an error and won’t overwrite or append data only when the target path already contains data.

[@sankuM]

There is "Error" writing mode also, correct? Or ErrorIfExists is same as Error mode?

[@lucky_raiser]

did you find the root cause of mode error?

[@sankuM]

@@lucky_raiser I didn't get it..!

[@lucky_raiser]

I mean, while writing mode = overwrite, and running the code, first time it will create a file but next time we run the code then it is not overwritting the previous file and giving error as file already exists, ideally it should replace the previous file with new one.

[@sankuM]

@@lucky_raiser Yes, there was some bug in the community edition! I had commented on other video about it and @manish_kumar_1 also confirmed that he faced the same issue..! I'm not able to recollect how we overcome that, sorry!!

[@younevano]

@@lucky_raiser It's throwing an error because, .option("mode", "overwrite") though can work sometimes to overwrite files, is less reliable and might not be consistently applied across all formats or Spark versions. Using .mode("overwrite") is more standard and explicit, reducing the chances of errors or unexpected behavior. It is the standard and recommended way to specify the write mode in Spark when you want to overwrite existing data at the specified path. It’s part of the .mode() method, which accepts 4 write modes/options like "overwrite", "append", "ignore", and "error".

[@Jobfynd1]

Bro make data engineer project from scratch to end plz ❤

[@manish_kumar_1]

Sure. I have explained in one video that may help you to complete your project by your own

[@NY-fz7tw]

i am receiving error stating that df is not defined

[@krishnakumarkumar5710]

Maneesh Bhai SQL ke kaise topics imp hai interview ke liye batayiye naaa

[@manish_kumar_1]

Join, group by, windows functions, cte, subquery

[@krishnakumarkumar5710]

@@manish_kumar_1 thanks for reply..

[@utkarshaakash]

Why didn't you complete the playlist?

[@stevedz5591]

How we can optimize dataframe write to csv when its a large file it takes time to write. code: df.coalesce(1).write()....only one file needed in destination path..

[@manish_kumar_1]

I don't think you can do much in this case. All the optimization techniques you can use before final dataframe creation. Since you are merging all partition at the end in to one and writing it so you don't have option to optimize it. If it is allowed you can partition or bucket your Data so whenever you read that written dataframe next time it will query faster

[@ATHARVA89]

Save vs saveastable kab use kiya jata h

[@manish_kumar_1]

Save me data as a file save hogi. Save as table me data to as a file hogi hogi. But Hive metastore me entry hogi and when you run select * from table then it will look like it has been saved as a table

[@vishaljare163]

@@manish_kumar_1 ya correct.when we save data as SaveAsTable() data get saved.but under the hood this is file.but we can able to write sql queries on top of that.

[@DsSarangi23]

i have attribute error in this video when I write df.write.format

[@syedhashir5014]

how to downlaod those csv files

[@patilsahab4278]

i am getting this error can anyone help me please write_df = df.repartition(3).write.format("csv")\ .option("header", "True")\ .mode("overwrite")\ .option("path", "/FileStore/tables/write-1.csv/")\ .save() AttributeError: 'NoneType' object has no attribute 'repartition

[@itofficer_7]

while creating df did you use .show() in the end just remove it bcoz most probably it is return None from there df = spark.read.format("csv")\ .option("header","true")\ .option("mode","PERMISSIVE")\ .load("dbfs:/FileStore/tables/write_data_file.csv") df.write.format("csv")\ .option("header","true")\ .mode("overwrite")\ .option("path","/dbfs:/FileStore/tables/csv_write/")\ .save()

[@BorsepunamBorse]

getting error while writing the file:-AttributeError: 'NoneType' object has no attribute 'Write'

[@manish_kumar_1]

Don't use show with variable assigned. When you assign df in another variable with show in it it becomes none

[@NY-fz7tw]

NameError df is not defined