how to read json file in pyspark

[@ChandanKumar-xj3md]

In an interview with Volvo, they asked about nested json files. Thanks for including this topic and a very defined explanation.

[@user93-i2k]

this is the first series where I'm learning new things which are not there in any other series...thanks Manish bhai

[@bobbygupta830]

maza aa rha hai manish bhai ye series :) yhe mera 3 series start kiya hua

[@SwetaKayal-g3u]

very good ..detail and nicely explained

[@kirtiagg5277]

I have watched multiple channel for py spark. your content is too good rather than others.:)

[@PamTiwari]

Manish Bhaiya Bahut Maja aa raha hai, I hope one day I will become Data Engineer!

[@neerajCHANDRA]

very good video series thanks for sharing knowledge.

[@HeenaKhan-lk3dg]

Thank you sharing All concept With US, We are very Thankful.

[@manishamapari5224]

You are a very good teacher you is sharing good knowledge

[@coolguy-cy8pw]

Bhaiya aap shandaar padhate ho🎉

[@rajun3810]

love you Manish bhai _ l love your content

[@mohitkeshwani456]

Aap bhut aacha padhate ho Sir....❤

[@Matrix_Mayhem]

Thanks Manish! Informative and Interesting lecture!

[@aniketraut6864]

Thank you Manish bhai for the awesome videos, thanks for giving the script.

[@yogeshsangwan8343]

best explanation... thanks..

[@AnuragsMusicChannel]

Fantastic! Thanks for the efforts you have taken to make this video buddy

[@rishav144]

great playlist for spark .

[@syedtalib2669]

When we try to read multi line json we have to provide .option("multiLine","true"), otherwise it fails with AnalysisException. Why is this not needed for nested json. it works with out this "multiline" option. Can you please tell why?

[@reshmabehera223]

Hi Manish, Thank you for the videos , its really helpful, One small question, for csv file reading corrupt data we had to create our schema with _corrupt_record column, however for json , how come it is not needed

[@AmbarGharat]

If you get the answer please let us know!

[@younevano]

One of the reasons: JSON is a semi-structured data format, meaning it allows for nested data structures and varied schemas. When reading JSON files, Spark uses a schema inference mechanism that can accommodate this flexibility. If it encounters a record that doesn't conform to the expected structure, it can easily isolate the entire record as a corrupt entry and store it in the _corrupt_record column. CSV files are structured data formats that expect a uniform schema across all rows. If a CSV record deviates from this structure (e.g., missing fields, extra fields, or improperly formatted data), Spark cannot automatically infer how to handle the corruption without an explicit schema definition. This is why you need to define your schema, including a _corrupt_record column if you want to catch those corrupt records.

[@AmbarGharat]

@@younevano Thanks

[@sonajikadam4523]

Nice explanation ❤

[@aryandash2973]

Sir, Is there any way to read multi-line corrupted JSON file. I am getting analysis exception while reading the file.

[@user93-i2k]

at 7:00, we have a beautiful explanation, why do we need json when we have csv

[@younevano]

csv stores strictly structured data while json allows semi-structured data!

[@akhiladevangamath1277]

@@younevano CSV files are Semi- Structured files, we have to just do 1, manish,26, , for the col we don't have data, but same like json, takes less bytes of memory, but we have to mention ", ," , but in json, we have liberty to not mention anything and have no problem with it

[@manish_kumar_1]

Directly connect with me on:- topmate.io/manish_kumar25

[@shreyaspurankar9736]

on 24th July 2024, I tried to upload a file in the Databricks community edition and got an error as "Upload Error". Is it happening to other guys too?

[@prashantmane2446]

yes i am getting same error tried other account too but erroor persists.

[@ravikumar-i8y7q]

I have to ingect json file or CSV file in adf then we have to create dataflow means we have to use different transformation after that we to write to databricks but The databricks part not seen any vedio. Either they are using only one databricks or adf to ingect CSV or json file , i need how to connect json file from adf and write into databricks

[@PiyushSingh-rr5zf]

Bhai apne share nahi kie nested json ka detailed video?

[@SangmeshwarBukkawar]

If in csv file 4 columns and 1 columns in the data is json or dict data how can handle this type of csv file

[@prashantmane2446]

databricks is giving error while uploading file ::: error occured while processing the file filename.csv [object object] please reply..||

[@rashidkhan8161]

How to upload yaml file in pyspark dataframe

[@rabink.5115]

while reading the data, permissive mode is always by default active, then why we need to write that piece of code?

[@manish_kumar_1]

No need to write. Code will run fine without that too

[@Uda_dunga]

bhai agr cluster terminate hojaye to kya krte h?

[@LakshmiHarika-k8o]

Hi Manish. Can you please teach us in English as well.

[@sjdreams_13615]

Just for Info, when you try to read incorrect multiline json, it raises an Analysis exception

[@manish_kumar_1]

Yes if json is not properly closed with {}then you will get error

[@saumyasingh9620]

Nested json part 2 when will come?

[@manish_kumar_1]

When I will teach explode transformation

[@saumyasingh9620]

@@manish_kumar_1 please bring soon. Thanks 😊

[@PranshuHasani]

Notebook detached ×Exception when creating execution context: java.util.concurrent.TimeoutException: Timed out after 15 seconds Getting this error while executing after creating a new cluster.

[@anirudhsingh9720]

Multiline_correct.json read krne pe mera bas first row hi show kar raha hai, bhaiya aisa kyu

[@manish_kumar_1]

Multiline TRUE karna parega in options

[@sachinragde]

can you upload multiple file

[@manish_kumar_1]

Yes

[@pankajsolunke3714]

sir thumbnail should be lec 8

[@manish_kumar_1]

I think you got confused with the spark fundamental playlist. There are two playlist and each has it's own numbering. Please check playlist and let me know if there is some mistakes in lecture numbering

[@pankajsolunke3714]

@@manish_kumar_1 Got it Thanks !!

[@swetasoni2914]

Could you share the spark second play list please @@pankajsolunke3714

[@ayushtiwari104]

arre sir dataset ka file share kr diya kro. Copy paste krwa rhe ho hr video me.

[@manish_kumar_1]

Bahut mehnat lag rhi kya bhai. Kaam me to aur jada lagega fir. Thora mehnat kar lijiye, it will help you only. Many people still get confused when I ask them to find an error in the file, Thora copy paste kijiyega to dekhiyega data and structure ko v. May be aapko pata hoga lekin sab ek level par nhi honge na.

[@ayushtiwari104]

@@manish_kumar_1 True True. I understand. Thank you.

[@kaifahmad4131]

Bhai button laga lia kar. un-professional lagta hai. baaki content is gloden

[@aditya9c]

corrupted record didn't gave me the the _corrupt_record. It is only giving 1 line record of age 20 df_corrupted_json = spark.read.format("json").option("inferSchema","true").option("mode","FAILFAST").option("multiline","true").load("/FileStore/tables/corrupted_json.json") df_corrupted_json.show()

[@debritaroy5646]

Same i have also not getting _corrupt_record. df_emp_create_scehma=spark.read.format("csv")\ .option("header","true")\ .option("inferschema","true")\ .schema(my_scehma)\ .option("badRecordsPath","/FileStore/tables/gh/bad_records")\ .load("/FileStore/tables/EMP.csv") df_emp_create_scehma.show()