what is Apache Parquet file | Lec-7

[@manish_kumar_1]

I said 500 GB in the video by mistake. It is supposed to be 500MB, and when dividing 500/128, we will get 4 partitions.

[@dayanandab.n3814]

All good. Samaj gaye

[@sahilsood2028]

thanks for clarification. Now it make more sense

[@FaisalMasood-q6g]

I have watched couple of videos for new skills , but your way of teaching is awesome.. No bakwa...no promotion....no stuck...no timepass...Only content you kept...which attract me more..Hope you continue in same way

[@Shubhamkumar-cq5wt]

Literally the best and most detailed video on parquet file format on yt. Thank you!

[@RiyaBiswas-r1p]

Never saw such a detailed video for parquet file, these videos are really valuable. Really appreciate the efforts put in making these videos

[@adityamaurya1585]

Hi

[@adityamaurya1585]

Hii

[@manish_kumar_1]

Directly connect with me on:- topmate.io/manish_kumar25

[@akashprabhakar6353]

Predicate pushdown - Rows filtering, Projection Pruning/Pushdown - Column filtering. Thanks for the session bro!!

[@younevano]

wow thanks for this! Makes more sense now!

[@bidyasagarpradhan2751]

Someone ask me in interview about internals of parquet file format and i couldn't answer it,Then i found your video.Now i can explain easily.Best video on parquet file format.

[@kamalprajapati9955]

This tutorial is too good. What a detailed demo based insight. I could never forget this anymore. Thank you for your efforts. These tutorials are literal Gold.

[@krishnasahoo6172]

Wah....Itta clarification...maza aa gya...Video kab khatm hui pta hi ni chala....!!! Excellent explanation.

[@shafimahmed7711]

I have gone through many youtube videos and paid videos but none of them explained like this on parquet file format. Thank you Manish Bhai .

[@roshniagrawal4777]

Such a detailed and amazing video , I am working in big data from many years but this level of detailing I never knew , thankyou so much for this detailed video, your way of teaching encourages/excites many learners. hats off

[@dayanandab.n3814]

a Complete Parquet video.. not found anywhere on youtube. Thanks Manish bhai

[@dayanandab.n3814]

I am sure, it takes huge efforts in recording 1 hr video. huge thanks for your dedication.

[@sahillohiya7658]

I love how indept you are going, please keep doing it ! We are loving it.

[@shubhamwaingade4144]

The best explanation!!! Your videos are giving me motivation and inspiration to keep learning spark!

[@anuragdwivedi1804]

i have never seen such detailed vedio on any topic,

[@dishant_22]

This is the best explanation for parquet file format available online. Thanks Manish.

[@ArunNair-z3m]

Hi Manish, thanks for such smooth explanation of not just information related to parquet but also things related to it, kudos to your efforts :D

[@ApoorvaShinde-on4ep]

This is so far the best video in which I got to know in depth knowledge of parquet and very easy to understand. Thankyou so much for sharing your knowledge.!! Could you please share the video having optimization of parquet?

[@alokkumarmohanty8454]

Hi Manish, the parquet file detail class was classic example for how to present something .if same like this avro and orc file format classes can be discussed then it would be really helpful. Nowadays the interviewer is asking on those as well

[@younevano]

Did you find any good sources for the rest?

[@kulyashdahiya2529]

GOLD, GOLD, GOLD, GOLD, GOLD,GOLD,GOLD,GOLD,GOLD,GOLD,GOLD........LIQUID GOLD. THANK YOU MANISH :)

[@ProgramwithVishal]

You are like gold mine in terms of knowledge

[@divyanshusingh3966]

Thank you bro you are helping a lot of DEs good work 🎉

[@PrabhakarKumarJha-g8t]

You are really awesome. Thank you. You are adding an infinite value.

[@lakkilakki772]

Hi Manish, great explanation of parquet i'm using parquet but didn't know about these features which made things fast how were you able to learn all this knowledge please suggest any documentation/resources to get deep understanding like this. you made my day. Thank you 😊

[@ankitachauhan6084]

the best explanation ! you are a wonderful teacher

[@asifquasmi4538]

Hats of Manish, Please keep doing the good work :)

[@afjalahamad2465]

really awesome explanation

[@rahulgupta-po4ki]

highly informative and detailed video on parquet. Thanks a lot Manish!

[@karm1311]

mzaaa aaa gya bhai

[@void_spirit8]

Highly informative video, super!

[@natarajbeelagi569]

Wow super info

[@susanthomas223]

Thank you so much for putting in so much time for making this video

[@gitanjalimadaan537]

Very good video!

[@curiositycure14]

at 21:26.....we will have 4000 row group right?

[@AyushMandloi]

Also please explain Bucketing and partitioning

[@NirajAgrawal-e6v]

Please make a video on avro file format in detail because I faced challenges when interviewers asked about avro file format questions

[@younevano]

Can you share any good resources to learn these?

[@yashkirtiyashkirti-yy2hj]

@@younevanodo reply

[@vaibhavmore7936]

Thanks for this Manish! Great Work!

[@amitpatel9670]

hey Manish been following your channel for a very long time. And thanks for the awesome videos. In the ending of this video. You said you will discuss how to optimize parquet file format but I dont see that video added in this playlist. Am I missing something?

[@coolashishful]

this video only, last 10 mins.

[@younevano]

@@coolashishful you mean to say Predicate pushdown and Projection Pruning is what optimizes the parquet file?

[@shivakrishna1743]

Very detailed awesome video!! Thanks

[@dollykushwah6352]

Hello Manish, excellent explanation, hats off to you. When will you give optimization video on parquet eagerly waiting for it

[@neelshah8247]

Excellent video. Thank you :)

[@nikhiljain8411]

How one will understand in which 1L records we need to fetch the data. Still we need to scan the complete file. Isn't it? Kindly explain

[@shubhamwaingade4144]

One doubt, I did not understand the logical partitioning completely, it resembles with file size we can set in spark config. Please help me understand it

[@prathamesh_a_k]

nice explaination brother

[@prathamesh_a_k]

can you make one video on ORC also

[@krunalsuthar1420]

Please make video on ORC and Avro as well

[@younevano]

Can you share any good resources to learn these?

[@lucky_raiser]

bhai, maza aa gya, thanks bro

[@AnubhavTayal]

Hi Manish, thank you for the information. Please can you elaborate whats the default value of 128 MB and when we have 500 GB data how does that convert to 4 row groups? Thank you

[@manish_kumar_1]

500 mb not gb. 500 divided by 128 I.e 4. 4 block of data will be created. So the thing is we have a default block size of 128 MB in hdfs and multiple cloud service provider also use the same block size. So let say if you have 140 mb data that means one partition will be of 128 mb and next partition will be having just 12 mb of data.

[@AnubhavTayal]

@@manish_kumar_1 thank you so much!

[@gouravchourasia9515]

You should have spent more time and detail on projection pruning and pushdown

[@tnmyk_]

Where is the nested JSON video? You said you will make a separate video on it in the previous lecture "how to read json file in Pysaprk"

[@manish_kumar_1]

Lec 23

[@harshtalwar9615]

Superb bro … very helpful thanks 🙏🏻

[@deeksha6514]

Thanks! for this masterpiece

[@mayankverma8989]

Sir can you please provide the PDF of the entire course the complete notes

[@nileshgodase1007]

Nested json to data frame explain kijiyee na

[@shwetatejpalshah2333]

You are a 💎

[@sumitchoubey1284]

unable to install parquet-tools. can you help or point n right direction

[@younevano]

Use ChatGPT to do some debugging. In my case (Mac OS), I had to use 'brew install parquet-cli' and later 'parquet show file-path'. And later installed pyarrow in a virtual environment coz it needs old python version compared to my global python version. Hope that helps!

[@patilsahab4278]

hii bro each row grop stores 128mb or 128 gb data you told 128mb bur for for 500gb you told 4 row groups you are talking about 500mb or 500gb

[@prashantmane2446]

error occured while processing file ?? yeh error continous hai...help:

[@ashutoshkumarsingh3337]

what a gem you are

[@navjotsingh-hl1jg]

bhai 500gb data mein 4 row kyun rakhe gaye and manish bhai 128mb hota hai . aap explain kar sakte ho aisa kyun bhai

[@SanjayShukla-qh8xj]

Bro nested json video is not available. Please provide in depth if feasible.

[@manish_kumar_1]

It is there. Practical wale playlist me hoga. Ek baar check kar lijiye

[@dineshboliwar9545]

sir please make short video to downloadd and install parquet tool

[@manish_kumar_1]

Sure

[@debopower2009]

Very nice.

[@MCAMadeEasy]

Manish bhai, nested json?

[@royalkumar7658]

Null kaise write hota hai disk pe??

[@pankajsolunke3714]

Hi manish sir,Thanks for bringing such valuable info ..I have a question like how can we handle schema evaluation in parquet

[@manish_kumar_1]

I didn't get you

[@mohitdaxini3067]

I think he wants to about shema evolution

[@sankuM]

@@manish_kumar_1 I think @pankajsolunke3714 is asking how to handle schema evolution in parquet if we can?

[@vipulbornare34]

Thankyou 😊

[@AkshayPawar-c8j]

Thanks Manish 🙂

[@Wandering_words_of_INFJ]

Manish, if we are writing parquet by making the files already sorted in asc or desc then the process of retrieval of data would be faster right? Because in row_number's meta data would have min and Max value in a certain range? Please correct me if I am wrong.

[@pankajjagdale2005]

informative Thanks

[@ShekharBhide]

sir, parquet file download nahi ho raha he github se

[@vaibhavshanbhag5016]

@manish_kumar_1 Sir kya mast content banaya, maja aa gaya, thank you!

[@avisinha2844]

Hello Manish, i really like your videos, thanks for the efforts you put in. Have a question, can you please tell a good tutorial/course that we can go through to get really good at pyspark, if not a single resource then what are the various resources that we can go through to get good at pyspark coding.

[@manish_kumar_1]

You don't need a course. Still if you want to go for a course then you can buy a udemy course by Prashant Pandey titled pyspark for Beginner. Rest depends on you ki how much questions you want to solve. Solve more problems rather than running behind multiple courses. Practice is the key to success not a number of course you have done.

[@sankuM]

sparkbyexamples is the RESOURCE we need for practice!! :)

[@royalkumar7658]

Where can we practice spark from?

[@royalkumar7658]

​@@manish_kumar_1 where can we practice spark from??

[@manish_kumar_1]

@@royalkumar7658 Leet code se. Aap playlist start se follow kijiye tab Pata chal jayega kaha se and kaise

[@pramod3469]

Thanks Manish

[@akashchandapureakashchanda1842]

bro you have installed java to read parquet file in command prompt

[@manish_kumar_1]

Not particularly. I had in my laptop already installed

[@younevano]

you can install parquet-cli alternatively if it's mac OS

[@Marcopronto]

Hi Manish, In the last video, you told that u will explain about nested json in further videos. Where can i find that?

[@manish_kumar_1]

I have not done yet. I will try to make one soon.

[@piyushkrvlog]

Yaar bhaiya wo bana do pls. Industry to usi pe chal ri

[@KavyaPristha]

Please drop your twitter or X account. I will promote you. You are the only person on youtube who is actually teaching something useful in DE filed. That TOO IN HINDI. Great work and great effort. God Bless You !!

[@manish_kumar_1]

I don't have ex😂😂. Sorry I mean this X

[@KavyaPristha]

@@manish_kumar_1 Hahaha. Please create one than. It pays better than KZbin

[@manish_kumar_1]

@@KavyaPristha oh is it. I did not know about this.

[@coolraviraj24]

best

[@rpraveenkumar007]

Hi Manish, what is projection pruning? Unable to find it on Google. Or is it Partition Pruning*? Can you please explain/clarify?

[@manish_kumar_1]

Projection pushdown Hota hai jisme columns ki pruning hoti hai. So Projection pushdown ya Projection pruning same hai.

[@rpraveenkumar007]

@@manish_kumar_1 thanks for clarifying!

[@shadabalam17]

Do i need to install python first before downloading parquet tool?

[@manish_kumar_1]

Think so

[@younevano]

Yes

[@ajaywade9418]

21:25 500 GB or 500 Mb ?

[@piyushkrvlog]

500 mb

[@aravind5310]

Your content is good.Why don't you do videoes in English.

[@manish_kumar_1]

english nahi aati hai 😒. Just joking, I may record a session in future but not for now.

[@izahmad90]

kzbin.info/www/bejne/sH6VgHR3q698qrM&ab_channel=knowledgeEpicenter (We are making videos for those people for whom no one is making videos.)

[@ranvijaymehta]

Thanks sir

[@dineshboliwar9545]

anybody help me please i cant read parquet file using command prompt

[@manish_kumar_1]

Koi issue nahi hai. Aap direct databricks me read kar lijiye. Ek baar video ko bas sahi se dekh lijiyega

[@dineshboliwar9545]

@manish_kumar_1 databricks me kr liya h command prompt ka nhi ho rha h

[@younevano]

you can install parquet-cli alternatively if it's mac OS

[@yogesh9992008]

Cmd-parquet-tool issue

[@younevano]

Use ChatGPT to do some debugging. In my case (Mac OS), I had to use 'brew install parquet-cli' and later 'parquet show file-path'. And later installed pyarrow in a virtual environment coz it needs old python version compared to my global python version. Hope that helps!

[@dataplumberswithajay]

Example of Modi ji for finding age >18 was highlight of the video

[@manish_kumar_1]

😂😂

[@radheshyama448]

😇

[@yogesh9992008]

Stage failure error show

[@sankuM]

Hey @manish_kumar_1, I was able to use the modes (append, overwrite, etc.) using this command: df.write.option("header", first_row_is_header) \ .option("sep", delimiter) \ .mode("Overwrite") \ .csv(file_location) All other ways of writing is returning error on Databricks if the file exists.. even if we're trying to append the data..! :| Unsure why is this happening...! :\

[@manish_kumar_1]

Same here. May be due to community edition. In production environment it does work

[@sankuM]

@@manish_kumar_1 oh..okay! Still weird, though!!! I'm yet to try databricks in production..

[@khadarvalli3805]

❤

[@starky4910]

ok sir ni manunga notebook apse 🤥😔

[@manish_kumar_1]

Thanks 😂

[@HimanshuSingh-yj2wh]

on disk level, how csv json store their data bcolumn based or row based ?

[@manish_kumar_1]

Row based

[@HimanshuSingh-yj2wh]

@manish_kumar_1 thanks

[@DevSharma_31]

import pyarrow as pa import pyarrow.parquet as pq parquet_file = pq.ParquetFile(r'C:\Users\DELL\Desktop\part-r-00000-1a9822ba-b8fb-4d8e-844a-ea30d0801b9e.gz.parquet') parquet_file.metadata parquet_file.metadata.row_group(0) parquet_file.metadata.row_group(0).column(0) parquet_file.metadata.row_group(0).column(0).statistics Not able to see any output with this file. Not sure why

[@manish_kumar_1]

Error v nhi aa rha?

[@coolashishful]

@@manish_kumar_1 sorted

[@chiranjivmansis1415]

Awesome explanation

[@wellwisher7333]

Thanks Sir