The Parquet Format and Performance Optimization Opportunities Boudewijn Braams (Databricks)

The Parquet format is one of the most widely used columnar storage formats in the Spark ecosystem. Given that I/O is expensive and that the storage layer is the entry point for any query execution, understanding the intricacies of your storage format is important for optimizing your workloads. As an introduction, we will provide context around the format, covering the basics of structured data formats and the underlying physical data storage model alternatives (row-wise, columnar and hybrid). Given this context, we will dive deeper into specifics of the Parquet format: representation on disk, physical data organization (row-groups, column-chunks and pages) and encoding schemes. Now equipped with sufficient background knowledge, we will discuss several performance optimization opportunities with respect to the format: dictionary encoding, page compression, predicate pushdown (min/max skipping), dictionary filtering and partitioning schemes. We will learn how to combat the evil that is 'many small files', and will discuss the open-source Delta Lake format in relation to this and Parquet in general. This talk serves both as an approachable refresher on columnar storage as well as a guide on how to leverage the Parquet format for speeding up analytical workloads in Spark using tangible tips and tricks.
About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: databricks.com/product/unifie...
Connect with us:
Website: databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / databricks
Instagram: / databricksinc
Get insights on how to launch a successful lakehouse architecture in Rise of the Data Lakehouse by Bill Inmon, the father of the data warehouse. Download the ebook: dbricks.co/3IMxugQ

[@robinjamwal1]

Great talk, Great Teach, Excellent Tutor! One of the best presentation I have ever viewed and listened.

[@SunilBuge]

Great overview to address performance issues with storage layer design 👍

[@manishsingh455]

This content explained most of the thing and It is really amazing .

[@prabhumaganur]

The best representation of Parquet file structure!! Simply Awesome!!

[@ikercrew8830]

You prolly dont give a shit but does any of you know of a tool to get back into an Instagram account..? I was stupid forgot my login password. I appreciate any help you can give me

[@kristophergunnar9551]

@Iker Crew Instablaster :)

[@ikercrew8830]

@Kristopher Gunnar Thanks for your reply. I got to the site on google and I'm trying it out now. Looks like it's gonna take quite some time so I will get back to you later when my account password hopefully is recovered.

[@ikercrew8830]

@Kristopher Gunnar it worked and I actually got access to my account again. I'm so happy:D Thank you so much, you saved my account :D

[@kristophergunnar9551]

@Iker Crew glad I could help :D

[@mallikarjunyadav7839]

Awesome video with great content and explanation. Very very useful.

[@kehaarable]

Awesome video - not too much extraneous or labored points. Thank you!

[@flaviofukabori2149]

Amazing. All concepts really well explained.

[@AM-iz8gk]

Impressive presentation well structured explanations.

[@raghudesparado]

Great Presentation. Thank you

[@BuvanAlmighty]

Best presentation in Parquet.

[@Pavi950]

Thanks for the content!

[@YinghuaShen-kw5ys]

Great, this makes me know more about Parquet. Thanks for the pre!

[@raviiit6415]

great talk with simple explanations.

[@lhok]

Best Parquet File presentation I watch

[@pavanreddy3321]

Thanks for great explanation

[@higiniofuentes2551]

Thank you for this very useful video!

[@ravann123]

Very helpful, thank you 😊

[@vt1454]

Great presentation 👏 👌

[@hatemsiyala4944]

Great talk. Thank you!

[@ashokkumarsivasankaran5428]

Great! Well explained!

[@aratithakare8016]

too good video. Excellent

[@tadastadux]

@databricks - what is the best practice to use or not use nested columns. For Example, I have struct of customer with Age, Gender, Name, etc attributes. Is it better to keep it as struct or separate into its own columns?

[@Azureandfabricmastery]

Thank you!

[@user-zz9lk2op1f]

Just excellent 👍

[@chrisjfox8715]

I haven't watched this yet but for the sake of prioritizing when I do, how well does this topic apply to platforms and systems other than Spark?

[@payalbhatia6927]

Superb

[@AmitParopkari]

Finally understood what parquet format, thanks. So I have one small doubt, does it mean that footer metadata is nothing but schema details, like underlying table details? Like way to mention table name, column names? etc. I'll also dig from my side, but just meanwhile ....

[@tasak_5542]

great talk

[@dayserivera]

Great!

[@higiniofuentes2551]

Seems the time and i/o needed before use the data in doing the sort first is not considered?

[@maxcoteclearning]

Thankyou :)

[@spacedustpi]

Thanks for posting this presentation. Could you clarify something? How does performance improve when you compress pages only to decompress it again to read it? I'm sure I'm not understanding something, but not sure what.

[@rescuemay]

He mentions around @19:30 that you only see a benefit when the I/O savings outweigh the cost of decompressing.

[@SQLwithManoj]

I/O is more expensive compared to the time taken by CPU to decompress the data, thus ColumnStore is faster compared to RowStore.

[@rajeshgupta4466]

Snappy provides good compression with a low CPU overhead during compression/decompression. The real win in performance comes from reduced I/O cost when reading a column chunk's page. The overall cost (CPU+I/O) is generally lower for reading snappy compressed as compared to uncompressed.

[@spacedustpi]

@MGLondon How old are you? I am American (and not from China), and stick to common meats. This is an example of hate/harassment. Are you a high school kid?

[@spacedustpi]

@harsh savla Good for you. Ecoli enters the body on vegetables.

[@salookie8000]

interesting how Parquet (columnar analytical focused) data can be optimized using dictionary-based compression and partitioning

[@rum81]

anyone who says parquet is columnar format is having just bookish knowledge

[@immaculatesethu]

Its a mixture of both Horizontal and Vertical partitioning and combines best of both worlds

[@jeremygiaco]

i like the way it compresses the data into dictionaries per file. reminds me a bit of an EAV database stored as a file

[@jeremygiaco]

How is storing json/xml (not parquet) more efficient than csv? You literally store the "column names" in each "row" in xml/json (at least when stored in a text file) . Also, there is definitely the notion of a "record" in csv.

[@happywednesday6741]

Example 1. If you wanted to add a new properties to records overtime, you only need to add it to the new records (no need to back date blanks for legacy records for example). So think scale and change at scale.

[@happywednesday6741]

Example 2. You can leverage hash/dictionary data structures in programming, these can find records at a much better scaling, look up hash functions and big o. Again think scaling related to data access, hashing vs at best search trees.

[@happywednesday6741]

Example 3. You can more easily partition records via collections paradigm. Again storage and access at scale.

[@happywednesday6741]

Example 4. You will more easily access and operate xml / json - like data from applications via APIs. Systems and interoperability at scale.

[@jeremygiaco]

@@happywednesday6741 i asked how it was more efficient to store it. if i have 500 million "entries" in a text file, I'm definitely storing it in a delimited format or parquet to take advantage of said dictionaries and not json/xml. you can parse either into objects directly from the file, or bulk insert into a db table. the json/xlm formats would be 10x slower to parse/read in based on sheer disk/network i/o alone... if we're talking about efficiency in processing it. no one is going to load csv into memory and start trying to scan row by row for data, it's going to get converted into objects or a db anyways. my concern is when people store json formatted files to disk to be read into objects later. what does that buy you?

[@thevijayraj34]

Bucketing explanation was not great. Rest was fantabulous.

[@chriskeo392]

Or whatever.... 😂

[@lax976]

Worst lecture ever