Speed Up Data Processing with Apache Parquet in Python

[@islam9212]

It hurt my eyes when I saw the calculator even though a python console exists. For a future video it would be interesting to include a comparison with the pickle, feather and jay formats.

[@chndrl5649]

The reason why the memory taken for both dataframe is because of the datatypes. Csv will convert most predefined datatypes into string which is much larger than numeric datatypes

[@tb9359]

Had never heard of Parquet. Thank you. It looks very useful.

[@jeremiahhauser7148]

Interesting, but I am not convinced. If I got it correctly, when selecting columns the time went down by a factor of 3 for both methods (4->1.3s and 0.24->0.08s). So parquet is better anyway, but whether it is specifically better for column-wise access still needs to be demonstrated. As the other commenter, I would also be interested in a broader comparison with other formats. Great channel, keep up the good work.

[@multitaskprueba1]

You are a genius! Fantastic video! Thanks!

[@dana-pw3us]

Why not compare sizes of files on a disk? Are they different?

[@JeremyLangdon1]

I think pandas tried to infer data types from CSV and often defaults to string. This takes much more space and CPU. Parquet has data types built in to the file so pandas does not need to infer anything. What would be more interesting is when reading the CSV, specify the data types to make it a more “even” comparison.

[@Gabriel-cf3bw]

Nice tutorial! Very introductory!

[@slothner943]

Usually go for feather format. Never understood the difference - just that for me and the data im handling (few columns) feather seems to be quicker.

[@JLSXMK8]

I have a related question: Since parquet files are "column-oriented", do you think they would be a good way to store database backups? Example scenario: Let's say you want to store a database backup, assuming that the data in the database is in a stable state; it contains a large number of product records; maybe their IDs, descriptions, how many purchases for a product, the product prices, etc. Would it be a good idea to store a backup of this database using a parquet file since the backups would be faster to load in case of the data becoming unstable via a transaction in the future? You could rollback the transactions too; however, what if too many of them fail, and all of them need to be rolled back?

[@KingOfAllJackals]

Parquet isn’t a generic file format. It IS a table so you’re not “store backups” in a Parquet file. I guess you could backup each table independently but nearly every real DB has much more efficient and powerful native backup infrastructure. Parquet however is where a lot of transactional data ends up for analytics. Columnar storage is more suited to large analytic workloads. Row stores are more suited for OLTP workloads. You would never want to use Parquet for things like “deduct $7.83 from customer 1234’s checking account”.

[@JLSXMK8]

@@KingOfAllJackals That is exactly what I thought of possibly using it for; I could use it to back up tables in the database. You did interpret that correctly. I would NOT edit the contents of the parquet backups.

[@N0rberK]

Tnx Capt.

[@julianreichelt1719]

nice

[@farshidzamanirad9691]

Awesome!

[@codewithmajid4841]

ok Boss

[@codewithmajid4841]

I am Junior data scientist From Pakistan