Projection and Predicate pushdown in Apache Parquet

[@pawarbi4675]

Enjoying videos Mark, keep going. One suggestion - slow down a bit and highlight the code/ explain the code a bit (not in detail) so we get context.

[@learndatawithmark]

Thanks and I'll try! Although I have been told to slow down for many years and I'm clearly not great at doing that!

[@galsl]

can you provide examples where predicate pushdown is not possible with parquet?

[@dhaval1489]

Can you do a tutorial of Duckbd + ibis, I am totally new to databases, I am actually familiar with excel, pandas and Polars, just starting

[@learndatawithmark]

Hi! Thanks for your comment. I haven't used Ibis before, but I'm gonna take a look and will try to make an intro tutorial 🙂

[@DerekMahar]

Where may I find pqrs?

[@learndatawithmark]

Sorry forgot to put the link. Here we go: github.com/manojkarthick/pqrs

[@michaelhunger6160]

does the pandas API also support predicate pushdown?

[@learndatawithmark]

It does projection pushdown via the columns parameter (pandas.pydata.org/docs/reference/api/pandas.read_parquet.html) And I think if you'd use the pyarrow engine you would get predicate pushdown too. Pandas delegates the Parquet reading to fastparquet or pyarrow under the covers, it doesn't have a reader itself as far as I'm aware.

[@jeevan88888]

@@learndatawithmark great insights Mark, just slowdown the explanation in the main part of the code.