Sound Data Engineering in Rust-From Bits to DataFrames

Spark applications often need to query external data sources such as file-based data sources or relational data sources. In order to do this, Spark provides Data Source APIs to access structured data through Spark SQL.
Data Source APIs have optimization rules such as filter push down and column pruning to reduce the amount of data that needs to be processed to improve query performance. As part of our ongoing project to provide generic Data Source V2 push down APIs, we have introduced partial aggregate push down, which significantly speeds up spark jobs by dramatically reducing the amount of data transferred between data sources and Spark. We have implemented aggregate push down in both JDBC and parquet.
Connect with us:
Website: databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / data. .
Instagram: / databricksinc

[@che5ari]

Great talk. Many thanks for this.

[@rajdeeproychowdhury3450]

Great talk, there is typo at 19:10 "graphana" -> "grafana"

[@Rene-tu3fc]

interesting talk. the video description does not match the content, though

[@gw1284]

Thank you

[@konstantinrebrov675]

I think that the IDE window is much too small. It should have been in the full screen, without the "Demo" title and without the face of the lecturer. In Visual Studio Code you can maximize the editor window by hitting F10, if I'm not wrong.

[@budiardjo6610]

this person is cool af

[@mehdiyahiacherif2326]

19:25 graphana :/ , good talk

[@talt6856]

Data.table is and R package… not Julia

[@weouthere6902]

What's up with dask..? Slower than pandas??

[@theLowestPointInMyLife]

Pandas probably uses c