Spark + Parquet In Depth: Spark Summit East talk by: Emily Curtin and Robbie Strickland

[@stephaniedatabricksrivera]

Emily's Parkay butter pics made me laugh. Really enjoyed this. Great job Emily!!

[@HasanAmmori]

Fantastic talk! I wish there was a little more info on the format spec itself.

[@gmetrofun]

AWS S3 supports random access queries (i.e., Range Header), consequently pushdown is also supported on AWS S3

[@bnsagar90]

Can you please some text or link where I can read more about this. Thanks.

[@Tomracc]

this is wonderful, enjoyed start to end :)

[@flwi]

Wow, great presentation!

[@manjunath15]

Very informative and nicely articulated.

[@amitbhattacharyya5925]

good explanations , this would be great if some git code they can mention

[@maa1dz1333q2eqER]

Great presentation, touched a lot of important areas, thanks

[@HughMcBrideDonegalFlyer]

Great talk on a very important (and too often overlooked ) topic

[@tianzhang3120]

Awesome presentation!

[@clray123]

Eh so basically any sort of growing data can be only partitioned in one way (along the dimension of the growth - which for many use cases will be some meaningless "autoincrement" id). Which then defeats all the push-down filtering for any other dimension. Not to mention that if your data keeps growing in small increments and you need access to latest of it, you will have to jump through hoops to somehow integrate all those small increments into bigger files - because scanning 20000 tiny files ain't gonna be efficient (and this means lots of constant rewriting - that's why write speed DOES matter and it's not "write-once", but write-many)...

[@betterwithrum]

Where are the slides?

[@TheAjit1111]

Great talk, Thank you

[@bogdandubas3978]

Amazing speaker!

[@djibb.7876]

Great talk!!! I set up a spark-cluster with 2 workers. I save a Dtaframe using partitionBy ("column x") as a parquet format to some path on each worker. The matter is that i am able to save it but if i want to read it back i am getting these errors: - Could not read footer for file file´status ...... - unable to specify Schema ... Any Suggestions?

[@pradeep422]

The only thing I liked is the way Emily executed it.

[@ardenjar7942]

Awesome thanks!

[@thomasgong5538]

具有一定的指导学习作用。

[@deenadayalmuli2756]

to my experience, orc supports nesting...

[@mikecmw8492]

Why is everyone a "spark expert"?? Get real and just show us how to do it...

[@betterwithrum]

there are spark experts, just far and few between. I've hired a few, but they were unicorns