Spark, Dask, DuckDB, Polars: TPC-H Benchmarks at Scale

[@randywilliams7696]

Great video! Recently switched from Dask to Duckdb on my ~1TB workloads, interesting to see some of the same issues I found brought up here. One gotcha I've found is that it is REALLY easy to blunder your way into making non-performant queries in dask (things that end up shuffling, partitioning, etc. a lot behind the scenes). It was more straightforward for my use case to write performant SQL queries for duckdb since that is much more of a common, solved problem. The scale-out feature of Dask and Spark is interesting too, as we are considering the merits of a natively clustered solution vs just breaking up our queries into chunks that can fit on multiple single instances for duckdb.

[@MatthewRocklin]

Yup. Totally agreed. The query optimization in Dask Dataframe should handle what you ran into historically. The problem wasn't unique to you :)

[@ravishmahajan9314]

But what about distributed databases. Is DuckDB able to query distributed databases? Is this technology replacing spark framework??

[@andrewm4894]

Great talk, thanks

[@rjv]

Such a good video! So many good insights clearly communicated with proper data. Also love the interfaces you've built, very meaningful, clean and minimalistic. Have you got comparison benchmarks where cloud cost is the only constraint and the number of machines or their size and type (GPU machines with cudf) is not restricted?

[@richerite]

Great talk! What would you recommend for ingesting about 100-200GB of geospatial data on premise?

[@mooncop]

you are most welcome (suffered well) worth it for the duck

[@taylorpaskett3703]

What software did you use for generating / displaying your plots? It looked really nice

[@taylorpaskett3703]

Nevermind, if I just kept watching you showed the GitHub where it says ibis and altair. Thanks!

[@FabioRBelotto]

My main issue with dask is the lack of support of the community (very different from pandas!)

[@ravishmahajan9314]

But DuckDB is good if your data fits one single machine. But the benchmarks shows different story when data is distributed. What about that?

[@o0o0oo00oo00]

I don’t see duckdb and polars kick spark dask ass on 10gb level in my practical usage.😅 we can’t always trust TPC-H benchmarks.

[@bbbbbbao]

It's not clear to me if you can use autoscaling with coiled.

[@Coiled]

You can use autoscaling with Coiled. See the `coiled.Cluster.adapt` method.

[@kokizzu]

Clickhouse ftw

[@maksimhajiyev7857]

The problem is that in fact RUST based tooling actually wins and all the paid promotions just suck . The actual reason why RUST based tooling is sort of suppressed is very simple , hyperscalers (big cloud tech) earn a lot of money and if things are faster there is no huge bills for your spark clusters 😊)) , I was playing with RUST and huge datasets myself without external benchmarks course I don t trust all this market shit .Rust based EDA is maybe witch kraft but this thing runs as beast . try yourself guys with a huge datasets .