DuckDB versus Pandas: What's Left on the Table?

In this video I'll show some performance comparisons of duckdb and pandas on basic data cleaning task. We'll see that DuckDB comes close to saturating compute resulting in shorter time-to-insight than pandas.
Link to array functions: • Custom Array Functions...

[@xaviernogueira]

Great video. Subbed! Curious what you use for your terminal to get the nicr colors?

[@cpcloud]

I'm using Alacritty and starship!

[@holandacaua643]

Does this approach also work using the 'mssql' backend ?

[@cpcloud]

You can certainly use MSSQL with ibis, but the examples are designed to work with duckdb because it's easy to set up and get started with.

[@holandacaua643]

@@cpcloud thanks

[@kefahissa]

Nice. How did you manage to configure ipython to produce tables in that pretty way?

[@cpcloud]

We're using a library called rich for this!

[@aakoss]

How does it compare to Polars?

[@cpcloud]

Good question! Perhaps a Polars comparison is in order!

[@cpcloud]

Stay tuned, a blog post about this is on the way!

[@aakoss]

​@@cpcloudawesome, looking forward to it

[@bFix]

2.8 times slower on execution, but you shouldn't forget load times. and memory usage (but you looked at that already)

[@cpcloud]

What load times are you referring to?

[@bFix]

df = t.execute() where pandas loaded the db into ram. or am I misunderstanding, what pandas/ibis did there? around 14:00

[@cpcloud]

Pandas loads the entire dataset into memory. DuckDB operates in a streaming fashion. If you have enough RAM, it might behave similarly with respect to memory as pandas. If you're RAM-limited, then it will work, whereas pandas will simply fail.

[@bFix]

well yes, but loading it into ram takes additional time. so pandas is even slower Sure if you execute many operations in pandas the initial overhead for loading data in memory gets less (per operation), but not every script does that many operations.

[@walterdiaz2003]

reminds me of slick for scala.

[@cpcloud]

Nice, hadn't heard of that. Thanks!

[@walterdiaz2003]

@@cpcloud I forgot to say, excellent video though. Subscribed!