Introduction to Scaling Analytics Using DuckDB with Python

DuckDB is a free, open-source, fast, and performant singe machine OLAP analytical database that supports the full SQL language. It's Python API is extensive, can work with data that does NOT fit into memory, and is many times faster than pandas. I predict DuckDB will dominate the landscape soon! It is a Game-Changer!
Supporting Links
Support Me on Patreon
www.patreon.co...
My Playlists: / @bryancafferky
Slides: github.com/bca...
Code: github.com/bca...
Data: Yellow Taxi Data 2024-01 (Yellow Taxi Trip Records Parquet): www.nyc.gov/si...
My Video on Scaling Python Pandas: • What to Learn for Your...
DuckDB Python API Documentation: duckdb.org/doc...
My Video About Using Python Virtual Environments: • Python Virtual Environ...
DBeaver Documentation: dbeaver.io/

[@banihas22]

Excellent intro of DuckDB! I love how you setup with an overview to give better context ❤

[@mathewrtaylor]

Thanks for the introduction to a new tool. Was able to parse through a ton of info (5+ million rows) into a few tables in less than a minute, and start analysis in SQL super quickly. Appreciated your commentary throughout as well. Thanks for unlocking a new tool for me; it has tons of potential and will be well used!

[@BryanCafferky]

Great! That's awesome! You're Welcome!

[@bottle0ketchup211]

Awesome video, appreciate your knowledge sharing as always!

[@phillipdataengineer]

Learning so some with your videos! F* awesome man thank you!

[@BryanCafferky]

Glad they are helping.

[@maxl1448]

Great Video. Thank you Bryan. DuckDB is a fantastick thing.

[@eduardokorbesbinotto106]

Nice video! Have you tryed to run duckdb on databricks? Does this makes sense? Thanks!

[@BryanCafferky]

It does make sense. Using DuckDB on the driver node could allow much larger datasets to be processed at a high speed. I plan to try it.

[@eduardokorbesbinotto106]

Great! Please make a video about it, it would be awesome to know how it performs against processing data with spark

[@JavierLuraschi]

DuckDB = Awesome!

[@BryanCafferky]

Hi Javier, Yes. It is incredible.

[@krishnakoirala2088]

can we see databases/tables created like we see in SSMS?

[@BryanCafferky]

See the link in the video description to DBeaver, a SQL Database UI that supports many database products including DuckDB. Yes, using DBeaver, you can browse DuckDB objects.

[@jaguarprovo]

I gotta know what’s your server setup or laptop now. You bragged and I heard it

[@BryanCafferky]

My laptop has 64 GB of RAM and 2 TB SSD. Its actually pretty old now but still very fast.