How Many DataFrame Libraries Do You Need To Change a Lightbulb?

In this talk from PyData NYC 2024, Dharhas Pothina, CTO at Quansight, takes you on a journey through the DataFrame landscape, cutting through the hype to explore where these libraries truly shine.
Whether you’re a data scientist optimizing trading algorithms, a researcher analyzing geospatial data, or a developer building scalable data pipelines, Dharhas will help you navigate the options. From efficient eager execution to large-scale data processing, you’ll learn how libraries like pandas, Polars, Dask, PyArrow, DuckDB, Modin, and Vaex stack up in real-world scenarios.
Dharhas brings a wealth of expertise to the discussion, drawing on his background in computational modeling, high-performance computing, and visualization. He’ll cover third-party benchmarks, delve into library-specific design philosophies, and highlight compatibility initiatives like Ibis, Narwhals, and the Data APIs consortium. This comprehensive overview will help you make an informed decision about whether to stick with a familiar tool or explore something new.
Beyond his role at Quansight, Dharhas leads the development of open source projects like Nebari, Conda-Store, and Ragna. With a PhD in Civil Engineering and a passion for enabling scientists and engineers with scalable tools, Dharhas is uniquely positioned to provide insights that resonate with data professionals.
Join us for a session packed with practical advice, technical depth, and a clear-eyed look at the current and future state of DataFrame libraries.