Polars: Blazingly Fast DataFrames in Rust and Python

  Рет қаралды 27,125

Databricks

Databricks

Жыл бұрын

This talk will introduce Polars a blazingly fast DataFrame library written in Rust on top of Apache Arrow. Its a DataFrame library that brings exploratory data analysis closer to the lessons learned in database research.
CPU's today's come with many cores and with their superscalar designs and SIMD registers allow for even more parallelism. Polars is written from the ground up to fully utilize the CPU's of this generation.
Besides blazingly fast algorithms, cache efficient memory layout and multi-threading, it consist of a lazy query engine, allowing Polars to do several optimizations that may improve query time and memory usage.
Read more:
github.com/pola-rs/polars www.ritchievink.com/blog/2021...
Join the talk to learn more.
Connect with us:
Website: databricks.com
Facebook: / databricksinc
Twitter: / databricks
LinkedIn: / data. .
Instagram: / databricksinc

Пікірлер: 28
@JeremyChone
@JeremyChone Жыл бұрын
Great video. I have been playing with Polars, Python, and Rust, and it is a fantastic technology. Also, I think Polars' Python facade with Rust backend / clean API is a great model for other data-centric technologies that need performance and scripting facade interfaces. Big thanks to the Polars team for this gem.
@user-qm6up7kz4n
@user-qm6up7kz4n Жыл бұрын
Polars is just great! After trying it I won't go back to pandas.
@ControlTheGuh
@ControlTheGuh Жыл бұрын
Learn rust, python and rust synergies will be all over the place in data science soon. Great work with Polars
@jmnunezd1231
@jmnunezd1231 Жыл бұрын
Why do you think that? Can you elaborate more please? Thanks!
@ptarslatinum2806
@ptarslatinum2806 Жыл бұрын
Can you please provide some insight? I am thinking of learning rust and I wish to enter this field.
@incremental_failure
@incremental_failure Жыл бұрын
Haven't been so excited about data analysis for a while. Now comes the chore of converting pandas to polars in a 15k line project.
@saaddahmani1870
@saaddahmani1870 Жыл бұрын
Really great crate and great video, thanks.
@Josh-wb7ii
@Josh-wb7ii Жыл бұрын
For a lot of my job its not possible to use eagerly evaluated DFs, polars is a fantastic option.
@swalsh1123
@swalsh1123 Жыл бұрын
I'm going to try this out. Thanks
@headbangingidiot
@headbangingidiot Жыл бұрын
This is great, can we use polars on a databricks cluster?
@rakeshjamdare2049
@rakeshjamdare2049 10 ай бұрын
Sure, it terms of python, its just another library so can use it very conveniently!!!
@milo4433
@milo4433 Жыл бұрын
I wonder if this could ever be used with multiple machines similar to spark. Polars looks very fast and with the addition of distributed compute it would be very powerful. Perhaps it was not designed to allow that though.
@user-qm6up7kz4n
@user-qm6up7kz4n Жыл бұрын
Polars uses all logical cores of your machine in parallel. Doesn't that mean distributed compute?
@milo4433
@milo4433 Жыл бұрын
@@user-qm6up7kz4n I am asking specifically about cross machine distribution. For example if we had multiple 8-core machines (like VMS in aws) available, can we distribute polars work across those vms instead of being limited to a single machine and its cores.
@user-qm6up7kz4n
@user-qm6up7kz4n Жыл бұрын
@@milo4433 aah I see the difference now
@matthiasdebernardini3388
@matthiasdebernardini3388 Жыл бұрын
​@@milo4433 yea you totally could but would need to do that yourself, not sure the crate supports it out of the box
@kopodgorski
@kopodgorski Жыл бұрын
Wouldn't be a load balancer be enough to make it happen?
@josedejesuslopezdiaz
@josedejesuslopezdiaz Жыл бұрын
great API
@Birdsneverfly
@Birdsneverfly Жыл бұрын
Does polars with with scikit?
@jmnunezd1231
@jmnunezd1231 Жыл бұрын
I think they work most of the time. The good think in that if something fails, you can pass polars to numpy very easily
@samuelemmettbray
@samuelemmettbray Жыл бұрын
Super confused. I thought Polars was deprecating in favor of Arrow2?
@hiongun
@hiongun Жыл бұрын
need pandas compatible api, just get fast conversion from pandas
@headder7802
@headder7802 10 ай бұрын
It has methods to read and export to Pandas out of the books. It can also write or read Delta tables (locally or in S3).
@therendra
@therendra 5 ай бұрын
Nice presentation
Sound Data Engineering in Rust-From Bits to DataFrames
34:36
Databricks
Рет қаралды 10 М.
О, сосисочки! (Или корейская уличная еда?)
00:32
Кушать Хочу
Рет қаралды 8 МЛН
it takes two to tango 💃🏻🕺🏻
00:18
Zach King
Рет қаралды 28 МЛН
Can You Draw The PERFECT Circle?
00:57
Stokes Twins
Рет қаралды 95 МЛН
How to Stay Current in AI
4:07
Vadim Smolyakov
Рет қаралды 35
Why would a python programmer learn rust when there are no jobs in it
23:09
Rust and RAII Memory Management - Computerphile
24:22
Computerphile
Рет қаралды 213 М.
EuroSciPy 2023 - Keynote: Polars
57:18
EuroSciPy
Рет қаралды 3,7 М.
Rust for Python data engineers - Karim Jedda
27:30
EuroPython Conference
Рет қаралды 4,8 М.
Unlocking your CPU cores in Python (multiprocessing)
12:16
mCoding
Рет қаралды 290 М.
Apple, как вас уделал Тюменский бренд CaseGuru? Конец удивил #caseguru #кейсгуру #наушники
0:54
CaseGuru / Наушники / Пылесосы / Смарт-часы /
Рет қаралды 4,4 МЛН
The power button can never be pressed!!
0:57
Maker Y
Рет қаралды 54 МЛН
Теперь это его телефон
0:21
Хорошие Новости
Рет қаралды 1,7 МЛН
Xiaomi Note 13 Pro по безумной цене в России
0:43
Простые Технологии
Рет қаралды 1,9 МЛН
Carregando telefone com carregador cortado
1:01
Andcarli
Рет қаралды 1,9 МЛН
👎Главный МИНУС планшета Apple🍏
0:29
Demin's Lounge
Рет қаралды 487 М.