Hmm, I guess the only way would be to write efficient code. I'd profile the code to see what functions are taking the most time, and then focus on improving the slow/frequently called ones Use the right data structures/algorithms. consider using functools.cache to memoize anything that would benefit from caching. reprofile your code after each change to quantify what changes were helpful. You can technically write your own c extensions if your system has a c compiler, but that's probably not what you want.

I compiled the C++ with -O3 for this video gist.github.com/dougmercer/1a0fab15abf45d836c2290b98e6c1cd3

Thanks! This video I used a mixture of Manim to render the syntax highlighted code and Davinci Resolve to do the call outs. In later videos I use exclusively Davinci Resolve (+ a Python script to turn my code into a syntax highlighted fusion composition). I'm actually working on writing my own animation library that's similar to manim, but is more tailored for me/the sort of videos I make. I plan on making a video about it as one of my next two videos. so, stick around for that =]

@@dougmercer Thanks!! 🌟

Someone did a Dask + cuDF implementation. Seems super fast github.com/gunnarmorling/1brc/discussions/487

I tried them! Way slower for this task Using this pandas implementation github.com/Butch78/1BillionRowChallenge/blob/main/python_1brc%2Fmain.py takes about 150s, but polars takes 11-12s

Hahaha absolutely =]

That's typically how I use the open source library! I recently spoke with someone from Prefect at PyCon, and they said the automation features and more celery-esque features are coming to the OS library soon. So keep an eye on that =]

Well... I write Python, and let PyPy compile itself down to C =]

This is my first time hearing about either of those. Very interesting 🤔

That, *and* users don't necessarily need to write their own low level code to get that speed up

Thanks =]

Yeah, I wonder that too. I'm also curious if anyone has a Cython solution...

Polars/Duckdb are crazy fast. Thanks for watching!

I didn't use numpy in the pure Python implementation... See the code at 1:59. That's why I mentioned numpy was surprisingly so much slower in the numpy section. As for the other stuff... You're right that there might be other small tweaks you could make to the various implementations. Thanks for pointing them out

​@@dougmercer From the timestamp you gave I can't deduct if a,b are lists or np.array s because they were created by numpy functions as you have shown at 2:10 . But I'm pretty sure you DID convert a, b to lists, but it could be actually seen when you used mypy and created type hints. I just forgot about it because I was fully invested in pure python and numpy. Sorry, I didn't mean to say your benchamrks are bad, they are actually pretty good. Your video is a lifechanger for me. Previously, I used numpy without any hesitation, even in loops that doesn't leverage vectorization. And, strangly, numpy ave me speedup. So I tried to code your example in disbelive. But, undoubtfully, your code is a good example that I was wrong. But that leaves me with questions how could I get speedup in my previous code...? Maybe I was useing vectorization... Can't remember. Seems like I'm gonna benchmark my old code and test it against pure python implementation. Recently I watched one guy who was benchmarking nested loops. He was trying to proove that smaller outer loops gives +30% performance. But he didn't even realized what exactly he was benchmarking and fooled a lot of people and ignoring me even I gave him math proof of results he got. And conditions to get 30+% boost are extremely vague. So I lost my beliefs in such type of videos. But your... is really good. Thanks a lot!

@@user-uc6wo1lc7t Ah I see what you're saying. I believe I coerced them to lists but I'm not 100% sure. I'll have to double check later. And definitely agree- I always reach for numpy, and it's surprising when it doesn't help! In this case, indexing into the arrays to do a lot of scalar operations is slower. We typically get the speed up when we can do vectorization. Thanks for watching and the thoughtful comments =]

Thanks =]

Aw, thanks! glad you enjoyed it, and thanks for your super nice comment =]

I did -O3 compile the C++, but this channel is mostly focused on Python devs. So the idea is, what would someone unfamiliar with C++ write... That said, structuring as a 1D array does achieve some speed up, but still only slightly faster than Numba/Taichi , so it doesn't really change the story

I covered PyPy in my latest video ("How fast can Python parse 1 billion rows of data")

@@dougmercer I just saw the video, that was brilliant! Thanks

Thanks! I hope so too 🤞

That's fair. Some things are worth squeezing all the performance out It will be nice when PyPy supports sub-interpreters or no-GIL... it will eventually be possible to close the gap more

Oh no 💀 Numba + numpy or PyPy would probably help

Thanks!

It would be for something like this! Using this pandas implementation github.com/Butch78/1BillionRowChallenge/blob/main/python_1brc%2Fmain.py takes about 150s, but polars takes 11-12s

Still trying to find my style/voice ¯\_(ツ)_/¯ Check out my two more recent videos. They have less intrusive editing

@@dougmercer Thank you Doug : )

Thanks so much =]

It was -O3 optimized gist.github.com/dougmercer/1a0fab15abf45d836c2290b98e6c1cd3

Yeah

Hah! I'm at 9,991 subs, so good news! Hopefully crossing 10k mark soon =] 🤞

Neat idea... It's worth a shot! Feel free to fork the repo and give it a try

You could use numpy, but I don't think it would help (the bottleneck is reading the data in). I did see some use Dask + cuDF (CUDA) and that was very fast. However, it wasn't allowed in the challenge because the evaluation system didn't have a GPU

@@dougmercer ah. Reminds me of another challenge I saw where IO is the bottleneck. Even there I'm wondering if writing the content to the GPU memory and back is too slow

¯\_(ツ)_/¯

The clock visualization is a little confusing but... 256 seconds 2.56 seconds 100x difference

I tackled PyPy in my latest video if you're interested ("How fast can Python parse 1 billion rows of data")

There's something Zen about it 🧘

Thanks Russell =]

Glad it was helpful!

It's way slower Using the pandas implementation in here github.com/Butch78/1BillionRowChallenge/blob/main/python_1brc%2Fmain.py takes about 150s, whereas the polars implementation takes 11-12s

My focus on this video was libraries/approaches that let you either *write* python or interoperate with Python very seamlessly. I don't really care that Taichi or numba dips to LLVM. As for compilers written in Python... PyPy's JIT is written in RPython and compiles to C ¯\_(ツ)_/¯

"better" is probably a matter of taste. I prefer it because it feels more like writing Python and less like writing a config file. There are several comparisons out there that dive into the differences. Here's a third party link comparing them + another approach (Argo) neptune.ai/blog/argo-vs-airflow-vs-prefect-differences

I expect that the custom parser would have worse performance in plain CPython, but I didn't test it

1. A bit of intuition 2. A bit of being totally wrong (removing the constants that indicated which column was min, max, count, sum didnt speed up performance... I was trying too many things at once and accidentally bundled that change in with something else) 3. I used a lot of time.perf_counter() to measure that time that certain operations took and A/B tested them Normally, in CPython, I would typically profile using something like PyInstrument or other similar line-level profilers

Thanks Mauro! I hope so too🤞

Multiple cores help even for C/C++

I'm no philosopher, but this gives me ship of theseus vibes. so... maybe technically but I don't feel good about it