Nice video, this comes in handy as I was indeed asking myself what use cases warrant reaching to PyO3. I am wondering though, if we convert the call `out_dict.get(w, 0)` to a "dummy" `if w in out_dict` won't it be faster than actually trying? Something I also find missing here in the video is the memory and CPU (cores) usage. Not that I think Python would do better there, but it would be interesting to check.

Great job, keep it up!

Well I did played around a bit and what found was if you just use mean() from the Statistics package instead of the the home grown straight forward for loop implementation you get an improvement even over the bytes method. Below is how I modified the function makes it a bit more simple and readable as well a performance boost with out the fancy byte stuff. ( Nothing wrong with fancy byte stuff, that was a good catch) function get_mattr(word_list::Vector{String}, window_span::Int = 50) n_words = length(word_list) effective_window_span = min(window_span, n_words) n_windows = n_words - effective_window_span + 1 if n_windows <= 0 return get_ttr(word_list) end mean_ttr = mean(get_ttr(word_list[i:(i + effective_window_span - 1)]) for i in 1:n_windows) return mean_ttr end Here is a link to the data and the output graph I generated. drive.google.com/drive/folders/1-AelwjZZtAPGKf_bLkhkLOC0ZZUWBuTf?usp=sharing

Hey nice vid. I am a Physicist. I work for a photonics quantum computing company and use Julia for modeling in my work. One thing you may consider is using the BenchmarkTools for Julia. I am not sure but the tail at the beginning of your graph might be due to the JIT compiler optimizing. If this something you do often you could precompile the julia code, once optimized and that would negate the JIT start up time. I will play around a little bit and get back with you. I like the attitude of always willing to learn something from some one else. There is so much out there to learn if just listen and not jump to conclusion.

Amazing doucheFace thumbnail! Congrats you look like every unimaginative, lazy creator on YT. Clearly intelligent people choose stupid face thumbnails because looking like an idiot is a huge indicator that your content must be amazing! 😂

When I run it on linux. and click on start. it crashes and closes. I don't know why!

Probably worth noting, Polars is quicker because it's multi-threaded and uses all the cores on the machine, were as Pandas is single threaded

pandas has a join method. It's supposedly faster. You just have to set the join columns as the index before calling.

What is the difference between the original and the X version?

I have been learning rust so looking forward to watching this video!

Great vid!

I really enjoy this channel! I have been picking up rust from python trying to solve bottlenecks from speed

Blue & pink lines are roughly linear. Roughly at 70k, there could be some memory allocation or something else that has dropped the performance. More like a one time thing within each test above 70k. With this I tend to think that it's linear for normal hash reading ( blue & pink ).

I would assume the fastest way to access values is using dict.values() :)

you didn’t access them the way they meant to. You need to iterate keys and access each value using the key. Otherwise hashing is not required to get the map elements. Simply array can do that.

This is just what I was looking for. I am working on business logic and there are a lot of SQL statements. Python takes about 9 seconds to have all 22k entries in a dict, a C++ program I had ChatGPT just write up based on the Python code (it took like 20 iterations of me telling ChatGPT well now I'm getting THIS error) was 10x faster. I won't learn c/++, I'll do some Rust!

Thanks Professor! I'm learning on the job on my own how to implement python & rust together given my interests in software development, data science & biomedical science so this is an interesting series you made!

Great video, but the result is interesting. Mind if I can get eh full main.rs file and dataset. Would love to run the tests my self and perhaps improve upon it

Awesome, I was waiting for that video! I thought that Python's GIL blocking in your last video had a much stronger effect. I guess strings are in all programming languages pretty horrible, because utf-8 doesn't have a fixed byte size, so all programming languages have to use the slow techniques that python uses for all data types. Python is pretty good compared to other languages when talking about dicts and strings. The rest is very slow, but thanks God, it is very simple to speed it up if you need it.

FWIW i got about a 30% speed increase for julia when working on bytes directly (vs strings) and passing @view of byte-vectors. wds = split(txt) bwds = [Vector{UInt8}(word) for word in wds] and then passing the @view of bwds instead of wds into the individual functions note: i also dropped all the println() and I/O operations in my code as i was mostly curios about the speed of julia and not I/O or print. (but fair play if it is included)

Great video, I like your benchmark videos, but from Rust's perspective it's a little unfair. It's not representative when Rust is a slave of Python's GIL. However, I think the speed difference between the language is not that big, because UTF-8 chars don't have a fixed byte size. It would be nice to see a comparison between utf-8 and byte strings (fixed size). You could also add Zig to the benchmarks, I used it a week ago for the first time (RGB search in a picture). It was between 2 and 10(!) times faster than C, but I haven't tested it with strings yet.

Thank you for sharing this!

But only as long Mojo isn't out there to perform python to it's coming new standard limits even faster than c++. The future is going to be fast as hell bro 🎉

Bah, Mojo is not even open source. That's repugnant.

How about using glob as a generator? Would that reduce the gap beteen the os.walk and glob? Because in your code, glob() loads everything into memory and that may adversely impact the runtime.

I thought about this a bit more, and I think the MTLD_wrap algorithm has a time complexity of O(n^2). It might be interesting to try to fit a quadratic to the scatter plot instead of a line!

I tried PyO3 and its actually a really good library. BTW great content.

This is fantastic! Thank you for sharing!!

If you convert the text into an array of integers, where each integer is an index into an array (or tree) of unique words from the text, you could possibly speed up things by a lot, depending on how long it takes to set up the arrays/tree.

Nice! I would love to see the same thing, but with Numpy arrays instead of dicts in Rust.

What I learned the last couple of years is that Python is the best language for working with strings. It might not be the fastest, but the difference to the compiled languages is not that big (sometimes even faster - regex Python > regex C++), like with numeric data types. Practically all languages have a bad performance when leading with strings. However, working in Python's interactive mode, doing string operations is priceless. Guido van Rossum said in a recent interview with Lex Fridman that Perl is still the fastest when talking about strings (or regex - I don't remember exactly). However, it would be nice to see a comparison between Perl and other languages.

That's a really cool project! Thanks for bringing it to our attention!

This was great, thanks! Had no idea this was available. Going to implement it into my python ebook reader

Nice video, the speed boost from Rust is almost twice. Can you do polars and pandas performance comparison?

Also try mojo.

Instead of split(" "), I suggest split(). Omitting the argument splits on each consecutive whitespace.

Have to tried different sizes of datasets, to see whether there is some underlying system cause... 230 million words is a lot, way more than I know... and it´s all in one file... and not very parallel...

what about Go ?

So rust essentially takes half the time than python....nice, but I thought rust would be a lot faster.

So at the end is Julia vs. C

This is the second time I have viewed this video. Thank you for performing the benchmark-testing work I would have had to do- saved me quite a bit of time. Now a question: Do you believe Mojo will progress to point where its dictionary performance will equal or exceed Python's?

That's a significant speed improvement, 3x folds faster in the newer version. However, it still doesn't explain why mojo code is still slower than actual identical python code, given that mojo was going for machine code compilation but with python syntax and ease, and not bytecode... For the little documentation overview i have read, mojo team explained that mojo is not python, but python will be mojo, in the sense that python will instead be an interpreted subset of the compiled mojo, and that features in python not yet implemented in mojo will dynamically switch to run in an included actual python runtime, but in the future, mojo will be self-contained to run all python code on mojo runtime.

How come you know that dunder methods provide high-level sugar for Python, but you call the methods directly in Mojo? You don't to call object.__len()__. object.__setitem()__ etc directly in Mojo, they work pretty much the same way they do in Python.

And ome could also try out the Dictionaries.jl package in Julia that is much more performant and efficient than he base Julia Dict type.

Thank you so much for this video (and the Others). Really interesting.

Please try splitting the data in half and assigning each halves to a dictionary. Then measure the time taken to copy or interchange elements from one dictionary to another. Maybe the problem is in the file management and not in dictionary!!

I wonder if PYPY version of python is even faster, since it does JIT compiler…. Would love to see that result

the real benefit of Mojo is it can easily target other hardware without having to write C code. Google's work with cuda acceleration and pandas is a good example.

hashmaps are made for O(1)ish retrieval, insertion is far less important. So yeah, insertion gets slower, especially if you haven't chosen a hashing method specific to your data (this can be really important), but the true test of a hashmap/dicitonary is how the size of the corpus affects retrieval. It would be interesting to run the experment on the retrieval end, now that you know the performance of insertion, ansd see how they compare. (You may also want to see how pickling the dictionary and restoring it performs, since this is done a lot more than generatign the initial map in research.) Also, for initial trials, maybe speed things up and get a rougher idea by increasing the quantity by orders of magnitude - 10, 100, 1000, etc. A sparser plot some times helps things jump out of the data more clearly. Thank you for the post, it was good fun!

I think that "small" waa about 10-50 keys as most objects in everyday python are actually these tiny dicts.