Why I Switched from Python to Rust for AI Deployment

In this video, I share my journey of deploying the same AI project with both Rust and Python, highlighting the challenges, successes, and performance differences. I also compare The performance difference between Polars and Pandas.
🤔 Key Points:
Polars VS Pandas
Speed comparison between Rust VS Python
Scalability comparison between Rust VS Python

[@zendr0]

So what's the conclusion? Is it ready for ML or datascience production systems

[@codeinajiffy]

I wouldn't say so. Python still has more libraries for ML and is easier to develop. I would use Rust for only deployment and only in the condition that Python is too slow for your use case.

[@zendr0]

@@codeinajiffy gotcha. Have you tried the candle crate by huggingface. They say it is Rust alternative to Pytorch. You should give it a shot. May be make a video on that if you like it.

[@codeinajiffy]

@@zendr0 That sounds super interesting. I will consider it in my next videos if I find it to be a useful.

[@berkes]

It is. But very much "it depends". Rust isn't just "faster to run" (less CPU and less memory, which the video dies not go into). It's above all stricter and thus protects you against many mistakes and bugs. Not all, but many. While that may even slow down a developer in the initial days or hours, it starts paying back within weeks. And seeing that we typically do some POC or exploration at first, Python (or even bash) is probably a much better fit there. But the moment we put something into production, make it a team project and/or start maintenance and continued development, the strictness of rust pays off. Often withind days. I typically cludge together a PoC in python. Once/if I'm happy with the direction and setup, I'll throw out that hodgepodge of python, bash/sed/awk, JavaScript and whatnot. And write it in Rust.

[@julyort6762]

​@@berkes Yeah so design in python and deploy in Rust seems really good

[@Mn16Cr45]

Now you can switch from Rust to Mojo and make new video about why I switched

[@jorge1869]

Agree❤

[@421048]

damn, I searched about mojo, and tbh I cannot wait to it become stable to use in serious project man. fucking sh111t

[@huvineshrajendran6809]

As much as I would like to agree with you, Mojo is not quite ready yet

[@DataPastor]

And Mojo’s license is quite restricting, hindering the adoption of the language.

[@blaisepascal3905]

Is Mojo mature enough though?

[@MrRavaging]

@3:21 What about using Gating to handle tasklist management and resource allocation to prevent some requests from hogging all of the CPU and starving out other requests? Even as I'm writing this, I'm realizing that you can also apply background monitoring services and machine-learning to optimize this allocation process and prevent errors and bottlenecks.

[@julyort6762]

Anyway polar can run also in 1 thread, maybe you were using only the default which is multi-thread and that's bad for non intensive request

[@francismatsika3727]

I have been trying rust and the learning curve it steep . Unfortunately python has a huge ecosystem and u cant just jump it. Thats why i am looking carefully at Mojo which has similar memory management features to rust , the borrower checker and how referencing is done in rust. This can be a game changer. Mojo can bring in C perforance, Rust security and python versatility in one language

[@kevinmcfarlane2752]

From what I can see, Mojo is going to clean up in this area in the long run. Rust isn’t going anywhere but it will clean up outside of AI and scientific computing tasks. Right now, F# is more mature for AI and can play nicely in the existing ecosystem.

[@TheWuhaner]

i am not clear at 7:3 in the video, that you said "squeeze out of rust by removing polars". Do you mean you program to implement functions like Map , filter, aggregation in pure rust? I am asking because I found a similar issue when benchmarking the performance of Pandas in Python and Polars in Rust. I didn't obvious improvement switch from Pandas to Polars.

[@codeinajiffy]

Yes, I reimplemented the aggregations I wanted with only RUST using Maps. Because it's a specific use case where I know exactly what keys to put in the maps, I was able to increase performance a little here.

[@TheWuhaner]

@@codeinajiffy Thank you so much for the explanation.

[@NagenBaruah-rg8mi]

Chaining increases polars performance dramatically. And use polars specific syntax as far as you can. That will easily give ~60℅ boost.

[@vectoralphaSec]

What about Mojo? The language that was literally designed for AI from the ground up? Plus Mojo's speed could be tested too.

[@codeinajiffy]

Yup, I will look into that very soon. It's just a new language that might not yet be ready for production, but we can give it a try.

[@mahakasem810]

​@@codeinajiffy Their benchmark was recently revealed a lie after someone uncompleted the binary. Turns out something shady is happening in their compiler that doesn't match their claims. Big disappointment.

[@houstonbova3136]

Mojo is low-key trash.

[@vectoralphaSec]

@@houstonbova3136 Its better than Rust in every way for AI though.

[@houstonbova3136]

@@vectoralphaSec Having done Data Science in a mix of professional and hobby for the last 5 years I’m going to have to hard disagree with that. I strongly prefer just using Python for exploration and then subbing out to raw Rust when I need gains. But, I also understand Rust. Mojo rarely matches its performance in practice and reduces my ability to create effective FFIs or directly manage the data when needed.

[@factsnacksnat]

Although Rust is powerful programming language, I still use python for make sure for being hired in industry.

[@CoolDude911]

Using two packages is not comparing the two languages. There are plenty of dataframe libraries that make different compromises on the backend that can do better than pandas.

[@djthedadiofficial]

look at virtually any published benchmarking of pandas vs polars and polars is many times faster. Without seeing your code it's impossible to prove, but this was almost certainly a skill issue in terms of writing the rust. I am also extremely skeptical about your Golang conclusions, were you using goroutines? as for your last slide, you can spawn threads in Python too but, for all the newbies out there, alas you're unlikely to see almost any of these options. Much more likely to see something like Java, Kafka, Flink, or even Postgres with plugins. For applications that need super quick inference speed, very likely to see C or CPP

[@Mbd3Bal7dod]

running multi workers for the api will solve the concrent request issue ,

[@codeinajiffy]

Nice suggestion, but multi-workers here also increased resource consumption by quite a bit. It was quite close to increasing k8s pods.

[@gamex-animex6289]

where did you get this animation at 0:25?

[@ajaypatro1554]

What if you used dask dataframe and dask array or maybe cython

[@dimensionalcookie]

Hi, if possible, can you please do a tutorial how to deploy AI on production? Thank you. I know you when im interested in Go and Rust, and now im working in AI industry too. Those banger videos about AI project on your channel really help, but its a coincident when I realize this video is on the same channel.

[@bacharsaleh6984]

I think not is time to show us how you created that recommendation system :)

[@johanngambolputty5351]

The only thing that was stopping me from moving from python to something like c/c++ before was ergonomics, not that c++ can't be concise, but a lot of libraries seem to require verbose interfacing. Rust, like python, seems to be very concise, so it offers a best of both, in brevity and performance, that I always thought were mutually exclusive. I was never under the illusion that python was efficient, it just had great syntax, which made prototyping a joy. It feels like I can prototype in rust same as I used to in python, and then get the proper implementation for free at the end. All that said, wasn't the GIL getting removed and couldn't you use multiprocessing.pool?

[@codeinajiffy]

I didn't try multiprocessing.pool but that is a good suggestion. It might not be consistent between requests, but I will try it out.

[@jnorris32]

So did you switch from Python to Go or Rust?

[@cloudguru3018]

Now you can switch to Mojo

[@aaronelmquist8607]

Yeah relying on just FastAPI operating asynchronously is just the wrong approach for python under a heavy CPU workload. Those operations really would need to move off the main thread with something like celery to keep things from blocking.

[@houstonbova3136]

I’m curious if in the Rust implementation of polars, did you use lazy execution or eager?

[@codeinajiffy]

I used Lazy mostly to optimize the execution as much as possible. But yes you will need to understand lazy and eager to make this work fast.

[@nurmohammed9642]

Biggest advantage of using rust is Zero-Cost-Abstraction, For example `sumation()` function could be write as: (1..=n).sum() and it would still out performed.

[@izidorobaltazar907]

Interesting. I tried the simple counting at 4:54 mark Python vs Rust. Python with numba decorator wins on my system - Rust is 3x slower.

[@chats-bug]

Numba vectorizes the operation, you can do something similar in rust as well.

[@its_code]

Why not mojo language

[@codeinajiffy]

I am considering trying it out next actually. It seems very interesting.

[@its_code]

@@codeinajiffy 🥰🥰😝

[@jasonshen]

to be honest rust is cool but when it comes to ai python is still the best option, speed of the development and the libs available, i did turn to rust from python, trust me i regret as the project progresses

[@Singlton]

when trying to learn rust polars functionality , each time I run the code the compilation time takes around 10 seconds! any body knows how to reduce the time spent in compilation to make it ideal to learn and try polars functionality as a newbie ?

[@LibreGlider]

Are you doing debug builds or release builds? Debug builds will be faster you can also set your debug builds to be incremental. You can also use sccache to cache compiler output. You could change the default linker or send -Z threads=8 to the rustic compiler. That is an unstable option but it will break up the front-end compilation of each package into 8 slices. But a 10 sec compile time isn't that bad considering the performance gains you get. Oh there is also cargo-watch, it will recompile your code when the file is saved. It also depends on how you are running/testing this code. Are you reading in your data from a file or stdin or are you hardcoding data in the source code?

[@vuhuy8952]

polars is doppelganger of spark when write code...

[@anon_y_mousse]

Well, I didn't learn anything from the video, but it did add yet another data point confirming what I already knew. Python makes it immensely difficult to write efficient code and requires external libraries written in better languages to achieve any degree of speed and general usefulness. Rust is basically a more obtuse C++ that nearly everyone in the future will use because the propaganda machine surrounding it is exceedingly effective. I know that a lot of people will claim that Rust is easier to learn, but since you can write your code in C++ exactly the same way as in Rust to achieve the same goal the same way and it will be equally as fast, the ease of use of C++ for simple projects shouldn't be understated. If you write idiomatic modern C++, it's as safe as Rust.

[@mahakasem810]

Sad but true reality.

[@Codotaku]

"it's as safe as Rust", sorry but that's an absurd statement if you knew what you're talking about

[@anon_y_mousse]

@@Codotaku There were some qualifiers on that, but if you missed them and all the rest of what I said, then you must be a Rust user.

[@LibreGlider]

@@anon_y_mousse Idiomatic C++ won't guarantee memory safety because it's unreasonable to expect devs not to make mistakes. Not to mention the safety you get out of the box with Rust, thits a fact. 100% idiomatic C++ 100% of the time?

[@anon_y_mousse]

@@LibreGlider If you write idiomatic C++, then the compiler will catch most errors. The more insidious errors that aren't logic errors can be caught by external tools like the open source and often used `valgrind`, which catches everything the compiler misses and more than `rustc` will catch. If you had learned to program 20 years ago when they were still teaching people how to actually program instead of merely use a tool, then you would know these things. So I feel sorry for anyone learning how to use tools now instead of how to actually solve problems like an actual programmer, but we get what we allow to exist instead of what should because we've slacked off and not done our job for future generations. Sorry, my generation's intransigence in doing our duty for your generation has royally screwed you.

[@guttenbergovitz]

You nailed it. Rust shall it be... 🦀

👏👍👍

[@andrew.derevo]

Rust is definitely great programming language, but it absolutely doesn’t work great for AI stuff. You don’t have any libraries, extensions, manuals, almost nothing. you have to do everything by yourself. This is great if you study, but if you need things to be done fast, the python is an only choice for AI right now. Praying for mojo😅

[@martincontrerasu]

is there any git code link to check your recommendation project?

[@codeinajiffy]

no github link because this was company code. I am thinking of doing another project that I can share.

[@UnbenutzerKanalname]

Try Julia for ML

[@enchantedshit]

please provide free resources for rust

[@codeinajiffy]

I learned Rust from this video: kzbin.info/www/bejne/r5ivkKuZltuth5Y . Then just jump on project and figure things out along the way. This should enough if you have some background in programming.

[@RoyAAD]

Where's the AI?

[@ArminMahyar]

Only c++, rust is a copy of that, dont be naive

[@SupremelyAverage]

I agree. But it's quite better imo

[@herkerr]

At the end it’s all assembly. Why complicate it?

[@slip-shape994]

That means you even don't know C++

[@ArminMahyar]

@@slip-shape994 How old R U? U Tell me what c++ is?

[@Miles-co5xm]

C--

[@Singlton]

do you have discord ?

[@codeinajiffy]

Not yet, I am thinking of creating one.