Much Faster Pandas with cuDF GPU Processing - CPU vs GPU Speed Benchmarks

[@theMintyRaven]

hi, can you make a short video on how to use cudf in python scripts? there are some solutions on internet which don't work, it would be really good if you do this video

[@PythonSimplified]

Sure! I'll give it a go next time I'll be filming! thanks for the recommendation! 😀 In the meanwhile, try the following: import cudf.pandas cudf.pandas.install() import pandas as pd # the rest of your code Let me know if it works on your end! 😉

[@theMintyRaven]

@@PythonSimplified Thank you, there are some functions which do not exist in cudf and I need to use pandas and cudf but I don't know how specify which task must go to pandas and which one needs to use cudf... unfortunately there is not a good documentation for cudf and i appreciate if you cover details like this

[@oleg.mammoth]

CPU R9 5900x - 1min 3 sec, GPU RTX 3090 - 1.6 sec. Furthermore, Pandas is single-threaded by default, but there is a "modin" project/library that allows to scale pandas so that it utilizes all CPU cores and threads, which helps especially well on large data sets and when a CUDA device is not available. To accomplish this, simply replace the import statement with "import modin.pandas as pd" in the script (previously installed the library, of course). For the task in the video, this reduced the execution time by more than half comparing to ordinary pandas using. Thank you for the video!

[@kamertonaudiophileplayer847]

It's amazing, I started to work on it more than a decade ago. But now, everyone can do it in seconds. Thank you.

[@tanishaness]

i love your way of teaching, this video came just in time

[@PythonSimplified]

Yeyy!! Thank you so much!! Super happy to hear! 😀😀😀

[@MaxStudioCG2023]

@@PythonSimplified you are gorgeous !

[@slademeister]

Excellent tutorial as always! My Alienware R11 has an i9 10th gen and only 16Gb RAM and a 1660ti. With regular pandas the kernel crashed on a couple of tries because it couldn't handle that amount of data (I think) but with cudf it did the labelling task on the 25 mil rows in 1min 35s.

[@PythonSimplified]

You also have Aurora R11???? Me too!!! 🤩🤩🤩 My theory is that regular Pandas crashed because of the amount of RAM (I have the same CPU and same kind of RAM, but 32GB). You can always utilize cool techniques inside read_csv to avoid crashing: 1. Add a columns property and only read the columns you need. 2. Add a chunksize property and set it to some integer. Read_csv will then break the data into read-only segments in the size you specified. You can iterate over them and turn them into standalone DFs. 3. Don't worry about it, and keep running everything on GPU 🤪 hahaha

[@scrumtuous]

Love the idea of moving processing onto the GPU! Great video! ❤

[@JamesLee-lq8qb]

amazing! always knew GPU would speed it up and you proved it.. with cuDF

[@PythonSimplified]

They just released a new update that made it even faster! (about two weeks ago) Hopefully you got to play with the newest version! 😉

[@JoelPasapera]

@@PythonSimplified polars can be combined with cudf?

[@anilkrishna7382]

Hi Maria ,Love the way you teach and I can understand everything you teach.

[@soultribe9]

You´re always amazing M, great video!!! 🤘😸

[@AlexShoyhit]

I love you explanation i love how you simplify things

[@grasshopper3085]

3:18 "We'll of course carefully read the license agreement" 😆

[@PythonSimplified]

When I skip reading those - I always think of a South Park episode where Kyle failed to read the Apple license agreement 🤣🤣🤣 (I don't want to spoil it for those who didn't watch, but checkout the Human centiPad episode if you're curious)

[@grasshopper3085]

@@PythonSimplified I'll check it out!

[@akashmahmud1604]

For me CPU : 9.64 s and GPU: 798ms Thank you so much. I love your videos.

[@kehaujung]

this tutorial is awesome, thanks for the guide

[@PythonSimplified]

Thank you for the lovely comment! 😀

[@legocars3.061]

Can i use this for password cracking?

[@nirmesh44]

lovely. ur channel is great

[@PythonSimplified]

Thank you so much! :)

[@adityak2710]

Please make video on creating openCv python stanalone applications. I saw ur prev videos but they didn't work. I always get error to include a file though there is nothing wrong. Please help

[@PythonSimplified]

Which specific video and which specific error? I can help, but I need to know the details to do so.

[@vasylpavuk391]

Hi Maria. Have you tried the Polars library? It is inspired by Pandas and works similarly but much faster because it is made by using Rust programming language. Greetings from Ukraine.

[@PythonSimplified]

I'll cover Polars on the channel soon, would love to test it against Pandas, cuDF Pandas and DuckDB 😉 Once and for all, we'll see which is better for which tasks! It's definitely on my to do list 😃 Cheers from Vancouver, Vasyl!! 🍁🍁🍁

[@katrinabryce]

My WSL is set up as Debian, and I have Python 3.11 installed on it, so I wasn't able to replicate this. Apparently 3.11 isn't supported yet 😢. I will try some other time to set up a second WSL with Ubuntu and see what happens. I have a Threadripper Pro 3945WX with 256GB RAM, and an RTX 3080Ti, and generally speaking, CUDA is about 4x faster than CPU. Only having 12GB of VRAM is a major problem though. Typically the datasets I work with are around 100m rows. I reduced the exectution time of one Pandas project from 10 minutes to 0.1 seconds by re-coding it in Numpy, and Polars gives me not quite so good speed increases for a lot less effort.

[@PythonSimplified]

You're 100% right, Katrina! cuDF doesn't support Python 3.11 just yet, so it might be the reason why the installation fails. I don't think there's a problem with Debian compatability as Miniconda should take care of installation command discrepancies 🐍🐍🐍 This cuDF Pandas accelerator mode was only released a month ago, so I'm sure Python 3.11 support is on the way... but I'll double-check it with Nvidia just in case! 😉 For 100m rows, I think it's worth jumping back to Python 3.10. Losing 10 seconds here and 10 seconds there doesn't sound too bad - but it compounds to minutes and hours very quickly. Plus, I didn't have time to cover it in the video, but cuDF has plenty of functionality beyond what Pandas can offer! I wanted to end the tutorial with stripping punctuation + stopwords and stemming the words, but it would have made the tutorial way too long... 🙃 Essentially, what took me 7 minutes to do on my recent midterm project - was done in seconds with cuDF!!! It's insane!! For preprocessing tasks - I don't think I'd be utilizing my CPU ever again. But I still need to check out Polars and see how it compares with cuDF. Sounds like a great idea for a new tutorial! 🤪

[@PythonSimplified]

Hi Katrina, I have some great news! Just spoke with Nvidia, and they're currently working on Python 3.11 integration! 🥳🥳🥳 For now Python 3.9 and 3.10 are fully supported, and they will announce when they are done with 3.11 (I think because 3.11 is a relatively new version, and many of us haven't switched yet). Also, quick 1 months of usage update: I'm not working with traditional Pandas anymore (or even Windows). Fully switched to WSL and cuDF! Will arrange a nice live stream soon t6o catch up 😉

[@Жизньвчастномдоме-н7х]

When I have installed cuDF, and oped this envirement on Pycharm, stopped working os.walk function. Just does not find any paths and files. But if switch env to the standart python env, os.walk works properly. How to fix it?

[@renanmonteirobarbosa8129]

Dude, you finally discovered cuDF

[@ahmedeveloper]

pls can you tell behinds background device use

[@louiss3409]

So, I was attempting to follow this and the cuDF install cannot be installed without anaconda apparently.

[@kbess4]

Thks from Brazil! 😃

[@ManasBishtVBprogrammer]

Thanks a lot for tutorial!! So if some functions are not present in CuDF, will it automatically switch to normal pandas for CPU, or will it give some error/warnings?

[@nirajsharma8568]

Python Simplified thanks for video i have one question can we use cudf on window 10?

[@PythonSimplified]

Only if you do so via WSL, cuDF and many other CUDA goodies require a Linux infrastructure 😉 So type wsl --install inside your command prompt terminal, navigate to your new WSL terminal, and from there - you're good to go!

[@ikrmemento]

Wow those helped thank you man🌺

[@Ggorre-kz3md]

are you planning a set up tutorial of cudf for linux? been looking all over for one using cuda 12.3…

[@PythonSimplified]

This one should work for Linux too, as I'm using WSL (well, it works with Ubuntu for sure, not sure about other distros... my Kali linux machine is no longer operational so I can't really test it). And for CUDA 12.3 just select CUDA 12 in the RAPIDS installation guide, I had the same situation, and everything worked 😉

[@oleg.mammoth]

On Ubuntu 20.04 I failed to install cuDF using pip in venv, but when tried using Conda (similar steps as in the video) then it launched successfully.

[@henry-o8i]

Thanks for the content. I try to follow along. I set up the faster_pandas environment in WSL ( or do i need to install it in powershell). When i run the %load_ext cudf.pandas, i get an error message saying No module named 'cudf'. I wonder what did i do wrong?

[@marcosissler]

Any way to cuda works with max M1? Or pandas uses the M1, M2…?

[@MrJxtr]

Is this possible with normal windows or must this be in wsl? 👀

[@mdjuneid3553]

How can we do this in vscode ..?

[@MuhammadNurdinnewspecies]

Still good to run it during training model like pytorch or tensorflow?

[@PythonSimplified]

Yup! When you install RAPIDS and select cuDF in the "additional packages" section - please make sure you select pytorch and tensorflow as well, and you're good to go! 😉

[@eduardfrades6972]

HI, I am trying to download test data zip file but it looks it doesn't exist. any alternative?

[@ZurioSi]

Amazing video :)

[@PythonSimplified]

Dark Wolf??? Is that you????? How's it going?? 😃😃😃

[@adityak2710]

please make video on how to .py file of computer vision to .exe file. The normal python file is converted to .exe file but the py file of computer vision doesn't do!!!!!!! Please help

[@adityak2710]

I watched your previous videos but they too didn't work

[@HarmeetSingh-ry6fm]

I'm not able to replicate your code. Can you specify the version of everything that you have used mainly wsl's version. thank you so much for this video

[@PythonSimplified]

I'm using WSL 2 and RAPIDS 23.10. What errors are you getting and at which stage of the tutorial?

[@shakils1921]

Love you maria my best python teacher,

[@القائدحمزةديكو8]

Can you help me i want to code inteface with kivy of substraction

[@joseleonardosanchezvasquez1514]

Great thanks. Where are you???

[@PythonSimplified]

Wrapping my last university exams, just moved to the Fraser Valley where I have a much bigger office that I'm still yet to properly set up, and getting ready to fly to San Jose for GTC-2024 next week 🤯🤯🤯 In other words - I'm running around like a chicken without a head hahahaha but the good news is -> I wrote an incredible cloud computing tutorial (on CPU and GPU!) and will film it as soon as I'm back to Canada 😉 Hopefully you'll get to see it this month! fingers crossed 🤞

[@starlightknights]

Thank you. It's very helpful to me.:)

[@skroyeducation2166]

I want to deploy my kivy program into app but I haven't Linux system as you used it I have already tried many other ways but they all are not working it get crashed when I used Collab please make a detailed and separate video on this topic

[@PythonSimplified]

I actually have a really nice tutorial that shows you how to do it with Windows! ⭐ Convert Python to Android with WINDOWS & LINUX + Fix Common Bugs: kzbin.info/www/bejne/jKS3koBmapylmsU It also uses WSL, just like we do in this cuDF Pandas tutorial. The Colab conversion way doesn't work for the past few years, but you don't need a full blown Linux Machine either. Just install WSL from your command prompt with wsl --install and you have Linux installed on your Windows system 😉 Good luck and I hope it helps! 😀

[@skroyeducation2166]

@@PythonSimplified thank you so much for replying. Let me check it...

[@alissondepadua]

One month is too much to wait for the next video =///////

[@dynex84]

at first i can read_csv on gpu, but for sometimes after that i dont know which modification that i made, now read_csv done on cpu, it become slower, what have i done?

[@dynex84]

sorry i find the answer #this will read_csv on GPU data=data.drop(columns=["1","2","3"]) #this will read_csv on CPU data=data.drop(columns=[1,2,3])

[@dynex84]

test GPU and CPU, PURE CPU CPU times: user 35.2 s, sys: 5.85 s, total: 41 s Wall time: 41.1 s Accelerated CPU with GPU CPU times: user 5.6 s, sys: 30.5 ms, total: 5.63 s Wall time: 5.93 s Read csv file 800K rows and 165 columns Time taken by GPU: 1.015556812286377 seconds Time taken by CPU: 9.871984720230103 seconds

[@sina001luket]

Ohw Beautiful system 🤖🤯

[@DeepakKumar-i7t8u]

Can you please make a video on flask appBuilder framework which is made over flask framework ?

[@shaikshahed413]

Mariya your beauty is a distraction for learners😂

[@leosmi1]

Now we need support for AMD ROCm and OneAPI for Intel

[@PythonSimplified]

You'll need something like OpenCL for AMD based GPUs 😉 Anything CUDA related is proprietary to Nvidia on the hardware level, so it can't be replicated with equipment from other manufacturers 🙃

[@leosmi1]

thank you@@PythonSimplified

[@yliu4093]

Thanks. Is this same process in windows 10 system?

[@PythonSimplified]

Yup!

[@jorge1869]

Hi, we need a video about the streamlit framework. Regards

[@Ben-hu2jp]

Will Python 3.11 be supported soon?

[@deAraujoAndre]

A geek beautiful woman talking about very interesting topics. Like! 😀

[@renlog7755]

Puking in my mouth ❤️ 🤢

[@entertain7]

I am currently doing university final project on ward to sign language translator app, i need a little guidance from you. Thanks

[@hectorcanas4303]

I really love ur vids!!!!

[@mschon]

It's Amazing!!! Very good!

[@themothialbani4523]

Like the way you speak ❤

[@oliviere1215]

I love your videos :) It would be great to have the install workflows with pip and Python venv. Mixing use of Conda and pip sometimes beaks things. I have far less environment issues since stopped using Conda! Could you also indicate the Python version it was tested on?

[@PythonSimplified]

Absolutley! Python 3.9 and 3.10 are fully supported, and Nvidia is currently working on integrading 3.11 as well (they'll announce when it's good to go 😉) I rarely use conda install anymore, switched to pip in terms of package distributions (I find that some of their packages are outdated), but I really like the way Anaconda manages virtual environments. It's super convenient to me, while VENV feels a bit unintuitive... but as usual, it's all a matter of personal preference 😃 Also, super happy you liked the video!! thank you so much for the lovely comment! 😁😁😁

[@vishnusudheer6049]

Do you have any videos that teach making android apps from python. I have to make an app for a contest where judges will score contestants in their respective android devices and in the end consolidated result will be given can you please help

[@Yachid]

hey Ms-Py! 😉

[@kevinmcaleer28]

Lets Roll 👉👉

[@PythonSimplified]

This time I was about to end the intro with 30 **zoom to 105%**. times **zoom to 108%**. faster! **zoom to 111%**. Then the Intro music kicks in. Changed my mind last minute and included the usual catchphrase instead (was worried that folks will be upset about the abrupt sentence ending) hahahaha 😅😅😅

[@jimmybruneel4540]

CPU Time : (i7 9700K and 32GB memory ) 1 minute and 48 seconds using my GPU 3.45 seconds (2080 TI)

[@60pluscrazy]

Good onne 🎉

[@PythonSimplified]

Thank you!

[@ChandruGuru-v1t]

Hi.., i am big fan you.here i have a small doubt?, can you make video with brief explain of pop up frame😢 example user profile is there in top right corner. I will click this action literly pop up a new window with borderless and without window title buttons., If you can't make video plz rply this comment for what i do this 😢

[@littlebrit]

It is Zaporozhets, but with sporty spoilers. Might go faster.

[@louiss3409]

I have been a Java developer, and did some C++ programming in my career, it is like you have person insight into everything I have done in my career up till now, I appreciate the work you have done here, though I did try and contribute an alternate method to your docker tutorial and you deleted it, I guess I don't understand why as there was nothing negative in it, a joke about not getting paid. Anyway, thanks.

[@PythonSimplified]

Hi Louis! :) I never delete comments (unless they are phishing attacks), but KZbin does. If you include certain characters that very often appear in code (like HTML tags) - they may be flagged by KZbin. Your comments will then either be stored in a "held for review" section or they'd be removed automatically. I'll check this section on the Docker video shortly and see if I can find your comment there. It's just KZbin's way of handling bots and scrapers. Thanks so much for the lovely feedback and I hope your next suggestions will pass through KZbin's filters with no issues! 😀😀😀

[@louiss3409]

@@PythonSimplified I guess you learn something new every day, you would think they would have been able to master the use of tags so that they don't have to do something like that, maybe it was for email? Though I would think they could strip that out with something like a Baysian filter and simple AI, Or say removing tags from HTML from one of your screen scraping tutorials. If you ask Gemini what I typed in it would be able to you that "louis# podman " is a command line command, and that it is not likely a hacking attempt. It certainly would be easier to strip the characters like if you were building a prototype but I guess I viewed KZbin as a little more professional. I have seen you employ all the skills needed to solve all of these problems in code, and Google says they hire python developers, which begs the question if you make more on KZbin then working for Google directly? I had some ideas/features I thought were interesting or intriguing to me on this topic. Anyway, if I were in Google's HR, I would seriously consider hiring you, that was an excellent place to start looking for the cause. I didn't think there was anything offensive about it as it was clearly informative satire. Is there a link you could share that has this documented? Google has so much documentation and you are correct to assume I haven;t nor would I be able to read it all. I did have a joke in there that Gemini didn't provide a yes or no on offensiveness it just said it was very likely a joke than something offensive. The line was "I would have done more but I don't get paid for this sh1t" which it repeated back to me. Well since I checked to see if it would likely be protected speech in the US and Canada which it was it was less certain about if it violated Google's policy, though that would mean that a lot of comedy would be in violation. Maybe AI still needs a little help on protected speech. 8)

[@louiss3409]

@@PythonSimplified Well I took a screen shot, the comments are still in my history even though they aren't listed on your site, I can post them on my linked in account if you want to see them there. If not no big deal I guess, I guess the truth just isn't that important to me.

[@sina001luket]

In the 2000s A Boy looking for old graphics cpu

[@henderstech]

I suddenly want to start learning python again.

[@powercore2000]

I heard pandas and came as soon as I could

[@chamamemestre]

Me: "Looks at my Intel Irisx. Well, I can always bookmark the video I guess..."

[@emekaobiefuna4509]

😂😂 Seems we're in same predicament! I know better now to get a pc with Nvidia GPU when next I wanna buy a new laptop.

[@paulocoelho558]

Sadly, it's not just pandas-we seem to be mishandling the whole animal kingdom. It's a bit off-topic, but it's been on my mind. Great video anyway. You are a cute simplified snake girl... 😕😞🌎🌍

[@SoloRush-hl8jv]

when did u upgrade from RTX 3090 to RTX 4080 speeeeedd 👻👻

[@ManPursueExcellence]

Where is she from?

[@lebesguegilmar1]

Thanks

[@Proxyone444]

❤

[@noitnettaattention]

I must calm down and concentrate on studying,... studyyyyying.... Not the beautiful and dreamy blue eyed teacher girl, with hypnotic voice, alluring smile, wet gorgeous hair ... ehhm, ok where was I ?....

[@Tobs_]

reported to youtube, clickbait video, no actual pandas 🐼 in this video.

[@PythonSimplified]

That's exactly why I hope KZbin doesn't recommend this video to bear lovers!!! 🤣🤣🤣🤣🤣

[@Nico_cl]

🤣@@PythonSimplified

[@munawarabbasi9683]

How do I report this comment?

[@Tobs_]

just ring your local police station.

[@PythonSimplified]

@@munawarabbasi9683 Tobs was joking 😉

[@cyber.33]

salam dokhtari?

[@mateusnanet]

0 calls is quite impressive.

[@PythonSimplified]

💯 Agreed!! At first - I was setting the encoding to "latin-1" (I always do it for this specific dataset). And I couldn't understand why read_csv was falling back to CPU!!! It was driving me crazy!! But when I finally removed the encoding argument - the GPU finally kicked in and saved the day!! Great Success!!! 😀😀😀

[@SoloRush-hl8jv]

speeeeeeeeeedddddddddddddddd

[@bgnikstov]

You are so funny your voice is like AI creature

[@advanced_code_made_easy]

I love you ❤

[@mikevladi]

A great walk-through! Do not want to be a purist, but "kernel" is pronounced kErnel (stress on the first syllable), not kerNel [kzbin.info/www/bejne/b3zWe4Wvj69jrNE]. Google for "pronounce kernel". That's a small thing, really, but can make a big difference in some contexts. Unless, you do this on purpose …. 🙃

[@PythonSimplified]

I'm an immigrant with an accent... if I could pronounce everything perfectly - I definitely would! 🙃

[@mikevladi]

@@PythonSimplified And I am officially your subscriber now ;-) 💯%

[@PythonSimplified]

@@mikevladi welcome aboard! :)

[@cleofaspintolimalima1627]

or beautiful voice

[@cstrq]

just use polars

[@PythonSimplified]

Polars is definitely faster than Pandas... but is it 30 times faster like my cuDF example? 🤔 Checkout the challenge at minute 19:09 if you think that Polars beats cuDF, I'd love to see your solution! 😀

[@mokasurvey]

I love you 😍

[@diwakar_tsn]

I think I'm first here 😊

[@PythonSimplified]

Almost! @tanishaness got here first 🤪 (but still very fast!!!! a perfect match to the spirit of this tutorial hahaha 🤣)

[@stephenwatson8981]

CPU -- 45s GPU - 1.02s

[@louiss3409]

Python has serious security problems.

[@cristianlopez1988]

I love You....

[@SDRicky]

Generational humans, we love girls Programmers : code , code, code................................and python chick

[@evy_parody9320]

Share your LinkedIn profile please. I need to follow you and I have a bunch of questions plus I know you definitely have some resources I can use. Love your work

[@10x_discovery]

Could you make a collaboration with @NicholasRenotte. All the best insh'Allah