Complete Python Pandas Data Science Tutorial! (Reading CSV/Excel files, Sorting, Filtering, Groupby)

[@KeithGalli]

Hey ya'll! I created a second channel with more Python content (including additional Pandas tips & tricks). Please consider subscribing 😊 kzbin.info

[@sam7250ii]

You cleverly edited the code between 25:50 to 25:59 list(df.columns.values) to list(df.columns)😉👍

[@Vribejs]

Error:Cannot mask with non-boolean array containing NA / NaN values - gives me error when usinf df.loc (on 40:49 in video)? df.loc[df['Our Global Company'].str.contains('Smith', regex=True)]: this is code, I imported another .xlsx table when practising.

[@yidizhou9899]

@@Vribejs go google it... you can't expect him to do it for you. He checked the documentation just to give us a good overview of pandas.... google out your error if not you will not learn.

[@chiraggupta1897]

i have been working on a excelworkbook having 8 worksheet and i m performing operations on data nd want to place dataframe in the 6 sheet in place of its data .but everytime i do all other sheets gets vanished nd a single gets get formed with the dataframe .plzz help me in appending df into an existing excel

[@benten5018]

Hey Keith , can can please help me to download the csv.file on an android tablet. sorry for bad english.

[@jcspaziano]

I know this is 5 years old but I learned more about using Pandas from this one video than all the other videos ive watched on the topic combined! Just awesome! Thank you!

[@KeithGalli]

Glad that it is still helpful!!

[@_Nelyen]

This video was super helpful, thank you Keith! In case anyone gets to the end of this video, around 48:00, Keith talks about the groupby operator and starts to go over the section "Aggregate Statistic using Groupby (Sum, Mean Counting)". You might run into errors due something that changed after Pandas version 2.0.0. Instead of writing: df.groupby(["Type 1"]).mean() Try writing: df.groupby(["Type 1"]).mean(numeric_only=True) After version 2.0.0 the numeric_only value was changed to False versus True as it's default, causing errors such as "can not convert strings". Hope this is helpful, have a good one!

[@ljubicabrenjo2794]

Thank you very much, I ran into the problem, this is really helpful! :)

[@mayowafele9691]

Thank you

[@NachtHere]

Thanks Man .

[@mrme2120]

Thanks Dude

[@rajdevanshu65]

Was facing the same issue, thanks a lot.

[@piotr5830]

Hi Keith - not sure you will read this but wanted to sincerely thank you for this tutorial. 3 years ago this was the first python video I ever watched after graduating from unrelated subject. Today I'm typing this from a business class lounge at JFK, on my way to London where I just got a job as a quant developer at a hedge fund, building pricing models and infra for trading. Worked hard for this but if not for your videos I could be at a very different place. Thank you from the bottom of my heart, your work means a lot to many people. Cheers!

[@Eizengoldt]

Bro hire me

[@mydailystruggles]

can you tell us more about your journey? :)

[@mcxanbur]

screenshotted and favourited

[@nikithroumpari2553]

A strugling biologist here thanks you! We are mostly dealing with big data and it can get a little overwhelming, but you made it a lot easier!

[@waterwanderer.mattyy]

Awesome

[@jankucera8505]

big data in a csv file? lol

[@mfundokhumalo3933]

Hey, you might enjoy SAS

[@manan-543]

they dont lie when they say data is everywhere and every field needs data scientists

[@prubin18]

Great video! One of the best pandas tutorials I've seen. I have one comment though. When you run (at 40:00) df.loc[df['Name'].str.contains('Mega')]) You are actually including Meganium in this filter, even though it is not a Mega pokemon. So, one needs to include a space after Mega, such as: df.loc[df['Name'].str.contains('Mega ')]) One can see that this makes a difference because when you run len(df.loc[df['Name'].str.contains('Mega')])) and len(df.loc[df['Name'].str.contains('Mega ')])), to know the number of rows, there are two distinct outputs (respectively 49 and 48)

[@KeithGalli]

Video Outline! 0:45 - Why Pandas? 1:46 - Installing Pandas 2:03 - Getting the data used in this video 3:50 - Loading the data into Pandas (CSVs, Excel, TXTs, etc.) 8:49 - Reading Data (Getting Rows, Columns, Cells, Headers, etc.) 13:10 - Iterate through each Row 14:11 - Getting rows based on a specific condition 15:47 - High Level description of your data (min, max, mean, std dev, etc.) 16:24 - Sorting Values (Alphabetically, Numerically) 18:19 - Making Changes to the DataFrame 18:56 - Adding a column 21:22 - Deleting a column 22:14 - Summing Multiple Columns to Create new Column. 24:14 - Rearranging columns 28:06 - Saving our Data (CSV, Excel, TXT, etc.) 31:47 - Filtering Data (based on multiple conditions) 35:40 - Reset Index 37:41 - Regex Filtering (filter based on textual patterns) 43:08 - Conditional Changes 47:57 - Aggregate Statistics using Groupby (Sum, Mean, Counting) 54:53 - Working with large amounts of data (setting chunksize) Thanks for watching friends! :) Let me know if you have any questions

[@dtran288]

YES!!! THANK YOU!

[@shadow2frost325]

Thank you so much for posting this! I have a test in Python soon, so I've been watching this for a review. You explain everything so well and make it easy to follow. I also like how the data was from Pokémon - it makes it more relatable.

[@dchitan1234]

great tutorial

[@tejasnareshsuvarna7948]

A reference notes to help you while you watch the video. docs.google.com/document/d/16qcfjwLp1vV-5VnIOGuDC2vxkHQ534_RzQd2Gihk7x8/edit?usp=sharing

[@Tropax1]

Hey dude, love this video by the way but I have a question, can this data be used for machine learning? I have my exams coming up where I have to find a dataset to make predictions and stuff. Are these pokemon cards, do they have label and features if you understand what i'm talking about? Any help would be greatly appreaciated. Thanks in advance.

[@amiliavachford183]

thanks for useful video If anybody have a problem with calculating the mean of Type 1 grouped data, use this: df= pd.read_csv('modified.csv') df.groupby(['Type 1']).mean(numeric_only=True) instand of this: df= pd.read_csv('modified.csv') df.groupby(['Type 1']).mean() That way, it won't include string-type data in the mean and sum functions.

[@vissokis]

thanks it helped a lot...can't understand the error while all the values are numreic already

[@llamaland1737]

so is it got updated now, since you can only perform the method on int or float columns ...

[@Tristana_21]

thanks! help a lot

[@RisingLoaf]

This 1 hour video did more for me than entire semester of my Data Analysis course... Amazing

[@juliachristiaanse2985]

SAME DUDE omg

[@braxion-xyz]

me too!

[@iahtesamalam]

Same here bro

[@mrman880]

Were you paying attention?

[@uchindamiphiri1381]

am a self taught but this one saved me

[@faizalimuhammadzoda4731]

There is something to the way Keith teaches that keeps me coming back. Besides being a good teacher and utilizing techniques which help people grasp the material quickly and remember for long time, he sends forth a wave of positivism. He is such a positive, energetic person. Thanks for sharing your knowledge. May it grow and enable you to bless more people with it.

[@hughjazz8416]

I have bought multiple Udemy courses on pandas and this one blows them all out of the water, and it’s free! I’m deff subbing!

[@dicspringdkz8234]

Keith You are more than a teacher. Your level of simplicity in explaining Python in details is out of the moon. Keep up the good work. Your video is always my “go to” any time. Again, thanks a lot for using your skills as a blessing to people around the world.

[@adedokunagunbiade5324]

I watched the entire video in 30 minutes and learned more than I did with hours of video content. Amazing work.

[@shayonghoshroy7208]

Best Pandas tutorial on KZbin, especially 24:25

[@evangelosspyromilios5994]

hahahahaa

[@cr2093]

LOL

[@fernandourrutia2566]

jaajajajaaj

[@arielsfitness7073]

Huge hhhhh

[@pappymu]

29:19 is where he's getting texts from actual pandas

[@MatBat__]

Bro I started a data science internship in the beggining of the Year, we use a lot of pandas and you are saving my life from day 1. Thanks again, you are a god send! Subbed on both channels, cheers!

[@garthhorne617]

I have been learning python and using pandas for about 3 months now and done innumerable searches on the internet with questions regarding use of specific statements and coding. I wished I had come across your video earlier! You are a born teacher and know how to layout and explain complex terms and concepts. How can someone that looks so young have such a strong grasp on presentation and user needs? The concepts you explain are the same things I have sought information on for 3 months but all in one place and succinctly explained. Thank you for all your work.

[@Orion3000k]

Mannnn your one of the best Python go-tos PERIOD. Straight to the point and easy to understand. thanks for teaching us all!

[@pivo6499]

I can't believe I watched this for free, thank you so much!

[@johnwiley1221]

This was pretty good. I would also check udemy or r/learnpython for other free resources. Found a 30 hour FREE pandas course there the other day

[@johnwiley1221]

www.udemy.com/course/the-ultimate-pandas-bootcamp-advanced-python-data-analysis/?couponCode=FF041817B54B4BC9EB6B

[@quartercast]

@@johnwiley1221 It's not free now, unfortunately :(

[@musclemusic123]

ki

[@shambhav9534]

The documentation is also free.

[@nimaonta1725]

Dude you deserved all the subs for this video alone. You explained everything so good. keep it up :)

[@crtnnn]

Started my PhD in hydrogeology and learning Python from the scratch. I love your work, keep it up!

[@gustinelimurilo]

53:30 you can use .size() to get the count of each Pokemon type instead of adding a new column. It would look like this: df.groupby(['Type 1']).size() Great tutorial!!

[@LureUnitFtw]

One of the best tutorial that I've ever seen in KZbin! Thumbs UP!

[@AndrewMann205]

Between jobs for the first time in decades I wanted to learn data science using software other than just Excel and Access. Your video was well explained and frankly better than anything else I have seen so far involving Python and Pandas. Thank you for a job well done.

[@bentrash7885]

Awesome tutorial! One advice I'd have for any python developers is to get in practice of working within virtual environments. Really helps to avoid conflicts when you're working on a project which may require some older versions of a library but your other projects may require latest ones, stuff like that.

[@bjbmbc]

Gold medal bro, I was searching extensively for a good data science resource and reddit just sent me to random coursera/edx courses that used to be free but don't appear to be anymore. Your content is highly organized, extremely concise, and well thought out. There is a reason that only .01% of the votes are downvotes. THANK YOU!

[@remy0705]

This 1 hour course is all I need for my data analysis course. This is the best video I found on KZbin. Thanks ❤️❤️❤️

[@RockIT1]

I like the way he interacts with his viewers

[@jiangxu3895]

I just went through your numpy tutorial. And that's the reason I come here. Thumb up!

[@KeithGalli]

Appreciate it!!

[@kanstantsinhupalau6337]

Saved my day! I started learning Pandas, but when I missed several months during circumstances and this video about basics helped me quick comeback. Thank you!

[@hautboisjc]

Came here after watching your "Real World Data Science Tasks with Python" video and I didn't expect there are still things I don't know with Pandas. Thank you! Liked and subscribed.

[@kylieying2]

Thanks for posting! As an MIT student taking a data analysis class, this video was very helpful, more useful than the other tutorials online!!

[@kipishism]

Found it very useful too!

[@kregg34]

"As an MIT student" Weird flex but ok

[@klauscheang7063]

Excellent!! I like the way you organize the videos on different topics and functions of working with data. Please make more videos on how to work data science in Python. E.g. Statistical analysis (descriptive statistics, t-test, linear regression) or data processing tutorial (like what we do in SQL).

[@MiguelMusic123]

This video helped my massively! Been learning through online python courses with people trying to act and saying unnatural jokes, but your video felt super natural and easy to watch. Many thanks!

[@MichaelPeterDalsgaard]

I swear this is the most useful python channel on KZbin. Top stuff.

[@disagio9517]

I came for the tutorial, stayed for the cutesy pokemon stuff, really warmed my heart

[@DavidWhitt]

Dude... you should make more videos... you are a natural born teacher!!

[@mdhidayat5706]

Awesome tutorial Keith, I learnt a lot by following your hour long tutorial. Created a new notebook instead of using the GIT version as it doesn't show what happens before you commented the code.

[@MennaMahfouz-zr1ld]

I am a freshman and I feel lucky to know this great channel. thank you, I am using your channel to learn to do my recipe recommendation system project.

[@RunyCalmera]

This was great man. Even in 2020. Only thing and suggestion. Do not change a cell when you are elaborating on a new feature. Just click a new cell down and elaborate. Because then in the jupyter notebook you will see all the variations. If you change it it won't save in the jupyter notebook. I'm very happy with this tutorial. You break it down easily. I'm new to pandas and python and this has helped me a lot with pandas.

[@bijoysaraf650]

Very simple yet comprehensive tutorial on Pandas. You had my attention throughout. I do use Pandas for data analytics along with numpy. That said I learnt quite a few tips and tricks. Thank you for sharing your knowledge. Way to go Keith! Liked and subscribed.

[@modernafsolutions3233]

Wow man! Holy smokes that was such an amazing breakdown. I came into this knowing nothing about Pandas and now I want to get back to work with my personal data! Thank you so so so so much. I’m off to find the documentation!

[@KeithGalli]

Glad you enjoyed! Your comment made my day :)

[@ProdMGD]

Great video to get people up and running. It took me two hours to watch, take notes, and test out some examples. I feel like this was time very well spent. Thank you for this.

[@Aimad_off]

I just finished your NumPy's course sir, and I'm moving now to pandas, I just want to thank you for your efforts !

[@elwinmentaram6031]

2 years after this video was posted, I'm here watching and learning Tons of stuff. Thanks man!!!!

[@narayangautam6955]

Me tooo today i watched it Comedy 😂😂😂

[@bharathianjeneya2111]

On point Keith. 5 hrs worth training covered in an hour. Made my day.

[@DennisGorshteyn]

You break down all the details in a way that I can't believe this is for free. Very high quality stuff. I was up and running with this library in short order

[@nutrathriveyoutube7056]

This is an amazing tutorial! Please keep publishing like this. very well explained! I would love to see about matplotlib, numpy and if you can get inside machine learning

[@atraps7882]

Day 1 on my journey to learn data analysis with python, this vid and kaggle's free pandas course is just what i needed to give me more motivation to keep learning.

[@8rameshb]

The best tutorial I have seen so far on data analytics. I now see how python/pandas helps in data analytics. Thank you very much for making and sharing this video.

[@nikluz3807]

this is an excellent tutorial, especially the filtering/conditional changes section. I have always loved how google sheets has built in queries, and I wanted to be able to do a lot of the same things using pandas. This essentially gave me all of the power I needed! thanks!

[@jamesdonly518]

Ok I've been learning Pandas for a while now, over many different sources, and this one video has shown me much more helpful little hints and tips than all of the other material I've looked at previously!!! Thannnnnk you! Please do more Pandas stuff as this has been so awesome =]

[@brandongarza1366]

I haven't started this yet, but based on your previous videos I know this is going to be great. Thanks Keith, you are a great teacher.

[@philippedid]

A big thanks for your work from France . I have learned a lot about Pandas .

[@duyanh1823]

I am a Pokemon fan, randomly watch Python Panda for my project and find this. Such a big help. Thanks KEITH!

[@Diegtz555]

Wow, thanks for this tutorial. I'm starting on python and took a course of udemy, but it was confusing, with your explanations many doubts are cleared up. Thanks Keith:)

[@andyn6053]

WOW! This was just what I have been looking for! Fantastic tutorial! You explained everything very well and clear from start to finish. Best Pandas tutorial on youtube for sure! Thanks man :)

[@DrewLevitt]

In the chunksize section, you pick a well-documented bad practice, namely calling pd.concat inside a for loop. As the loop runs repeatedly, this operation becomes more and more expensive (because new_df gets longer and longer). Per the pandas documentation, the better approach is to append each df to a list and then pd.concat the list elements just once, after the for loop.

[@terabhaininja9]

Hello, can you please provide with a tutorial for that? Quite new and clueless here.

[@terabhaininja9]

dataHere = [] for chunk in pd.read_csv('modified.csv', chunksize=5): dataHere.append(chunk) newnew = pd.concat(dataHere) This looks right?

[@Illegirl09]

A fellow man of culture I see. Funny enough as I was multi-tasking through Python tutorial videos while playing x2 Switch SwSh pokemon. I'm nearly at the end of your tutorial video and so far, good job on this. Keep it up!

[@martistarti2374]

Omgeeeeee!!!! Thank you so much!!! I've searched sooooo many videos trying to help with the delimiter problem I've had (i didn't know that was the problem) and you're the ONLY one I've found that even mentions it!!! 🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾🙌🏾

[@bensondube5646]

Excellent Tutorial Keith. Very clear, at the right speed and interesting to learn from. This material is very suitable for a self learner. Keep it up.

[@cdgxflower2679]

I've been looking for a good pandas and python video for quite sometime now. I have to say that this is really amazing. You've explained it so well that a beginner like me could easily understand. Great job and thank you. Can't wait for more videos. (if possible, matplotlib)

[@orfeaspapaioannou2755]

dude this is an amazing introduction to pandas. Really helpful, thanks a lot

[@jesseraines870]

"STOP TEXTING ME IM MAKING A VIDEO! WHO HAS THE NERVE!?" 😆😆 Great video bro. Currently in a term of data science with python and struggling hard. This video has been tremendously helpful! Big thank you!!

[@piggeh6465]

Will now recommend this video to anyone who is interested in learning pandas! This video is awesome

[@paulblades2325]

Thank you so much for your time and effort. This is the best python tutorial I have watched. Straight forward and well organized. I appreciate the time stamps.

[@xnick_uy]

27:15 It seems that the dataframe got scrambled up a bit there, most likely from having the cell running multiple times. Even when there was an error message, it appears that either the Total or the Legendary column was moved to the left of HP. Upon running the cell again (with the corrected version?) it calculated a new Total adding the previous values and generating corrupted results.

[@cindyshaw2485]

Thank you, Keith, for making this super helpful tutorial. You're a great teacher!

[@kostas6915]

Apart from maybe silencing your phone while doing the tutorial, you are definately the man for the job!! Great work and VERY helpful!!!

[@indraneel6601]

Day 1 : 18:27 Day 2 : 55:00 Day 3 : completed Tq for the brief intro to pandas just for tracking my progress.

[@takako230]

Awesome video Keith! I'm a beginner programmer but your explanation is super clear! Thanks for the videos:)

[@RPGMadnessVX]

F for "MEGAnium" that got filtered out while being old school Poke 😂 Great course! Looking forward to learn more from you!

[@iftrejom]

I fixed it by writing "Mega " in the code.

[@williambaker4915]

52:49 You can also just write the code: df.groupby(['Type 1']).count()['Name'] That way, you don't have to add the count column.

[@TomNeedhamNeDrum]

This was genuinly so helpful, thank you! I am mostly through a data science course and have been struggling to figure out actual applications for the information I have learned. This was excellent!

[@KeithGalli]

Glad you found it helpful!!

[@tototoysentertainment9483]

i watched more than 10 different videos about pandas, this is the most easy and understandable one. Worth your time!

[@bidhanbhattarai8863]

Makes me want to play the old Emerald games again, wonderful tutorial, keep them coming

[@philipcoppage3592]

SQL person w/ limited exposure to Python here. This was useful as hell.

[@viveknayak9899]

Comprehensive, perfectly paced.... Lovely tutorial!

[@patiencelenkouengansop6440]

As a new to python language . this video is very helpful to me and your way to explain is making this easy to understand. Hopefully by the end of they year I will be good at it.

[@MikeStallings2023]

I am a retired software guy, enjoying your videos, thanks for these! You are making me almost want to go back to work again.

[@NoName-fi2ow]

What the hell, I imagined this topic in afternoon and video recommended after only few hours. And the shocking fact I didn't even searched about this topic from many days.

[@sahajsandhuv8]

thast AI at work

[@stephanierodriguez1035]

This was such a great introduction to pandas and on DataFrame. This is exactly what I was looking for. Since I hadn't previously downloaded pandas onto my mac, and didn't feel like installing anaconda either, I was running into some troubles installing pandas with just "pip install pandas" so I thought I would include the instructions as to how I did it. simply do: pip install pandas --user If nose and tornado aren’t downloaded do: pip install nose --user then pip install tornado --user (nose needs to be installed first) then terminal also suggested I add it to my path, so I did: sudo nano /etc/paths add the path at the end of the file do ^X and then Y then hit enter

[@skyblue021]

Thank you Keith for this video, absolutely amazing and valuable for many! THANK YOU!

[@KeithGalli]

Glad you found it helpful! :)

[@immanuelsuleiman7550]

it's been a year since I first saw this video pandas has been the best thing to happen to me and this is where it all started thank you Keith

[@KeithGalli]

That's awesome, really happy to hear that this video had an impact on you. You are very welcome :)

[@immanuelsuleiman7550]

@@KeithGalli I'm seriously impressed that you replied to both of my comments Take care and stay safe

[@joashbrijit]

You've just got me 30% of my whole assignment. Thanks dude

[@micsierra806]

Excellent tutorial; exactly what I was looking for. Liked and subbed. Thank you for sharing your expertise.

[@nicoledeasis664]

"stop texting me! I'm making a video!" "who has the nerve" hahahahahahha you explained well, thank you.

[@fixithvac9360]

that is funny :))

[@BrandonS-lk2qc]

I learned so much, thank you. Then at the end...that music tho. I lost it! LOL! Did not see it coming.

[@ranganathabr3321]

This is just insanely helpful. I find the originality of this guy very attractive. I have been wandering around trying to figure out my big data problem since I am new to pandas, and this video has been a great help. I was legit shivering frantically and now I feel more confident. THANKS, A LOT MATE.

[@Godspeed9114]

The way ur teaching it like sipping a wine 😜🍷 so easy. Thank you so much for making kifeeasier withur thoughts and Bro literally ur teaching is. Way awesome keep rocking!!! All love from India😍(Hyderabad)

[@pemadechen9901]

I loved the fact you used pokemon as data set it was fun learning I could also check my knowledge about pokemon hahha Love love

[@jasonaraosfuentes2130]

This is an extremely usefull tutorial. You explain so good bro. Thank you very much. Like and subscribed. Hugs.

[@SMFahim-vo5zn]

When I start making money with these knowledge, I'll give you some share!

[@HansOnProduction1984]

Keith, I stumbled across your video from random search on deeper understanding of pandas . I felt like you did a great job presenting the material. Well done man, it was easy to follow and understand. I did appreciate the part at the end with the chunk size and group by - would like further explanation of those concepts. Thanks.

[@georgeimus6102]

I just realized this a Keith video I haven't even watched it yet and I already know it's gonna be good! My data science guru

[@jsaylor525]

"It's not super important that you know about a DataFrame"? That's one of the main objects in pandas, I'd say its highly important.

[@SuperBhavanishankar]

"stop texting' im making a video!" x'D xD

[@royerzurumluoglu2466]

Who has the nerve

[@jyotithasal1261]

concentrate on topic only.

Lmao

[@Kevin-cy2dr]

Timestamp pls

[@damilolaadeyemi8383]

@@Kevin-cy2dr 29:30

[@paradise-flossed]

All the Meganium fans when he filters out for Name Contains "Mega": _intense sobs_

[@CrystalSunriseFox]

exactly my thought.. also yanmega

[@tancaken9990]

Wow, i actually only remember Meganium.

[@hhbbhvvbjhbbyjj]

the best python tutorials i ve seen

[@gillesderoo2027]

You are the GOAT. Your explanations using Pokemon makes so much sense.

[@saurabh-patil]

This tutorial helped me alot. Thank you so much!