Data Science Best Practices with pandas (PyCon 2019)

[@dataschool]

Want to skip the introduction and get right to the code? Start watching here: 5:14

[@edu1113]

i read that in ur voice ...

[@BhekumuziMachael]

Kevin Markham, he's a polite Data Scientist, he always try so hard to respond to every question u ask him even on Facebook inbox. He always make sure things are clear. Best teacher ever.

[@dataschool]

Wow, thank you so much for your kind words! I truly appreciate it!

[@TheAlderFalder]

I second that.

[@Daniel_CLopes]

For two times, during this video, you had me giving you a standing ovation, while being alone in my office.

[@dataschool]

That is one of the most awesome comments I have ever received... thank you so much!! 👏👏👏 P.S. Do you remember what the two moments were? I'm super curious!

[@Daniel_CLopes]

@@dataschool You're welcome! It was when you explained to_datetime AND when teaching how to use the groupby(). For someone seeing pandas for the first time, these two functions are 2 great discoveries!

[@dataschool]

That's great to hear! I think you might like my latest video if you haven't seen it already: kzbin.info/www/bejne/iJ2smombnsxmnsU

[@ErtizaAbbas]

one of the best humble data scientist i ever encountered, he is great teacher.

[@dataschool]

Thank you so much! 🙏

[@ireneshiang978]

Hi Kevin, I really want to thank you! I work in finance industry and new to Python. I spent $200 buying a 5-star rating python beginner online course and i was totally lost during the course. I had no idea what the teacher was talking about. Then I found your tutorial video! I watched your whole "Data analysis in Python with pandas" series videos and it is SO CLEAR AND EASY TO UNDERSTAND! I spent my whole weekend to watch these videos and it's the first course that makes me eager to watch the videos one by one. I should have donated that $200 to you! I will continue to watch other series and learn from you. your tutorials really help me a lot! Thanks for so much for your contribution and your effort to make free tutorials. Will support you on Patron!

[@dataschool]

Thank you so much for your very kind comment, Irene! 🙏 It's awesome to hear that I've been helpful to you!

[@umarhussain9334]

I'm the only pandas user in my company (other than my manager whom I'm training) and I started using it about 8 months ago. Felt quite good that I was able to nail all the exercises without to much issue! Thanks, for the theoretical examples, that's where I'm sorely lacking.

[@dataschool]

That's awesome - congrats! And, I'm glad you still got value from the theoretical parts of the tutorial 👍

[@constantfear]

@@dataschool I got value from it all, my methods are extremely messy compared to yours, its always great to watch a master in his element. Will defo check out your website thanks!

[@dataschool]

Thanks you so much for your kind words! 😄

[@mizzchoc10]

I dont feel less than or stupid when I watch your tutorials like i do with many other tutorials on youtube. I get more confident and excited.

[@dataschool]

Glad I can be helpful!

[@susmitvengurlekar]

Thanks a Lot!! for this talk at pycon. I learned many things. Particularly that one does not learn why this method, why not that method, which method can be applied, unless one ponders over the methods to answer the question by oneself. I highly recommend everyone to pause the video, set a timer and do the task when he says "Do this, I am giving you n minutes". Sincerely do that as if you were at the event. Doing that is helping me a lot

[@dataschool]

That's awesome to hear! Thank you for sharing 😄

[@rgseven6557]

Honestly, this has got to be one of the best channels for data science. You are an awesome teacher indeed. Thanks a lot for your efforts!

[@dataschool]

Wow, thank you so much! 🙏

[@mateuszsmendowski2677]

A highly full-bodied and qualitative presentation. Thanks for sharing Your knowledge and experience in a splendid way!

[@dataschool]

Thank you so much!

[@bl1266]

Danke!

[@dataschool]

You're welcome!

[@saglikguzellik]

I don't usually comment. This video thought me things more than a whole normal level bootcamp. Awesome content!

[@dataschool]

Thank you! 🙌

[@dhananjaykansal8097]

It's been so little time and whenever I have some problem I just go through your videos and the answer is definitely there. I can't thank u enough Sir. #LoveFromIndia

[@dataschool]

That's awesome to hear... thank you so much for sharing! 🙌

[@erfannazari6110]

you are amazing, everything I know is mostly taught from you, Big thanks

[@dataschool]

Thank you so much!

[@jamiecoop6809]

Thank goodness a new video is up. I have been going through Data School withdrawals

[@dataschool]

Ha! I hope to put out videos more frequently in the future 👍

[@ai.simplified..]

I like the way you explain plots usage,this is a fundamental skill that anyone work with data must have.

[@dataschool]

Thanks!

[@ai.simplified..]

@@dataschool you are great ,you are like a pandas reference , every few month I came back to review what you Taught us.

[@lolkids7833]

Good content.. great use of 1.30 hours on youtube for the first time. Thank you!

[@dataschool]

That's very nice of you to say - thank you!

[@lucaslee1452]

Appreciate that sir, this content is so useful for me that I can practice it in several different ways besides, in my opinion in this lesson, you are so humble and can tell that you are so open-minded about different opinions, such a great tutorial, and a mentor. May God bless you.

[@dataschool]

Thank you so much!

[@antonhassan]

The video gives better insight of pandas. Suitable for intermediate level. Awesome!

[@dataschool]

Thanks! Glad it was helpful to you! 👍

[@ASHOKKUMAR-jm6yn]

I'm a beginner .ur session was thought-provoking and informative ...lots of love from India tnq

[@dataschool]

Thanks!

[@JessePinkman6428]

The most productive 1hr 44min video!

[@dataschool]

Thank you! That's great to hear :)

[@jieqi6341]

Oh my god, you are brilliant! I love the fact that you share your way of doing things which can be taken as an effective way of approaching data as a data scientist, I just love it, the whole tutorial, thank you so much!!! (plus I did the survey, I just can't wait for more of your tutorial of data scientists using python!! You are great!

[@dataschool]

Wow! Thank you so much for your kind words, I truly appreciate it 😊

[@dpratte]

Very thorough and nicely done, Sir. Thank you!

[@dataschool]

You're welcome!

[@Cherryck007]

Great Work!! Learnt a lot!! Thanks for sharing

[@dataschool]

My pleasure!

[@grijeshmnit]

It is been 2 years that i am watching your vedio regularly... helped me a lot. 100s thanks!!!

[@dataschool]

That's awesome to hear! 🙌

[@atulkumar7030]

That's a real gyan on Pandas! Thanks for sharing.

[@dataschool]

Thanks!

[@siddharthsoni1862]

This is an awesome talk. Thanks Kevin, I am also a great fan of your pandas playlist.

[@dataschool]

Thanks so much! Glad my videos have been helpful to you 😊

[@slimsharky29]

Great video - nice to see how to approach data questions methodically with tools/functions we learnt seperately.

[@dataschool]

Thank you so much! I appreciate your kind words.

[@queensmate5224]

its a great video to learn pandas and how to deal with the given datasets..Thanks a lot Kevin..

[@dataschool]

My pleasure!

[@abcdefghijkl7215]

I did not know about the ast library. I had a similar column containing stringified lists in a different dataset I was working on recently. And I had to go through a lot of trouble to extract meaningful data from that list. Now I know there’s a better way. Thanks

[@dataschool]

You're very welcome! Glad that was helpful to you!

[@da_ta]

Same here

[@minhucnguyen8062]

Appreciate that teacher, what an amazing tutorial

[@dataschool]

Thank you so much!

[@joshuahabash8572]

Great video mate. All the love from 2020

[@matattz]

Great tutorial! I learned a few new things

[@dataschool]

Great to hear!

[@drumpf4all]

You’re great. I’ve learned so much from you over the years.

[@dataschool]

Thanks very much for your kind words! 😊

[@aleksandramazurek1364]

thank you for the video, learned a lot! I'll be back for sure watching your other videos :)

[@dataschool]

Awesome!

[@youngzproduction7498]

It is good to watch your work. Keep us posted. I wanna learn more.

[@dataschool]

Thanks very much! If you haven't already seen my pandas video series, I recommend checking it out: kzbin.info/aero/PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y

[@savoirrepubliquecondetutor910]

Hi Kevin. This tutorial is one of the best courses on pandas if not the best, especially for people who don't have advanced level in pandas. I have been using pandas for some time but I discovered things in this course that were amazing for me. Thanks. I have a question. I would like to know if you have videos on image processing, recognition, identification with scikit image or another library. Another question is if you have videos on tensorflow.

[@dataschool]

So glad to hear that this video was helpful to you! You might also like my latest pandas video: kzbin.info/www/bejne/iJ2smombnsxmnsU Unfortunately, I don't have any videos on image processing or Tensorflow, sorry!

[@beantkapoor6915]

Hi Kevin, that was a great talk, thank you so much! I have a question/suggestion at around 15:55 in the exercise 'Which talks provoke the most online discussion?' So, in order to find out that talk we divided the 'comments' column with the 'views' and that gave us the column named 'comments_per_views'. Rather than going the other way around, that is, dividing the 'views' column with 'comments', could we just multiple the column 'comments_per_views' with 1000. For example, if a talk had 0.0022 comments_per_views, it could be interpreted as 'The talk generated around 2 comments per 1000 views.' I hope this makes sense. Again, thanks!

[@dataschool]

Glad you liked the talk! As for your comment, I don't quite follow... if I want to know the number of views per comment, the only way is to divide views by comments. You can argue about which is more interpretable (views per comment or comments per view). I get that multiplying by 1000 makes it easier to read, but if you want views per comment, then there's only one way to calculate it. Hope that helps!

[@thomashosang2595]

Fantastic talk and teaching! Thank you so much

[@dataschool]

You're welcome!

[@brendleyohmua980]

wonderful guy, excellent tutorial material. Very good voice.

[@dataschool]

Thanks so much for your kind words!

[@Diana-wu1yv]

Thank you so much Kevin!! Your tutorials and videos really save me.

[@dataschool]

Happy to help!

[@leonardoalvarado7632]

Hello, this a very interesting course, your explanations are very clear, thank you!

[@dataschool]

Thank you! 🙏

[@cristianofroes4681]

very very good... I really appreciated that. looking forward to learn much more with you. thank you very much.

[@dataschool]

Thanks! I've got many more pandas videos here: kzbin.info/aero/PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y

[@cristianofroes4681]

@@dataschool I'm gonna watch every single one. Thanks for share.

[@dataschool]

Awesome! Hope you enjoy the videos 👍

[@JessePinkman6428]

Regarding Unpacking the ratings data, replace the single quote with double quote and load with json, it worked.

[@dataschool]

Good to know! Thanks for checking, I appreciate it!

[@wheatonrecurrence9525]

this is a very good tutorial, followed all the steps, thanks!

[@dataschool]

You're welcome!

[@jaikishank]

It was a great content for beginners like me. Thanks for sharing.

[@dataschool]

Glad it was helpful!

[@thebrokeconomist6538]

I love the python community in youtube. It's so informative and welcoming.

[@dataschool]

Agreed! Python has such an excellent community. Have you ever been to the PyCon conference? I highly recommend it!

[@thebrokeconomist6538]

@@dataschool I'll definitely try to go to one in Pittsburgh if I can

[@dataschool]

Awesome! I'm already looking forward to PyCon 2020 😄

[@etan]

It's awesome 👏👏👏 thank you so much!!

[@dataschool]

You're welcome 😊

[@confidencechidiebere4509]

In the Unpack the ratings data section, you wrote a function and did not use it to unpack the ratings series. You used lambda function. Any reason why this is so?. Thank you for your excellent videos. Corey Schafer (another born teacher like you) recommended your pandas videos and you did not disappoint.

[@Robino_del_Bosquet]

Thanks Kevin for the classes, very well done and helpful, as all of your videos 😊😎. Cheers for that!!

[@dataschool]

Glad you like them! 🙌

[@jubayerhossain8812]

Great tutorial! I love the way of teaching.

[@abhishekpawar921]

45:35 probably the most underrated trick!

[@dataschool]

Thanks!

[@maksudurrahman8510]

A brief yet thorough and great practice on pandas.I appreciate this.

[@dataschool]

Thanks very much for your kind words!

[@haneulkim4902]

@14:30 is there any difference other than style of df.column and df['colum'] ?

[@kamingleung3209]

They do the same thing, so you can use either one

[@kennethstephani692]

Great presentation!

[@dataschool]

Thank you!

[@zhalie12345]

Thanks Kevin, I'm learning a lot from your videos :D Hope u have a great day!

[@dataschool]

Thank you!

[@compedium]

this is a really well done tutorial. Thx for sharing it!

[@dataschool]

Thank you!

[@frankzheng5221]

This fits my study well. Thanks

[@dataschool]

You're welcome!

[@MrBhargavafirst]

love your teaching style really these ticks are very useful!! thanks again for your help kavin

[@dataschool]

Thank you!

[@ibanguniverse811]

Wow, update again, thanks bro, amazing vids ..

[@dataschool]

Thanks for your kind words! 😄

[@the-ghost-in-the-machine1108]

Great content, thanks a lot.

[@dataschool]

You're welcome!

[@FabricioM]

Great video!

[@dataschool]

Thank you!

[@janniks.9233]

Awesome tutorial, really appreciate the talk

[@dataschool]

Thank you!

[@da_ta]

You're fantastic as always. Just to clarify what's the difference between list_of_dicts used to initiate in functions and ted.ratings_list ?

[@dataschool]

Thanks for your kind words! Regarding your question: ted.ratings_list is a pandas Series in which each element is a list, whereas list_of_dicts is a parameter to a function (and each element in ted.ratings_list is passed to that function). Hope that helps!

[@da_ta]

Thanks

[@sovitgurung4062]

Thanks a lot. You are amazing.

[@dataschool]

Thank you!

[@kashishjain78]

Wonderful talk

[@dataschool]

Thank you!

[@noahrubin375]

Such an amazing tutorial. Thank you!

[@dataschool]

You're so welcome!

[@mrmuranga]

Superb! thanks

[@mountainscott5274]

At around 1:20:22 one of the audience members offered an alternative to get_num_ratings(ted.ratings_list[0]). The alternative was pd.DataFrame(ted.ratings_list[0])['count'].sum(). However we didn't get to see how the alternative could be modified to replace ted['num_ratings'] = ted.ratings_list.apply(get_num_ratings). Would it be to just remove "[0]"? That is, pd.DataFrame(ted.ratings_list)['count'].sum()? Or would it be a range like pd.DataFrame(ted.ratings_list[0:last row number])['count'].sum()?

[@dataschool]

Great question! After the tutorial, I added some of those alternatives to the notebook so you can see the results: github.com/justmarkham/pycon-2019-tutorial/blob/master/tutorial.ipynb

[@cocum2]

Kevin, you are an angel, and thanks for help me to like to become a data scientist!

[@dataschool]

You are very welcome! 🙌

[@galymzhankenesbekov2924]

the best intstructor ever

[@dataschool]

You are so kind, thank you!

[@PIYUSHKUMAR-mc6qd]

Bro, You are Just Awesome !! This is so good.

[@evyatarsaar979]

This is so good , thanks alot!

[@dataschool]

You're welcome!

[@anandvyavahare2031]

More such videos on Matplotlib/Seaborn and end-to-end project would be just perfect...

[@dataschool]

Thanks for your suggestion!

[@srikantaghosh2386]

You are the best bro

[@kopi-tennis]

Thanks for the great video, Kevin. I learned a lot from this and your Data Analysis series. Will you be doing a talk at PyCon 2020?

[@dataschool]

Glad you liked it! Yes, I'll be speaking at PyCon 2020: www.patreon.com/posts/ill-be-teaching-33799474

[@esatuulari1842]

Excellent lesson again. How do I plot the "talks per year" as bars instead of lineplot? Maybe even having the line following the top of the bars.

[@modhua4497]

Thanks Kevin Any chance you could share your instruction material? Thanks again

[@dataschool]

github.com/justmarkham/pycon-2019-tutorial

[@SubhashKumar-er5qh]

this is the best we got, could you please do things on visualization and Numpy. Thanks!

[@dataschool]

Thanks for your suggestion!

[@kkwesterlund]

Good stuff, thank you!

[@dataschool]

You're welcome!

[@fet1612]

34:00 4. Plot the number of talks that took place each year.

[@angelashang9778]

Hi Kevin, how are you? By the end of answering Question 4, I was trying to get the bonus exercise you asked done myself: calculate the average delay between filming and publishing. I figured out that there are 10 observations' days_between_filming_publishing is negative values, which do not make sense, I was assuming... I have a feeling that among the 7 out of 10, there are probably typo possibility causing the published_date is way ahead of filming_date. Imagine, how can publishing_date is ahead of filming date, that was impossible. Not mentioning that published_date is 335 days ahead of filming date. My question is how I can replace those 7 observations? Or just simply filter them from the dataset? Please kindly advise, thank you. Angela

[@santoshchaudhary831]

You are very nice teacher.

[@dataschool]

Thank you, I appreciate it!

[@boscojay1381]

thanks for sharing kevin!

[@dataschool]

You are very welcome, John! Hope you enjoy the video, and let me know if you have any questions.

[@vaas1205]

dude i love you. really.

[@dataschool]

😊

[@rajeshd3574]

Hi Kevin, the Talk was literally awesome... You are really good at explaining the complex piece into small ones. i have one doubt in the video , like during the time 1:21, one of the audience explained we can calculate the count using the data frame, how he actually achieved this? as shown below pd.DataFrame(ted.ratings_list[0]); but how he calculated the total count by using the DataFrame. ?

[@dataschool]

I can't remember off-hand, but you might find it in my notebook here: github.com/justmarkham/pycon-2019-tutorial/blob/master/tutorial.ipynb

[@AbhishekSingh-og7kf]

Thank you so much Sir!!

[@dataschool]

You're welcome!

[@MrYamrajji]

first time i dealt with ast.literal_eval for str.dict. Thanks for the support

[@santiagorestrepo5051]

thanks for ur work

[@dataschool]

You're welcome!

[@riptorforever2]

Thanks for the lesson!!!

[@dataschool]

You're very welcome!

[@aiwithr]

Great perspective.

[@dataschool]

Thanks!

[@PierreLouvet]

Hi, thanks for the ast literal_eval trick. I am not in data but in architecture, learning on my own with free python stuff. I was stuck with a string prb and my brain just told me "you have seen smthg about a string turned into the correct data type, go look for this video with the blue frame!" Thanks ;)

[@dataschool]

Glad it helped!

[@adamhendry945]

I want to better more efficient pandas code. How do I go to there...lol. Sorry, couldn't resist. I'm sure this is a great video and the author is very knowledgeable.

[@shirkhanaslanzade2406]

Thank you

[@dataschool]

You're welcome!

[@hvsalza4068]

best teacher, thanks

[@dataschool]

You're welcome!

[@sarikadatta3706]

I assumed you are Canadian since your last name is Markham. I hope you know Markham is a city in Ontario I guess you’re in the U.S

[@dataschool]

Ha! I am familiar with Markham, Ontario and my Dad grew up in Ontario, but I am in the US!

[@bhimkarki8698]

Thank you sir. You're amazing!!

[@dataschool]

Thank you! 😊

[@KumarGaurav-pf4bs]

Can you please share how to use for loops in python pandas in detailed, Like how we create multiple calculated columns with different different columns in dataframe?

[@kishanlal676]

You're amazing as always! Do you have solutions for the bonus exercises?

[@dataschool]

Thank you for your kind words! No, unfortunately I did not have time (when preparing the tutorial) to also write up the solutions for the bonus exercises.

[@COC_SOLDIERS]

Sir please help me to become datascientist....you are good teacher...

[@dataschool]

Maybe this guide would be helpful to you? www.dataschool.io/launch-your-data-science-career-with-python/ Let me know!

[@COC_SOLDIERS]

Thankyou for your kind reply... .I appreciate it thanks alot...sir...