Solving real-world data analysis problems with Python Pandas! (Lego dataset analysis)

[@KeithGalli]

Level up your data science skills with courses, projects, and competitions offered by DataCamp! Use my link below and check out the first chapter of any course for FREE! :) datacamp.pxf.io/c/3588040/1012793/13294

[@masternobody1896]

can you do some google job coding. so how can i get a job

[@KeithGalli]

Big shout-out to my mom for not throwing away my Legos! She's the real MVP

[@bobbyg603]

Thanks mom!

[@vishwasjajpura796]

Finally Keith will build his LEGO

[@ocraking]

nice Kevin Durant reference

[@KenJee_ds]

dude, loved the intro!

[@KeithGalli]

Hahaha thanks man :). Very happy that my mom didn't throw out all of my legos!

[@markomarjanovic8348]

Absolutely love the raw natural style you are doing, hope everyone else appreciates it too, keep going buddy, you are amazing!

[@DataProfessor]

Wow the Lego stop motion was awesome!

[@ahsanshah1866]

Data professor is here 😀

[@rafaelmello8194]

I'm a begginer in Python and I'm learning a lot from you. You are an awesome teacher. Your pacing and didactic are perfect. Thanks a lot for your effort

[@rksingh1997mp]

He’s back baby!!

[@alan6506305]

God, this is brilliant. I watched the other two videos of yours on Pandas. You are a great teacher and friend. Thank you very much for your hard work and kindness.

[@simonvanwijk5178]

Man so good to have you back! If it was not for you I would have not gotten a role as a DA as you helped me the most in the beginning.

[@lVaNeSsA90]

Thanks for being honest while you search for syntax in the beginning. Love this raw, step by step video. I'm using your videos on my project to get inspired ❤️ thanks for being a good tutor 😊

[@logannon]

Dude, I thought you were dead. Your videos have helped me so much. Glad to see you back!

[@leomiao5959]

The man is back. The hero is back for us!!

[@thebeeskhakis7145]

I'm so happy you're back. Your videos helped me get my new job!

[@FIBONACCIVEGA]

This video has been a true inspiration to continue learning. I'm doing the datacamp since I want to change my field and I've always liked programming and analyzing data. But he didn't know if he could use the learned knowledge to use it in real life. Now I know that everything I have learned is what is used in real life data analysis. Saludos

[@amansorout.6779]

Happy to see you back, fighting with something serious, you are not alone.

[@weitingteng3241]

Great great and great to see you back

[@stratascratch]

Good to see you’re back!

[@JW-pu1uk]

I really like the thought process in these videos. It's very raw, and really will translate well to an actual work project.

[@PaYaMv2]

Good to have you back my dude! Loooooooved this!

[@danielsantoyo2640]

Im so happy to see you are back! Panda and Numpy tutorials would be great !!! I’m currently trying to learn panda and numpy for data analytics and this video was super interesting !!! Thanks Keith keep going you are doing great 💯

[@Omzodijacky]

Man , I'm happy you are back ! you were truly missed

[@H99x2]

These type of videos are your strengths! Great tutorial and explanation Keith

[@YunusFidan_]

Good to see you uploading again!!

[@cyrilodoi6868]

So good to have you back man! 💯

[@terrytas13]

Welcome back Keith, so good to see your face again. Stay well my friend!

[@KeithGalli]

Glad to be back!! :)

[@qalinlekhaliif5518]

Thanks a lot man. Your videos are helpful and entertaining as well. We appreciate your great work.

[@Sensei10238]

Finally back! It helped me a lot in learning python! Thank you so much!

[@ben-tiki]

Another great video Keith! Glad to see yo back. Awesome that you got to work with datacamp. Please if you can make a video o OpenAI it would be awesome. Ive been using their API and its awesome

[@itsReshad]

Love the great content! Please dont stop! You have an impeccable way of teaching its amazing

[@tuandino6990]

I've been waiting for this

[@MashiroRedo]

Waited so long! Thank you

[@ocraking]

Dude, you ROCK

[@terrytas13]

Love the introduction!!!

[@Viralvlogvideos]

welcome back to your first tutorial after long back :P

[@kartikeyasharma9908]

Hi Keith, loving the video tutorials!

[@dharshankumar2522]

Keith is back...yeahhhh

[@1990andstillgoing]

props for sharing your knowledge man, its really easy to understand and apply what you're doing (Y)

[@Magmatic91]

Did this project on DataCamp. Was a lot of fun.

[@rafaelcastellarmartinez3498]

Hi Keith, just tried to do the project with you and i got that Star Wars was not the most popular theme in 2004 - Harry Potter and 2017 - Super Heroes, weird that datcamp test said ok, but i did the math manually and harry potter was the most popular in 2004, thanks for your videos. an student from Colombia Latin America!

[@adelekeemmanuel4917]

omg... i just did the exercise myself and i discovered the same thing too... Came ti check the video but im seeing something else

[@manfungnewmanyu1426]

Yeah!!! Your tutorial is very great and help me so much at the AI master course .

[@putyah]

Awesome video. Small detail: On the new era answer you typed the variable in. It would be nicer to drop every value that is Star Wars. Next select the remaining year as an variable. When the dataset is changed the variable is dynamic so the answer would still be correct.

[@KeithGalli]

Good suggestion! I agree that would be a better way to go about it :)

[@lucaspioli7970]

Love your videos! Keep going

[@sanjeetlal1873]

Legend's back❤️

[@baggid6257]

He is back~!

[@jongcheulkim7284]

Thank you, sir. I had lots of fun^^

[@azrmuradl6420]

Please provide more such kind of videos, or as you always do, give us tips about how we can find such kind of real world ds projects online.

[@ДимитърСираков-щ7ы]

keep up the good work!

[@tuandino6990]

Question 2: theme_count_by_year = licensed_lego_set.groupby('year')['parent_theme'].value_counts().unstack() theme_count_by_year.fillna(0, inplace=True) theme_count_by_year = pd.DataFrame.transpose(theme_count_by_year) Or you can use pivot_table function. By approaching in this way you can create a data frame that's easy to do plot (heatmap) and make high number pops out.

[@tuandino6990]

@Josh Yorko nice

[@davida99]

Yoooo love the vids

[@codewithkarthik7136]

nice video keith

[@aditiparashar9171]

you are freakingly smart!

[@kotharidhruv75]

w8ing fr more such videos

[@kirubaselvi6754]

Keith, Pytorch tutorial please

[@KeithGalli]

I definitely want to! I need to spend considerable time reviewing and building up my own PyTorch skills before I make a tutorial on it.

[@freddy4videos]

thank you, much love

[@ChileHeroico]

keep doing more videos pls :D

[@rodrigo100kk]

This dude is cool, this chanel too.

[@Levy957]

that task #2 was really hard to do alone

[@merterisen]

16:52 how did you change 'Star wars' text immediately?

[@KeithGalli]

Lol that was just video editing xD.

[@admonitoring-pi9os]

Hello there. I hope you are good. I am a little late with this comment because this video is already more than 2 years old but since i have started learning python now its the right time for me. where can i find the codes you explained in the video bcz no code is availbale in the project file at the github provided link.

[@guisande]

Hey Keith, I'm divided between going towards data science or cyber security. I love both but I kinda needs to make money by now. Do you think I can own money in a short time in data science? Working as a freelancer or supporting small companies... Edit: I'm glad that you came back. Really love your videos

[@adeshmishra1671]

Go for Cybersecurity brother, Since difficulty level is medium.. But while earning 💰 you can also learn data scientist!!

[@ratchakoon]

themes.csv which you provided on github does not have 'is_licensed' field. Is 'parent_id' filed as same as 'is_licensed' field?

[@KeithGalli]

A little confusing, but you want to use parent_themes.csv, not themes.csv !!

[@ratchakoon]

@@KeithGalli Thank you

[@raghavgoyal3324]

please upload a project every week

[@KeithGalli]

I'll try my best!

[@damarbowo]

Can I see your membership playlist? I can't find that playlist

[@KeithGalli]

Hmm I'm not sure what you are asking to see, can you clarify?

[@damarbowo]

@@KeithGalli you have a membership benefits. One of the benefit is got playlist or videos for member. Do you have an example the video or playlist for member join your channel? Hope you understand

[@KeithGalli]

I just started my memberships last week so I haven't posted any exclusive videos there yet. To get an idea of the types of content I'll post there, check out these videos kzbin.info/www/bejne/p5-2d2uPlrWrbZo kzbin.info/www/bejne/hZyooIN_hNypnsk

[@damarbowo]

@@KeithGalli I'll wait Keith. Regards

[@KeithGalli]

Sounds good!

[@shahoftrading]

question: when you merge when using left_on and right_on ...we get the merged df. So for the merged df and under parent_theme why are most if not all of those are "Legoland" and all IDs are 411? also how do we check the full tabular data -- print(df)?

[@baburamchaudhary159]

in line [99] ie. .groupby(['year', 'parent_theme']) and in next line: .drop_duplilcates(['year']) since we already have grouped by 'year' and 'parent_theme' [I think, it groups unique year and parent_theme] why do we need to drop duplicates by 'year'?

[@БулатМиннуллин-р8щ]

why didn't you use .agg?

[@gopikaprasad8607]

How to export the for loops result into excel?? Please reply

[@gersonchadijunior7499]

Hey Keith, I love so much your videos. I've been learning Pandas with you since your pokemon's video, but I feel that the last answer is not accurate and in fact the right year should be 2006, because it was the year with less Star Wars Sets released. Can I send you my code somehow?

[@ElianMrl]

Hey guys, would it be a good idea to use Datacamp projects in my resume?

[@nitiknayyar7659]

Damn I also started this project on Datacamp.

[@alkiviadessavoullis2021]

does anyone know why when I press continue or start project the Python Use python ... code checks gets highlighted pink and I can't work on the project ?

[@zeasammy7572]

Does DataCamp have video learning platform?

[@KeithGalli]

The typical structure of classes is short videos that overview the concepts and then a bunch of interactive problems with a code editor to drill down the technical side of those concepts.

[@sabbirahmed8012]

Hello Keith, can you please mention some resource to master natural language processing?

[@KeithGalli]

Hey! I actually did a PyCon lecture on NLP. That should be pretty helpful: kzbin.info/www/bejne/rKqymIqerLqgm8U

[@clayherz_]

if i solve the second question with this code, counted_2 = licensed_sets.groupby(["year", "parent_theme"])[["is_licensed"]].count() counted_2 = counted_2.reset_index().sort_values("is_licensed", ascending=False) counted_2.drop_duplicates("year").sort_values("year", ascending=True) is it wrong

[@letsjoinhands]

hello again Keith. For Q#2 I am getting a different result for new_era using this code: So the lego_all_lic is the DF containing all licensed lego set themes with the shape (1179 x 8) and that has been grouped by year to form lego_all_lic_yr. And the rest of the code I have written is quite simple to understand. Looks as if I have made a big mistake in aggregation but can't seem to locate it. lego_all_lic_yr = pd.DataFrame(lego_all_lic.groupby(by = ['year', 'parent_theme'], axis = 0).agg(Parent_Theme = ('set_num', 'count'))) lego_all_lic_yr.reset_index( inplace = True) lego_all_lic_yr.replace(to_replace = [theme for theme in lego_all_lic_yr['parent_theme'] if theme != 'Star Wars'], value = 'Others', inplace = True) lego_all_lic_yr = pd.DataFrame(lego_all_lic_yr.groupby(by = ['year', 'parent_theme'], axis = 0).agg(Parent_Theme = ('Parent_Theme', 'sum'))) lego_all_lic_yr When you look at the result it shows that 2006 was the first year in which Star Wars lost to other themes in terms of the sets released in that year.

[@letsjoinhands]

Ok so I misunderstood the Q basically. It wasn't about Star Wars themed sets vs All The Rest rather it the year in which Star Wars lost out to some other individual theme. Got the correct answer using: lego_all_lic_yr = pd.DataFrame(lego_all_lic.groupby(by = ['year', 'parent_theme'], axis = 0).agg(Parent_Theme = ('set_num', 'count'))) lego_all_lic_yr.reset_index( inplace = True) lego_all_lic_yr = pd.DataFrame(lego_all_lic_yr.groupby(by = ['year', 'parent_theme'], axis = 0).agg(Parent_Theme = ('Parent_Theme', 'sum'))) lego_all_lic_yr = lego_all_lic_yr.sort_values(by = ['year','Parent_Theme'], ascending = False) lego_all_lic_yr.head(50)

[@manu93ize]

bro Can you do a tutorial on data cleaning with Pyspark with real world example.

[@mufasao6776]

I see that you posted some of your hidden videos. Thank you.

[@rabinmainali3373]

I done it in following ways:(question 2) 1. i count each licenced film released every year. 2.Then count the only star wars film released every year 3.And i calculate the proportion of step2 and step1. Is it okey ? ,by the way the result is also 2017 for me.

[@Silly_Duck_Guy]

keith moment

[@letsjoinhands]

Hi Keith! this is how I solved Q # 1. Pls let me know if this is a bad coding practice, is acceptable or is good in your opinion. so I first made a function called is_lic. def is_lic(df_1, df_2): df_1['is_licensed'] = bool theme_1 = list(df_1['parent_theme']) theme_2 = list(df_2['name']) lic_status = list(df_2['is_licensed']) for i, s in enumerate(theme_1): for r, t in enumerate(theme_2): if s == t: df_1['is_licensed'][i] = lic_status[r] Then is_lic(lego_sets, lego_themes) Then all_themes = [ ] for r in lego_sets.itertuples(): all_themes.append([ r[6], r[1], r[7] ]). Then all_lic_themes = [x for [x, y, z] in all_themes if y is not np.NaN and z == True] star_wars = [theme for theme in all_lic_themes if theme == 'Star Wars'] the_force = int(len(star_wars)/len(all_lic_themes) * 100) the_force = 51%

[@KeithGalli]

So my biggest recommendation based on your code is to be more explicit with how you name your variables. So instead of "df_1" & "df_2" you might name those dataframes "parent_themes_df" & "lego_sets_df" respectively. Furthermore it would be better to name variables "i" & "s" something like "parent_theme_index" & "parent_theme_value". These types of changes will make your code more readable. Functionally, everything looks sound though. Nice work!

[@letsjoinhands]

@@KeithGalli thanks a bunch Keith. and now in retrospect when I think about how you were working on solving this Q in the video I realised that all the time you were using pandas built in methods to solve the Q. so yes we could use a smattering of python methods to do this (like I did) but using that libraries' built-in methods would be more simpler and advantageous most of the times. Is that correct?

[@igor-xadrezxadrez8541]

Hey, there's a red dot on your nose.

[@KeithGalli]

I got in a fight playing hockey!

[@Viralvlogvideos]

Big nose :P

[@AbhishekSharma-hy4nl]

Bro what happened to your nose😟?

[@KeithGalli]

Got into a little fight playing ice hockey! We won the game though so it's cool xD