How to combine DataFrames in Pandas | Merge, Join, Concat, & Append

In this video we go over how to combine DataFrames using merge, join, concat, and append. We also discuss the different join types and how to use them in pandas. It is explained how to stack DataFrames on top of one another and how to stich DataFrames together side-by-side.
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
$15 off Annual Dataquest subscription
app.dataquest.io/referral-signup/qybqz3r8/
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Did you find this video helpful? Consider subscribing for weekly tips, tricks, and tutorials.
/ @chartexplorers
Join my Discord Server
/ discord
Reference: The Real Python
realpython.com...
0:00 Intro
1:09 Merge (side-by-side)
6:40 Join Types
8:15 Merge Errors
9:55 Join
10:43 Concat (Stack)
12:48 Append

[@ttolu3992]

The most simplified tutorial I’ve seen about this topic, now I understand perfectly. Putting all tables and codes in discussion side by side made it easy. Thank you 🙏🏾

[@manavsaxena5579]

Amazing content, I am more of a visual learner and you have explained the concepts brilliantly with examples. Please keep it coming, I am gonna binge watch all yours videos. Subscribed!

[@ChartExplorers]

Hi Manav, thanks for your kind words and support!

[@chrisbagshaw5676]

Ugh I have a deadline tomorrow and this video actually saved me

[@ChartExplorers]

Wow, I'm glad this helped!

[@akshaypatil-qq9jv]

ya Saved me too 😅 Thanks

[@hadidy13]

Thanks so much! The only detailed explanation on KZbin, every other tutorial just write the code and you should guess what really happened.

[@leongkern6746]

you just saved my final year project sir! Much respect for the concise and visual learning material

[@khbye2411]

Note to self1: At 10:37 the df1.join(df2, on='ID', lsuffix='_l', rsuffix='_r') is the same as df1.merge(df2, how='left', left_on='ID', right_index=True, suffixes=('_l', '_r'))

[@manangoyal5026]

Amazing video! Wish I could like it 1000s of times myself. Within 13 minutes, using rightly put graphics you've taught these 4 topics to me with such an ease, that's highly admirable! Please keep making more videos, the world could use more people like you :D

[@AnilAnvesh]

The best video on KZbin on #Python #Pandas Merge/Join/Concatenate/Append. Gratitude for your efforts ❤️

[@ChartExplorers]

Thanks Anil! 😀

[@PeaceAction6464]

Very simple and easy to understand explanation, the concept has been explained in almost the first minute

[@PP-im6lu]

Wow. I can't believe how clear this explanation is! It helped me a lot! Just one thing though. It seems the concat (inner) is not exactly the same as the merge (default). concat (inner) seems to just get rid of the rows or the columns with NaNs, but merge would make sure there are matching values (whereas concat (inner) doesn't).

[@KarlosEPM]

Good observation. Also, merge, unless used on index, generates an entirely new index. This can be problematic if you are dealing with large datasets and are using few columns, relying on index for identification of values (rows). In the latter case, I create a column with the indexes (I hope someone smarter and more experienced shares a better solution :p). Cheers!

[@rolfbecker4512]

The explanations are easy to understand and comprehensive. Perfect! I will recommend your material to my students. Thank you!

[@ChartExplorers]

Thank you Rolf!

[@anthimostsirigotis5483]

Thank you man!!! In just 12min I managed to understand everything. I follow you from now on!

[@AlexFariaOliveira]

THANK YOU! I was days trying to merge 2 lists and was not finding a way your video made it all clear and half way though it I stopped and solved my problem!

[@kamrulfst]

Thanks for your video about merge,concate method. Its more clear to me now.

[@user-zv5hu2oj6m]

Great video that just clear my questions!

[@sashi6]

Practical and concise explanation. Enjoyed watching your learning videos

[@hedibena3697]

I would like to thank you for sharing this knowledge in an incredible and such easy way! This really helped me, thanks a lot!

[@leulsegedmesfin7239]

amazing explanation man,thanks.

[@keifer7813]

Watching this on my lunch break because my manager thinks I'm an excel whizz. She told me to run some macros but really I'm using python to clean up some excel files lmao

[@ellyothim355]

Superb brother, Wish I met this channel earlier

[@TippsausEuropa]

Coooool. Clear and easy and very useful . Best regards from Technischen Universität Berlin

[@trustedai]

Excellent tutorial with lucid and simplified explanations.

[@hetpatel7918]

I was very confused about this topic until i watched this video great explanation

[@ChartExplorers]

That's great to hear! Thank you.

[@quornnugget7799]

At 9:20 you spoke about the data type error. I am struggling with this as my table will not change the data type no matter what I try. Do you have any fixes for this?

[@splashstrike]

You're a great teacher. Subscribed.

[@ChartExplorers]

Thanks!

[@munirbhatti1103]

You are making too good videos, Please make more videos on python, Data science and ML. It will be really helpful. Thankyou!

[@Rishabh_Sar_Di]

Hey...why am I getting Error as 'You are trying to merge on float64 and object columns' while using pd.merge(df1,df2) ?? Though u could easily merge float64 and object columns !! Plzz assist

[@darkmatter4701]

thank you for making this video it really helped with my problem that i was stuck in, thank you.

[@hoaralatoan845]

Super clearly, a helpful explanation. Thank you so much.

[@Olimpico230]

thank you so much sir, you saved me too much stress

[@eldhose101]

Thankyou , this really helped me to understand.

[@InteligenciadeNegocios]

Thank you!

[@talalsafaq]

Ver Ver Very Good explaining ,, thank you very much

[@prizepig]

Exactly what I was looking for. Thanks!

[@Nightmare78hAlo]

A note for anyone getting errors for append, apparently that is now deprecated as of 2023/2024 for pandas , ie it's not part of pandas anymore so you can ignore append.

[@helengh5493]

Hi , I love the video , that was so clear and useful would you please explain what is the best way to merge and or concat the data frame which are not created in current working directory , should we read them one by one or is there any other short way?

[@Puma_178]

hi, I loved your video, very clear explanation and good example, thank you

[@user-ro3rw9vp4f]

Absolutely brilliant 🔥 Thank you very much!

[@TraderSam96]

Simplified,thanks alot

[@keerthanasrinivas474]

This was exactly what I needed. Thank you.

[@mooventhan110]

Neat and Clean Easy to understand Thanks man :-)

[@ChartExplorers]

Thanks! This is something that always tripped me up, so I'm glad you found it easy to understand 😀

[@josesanchezpalacios8991]

With pd.concat([df1,df2], join='inner') I was expecting to see Col_4 once ? why is repeated ? any way of avoiding that?

[@vivekshindeVivekShinde]

amazing loved the visual explaination

[@JuanPerez-iu9vk]

Great! Thank you very much!

[@aandyrea]

Thank you SOOOO much! This was perfect!

[@mohammadyahya78]

Thank you very much. Very clear explanation. At 4:26 please, is there a way to avoid comparing the row that has same ID from datrafeme 1 to all these that have same ID in row 2 please?

[@idongessien2245]

HI, I'm trying to merge datasets but keep getting memory error issues. How can I get pass that? Now my computer just freezes each time I attempt. I always have to forcefully shut down my laptop and reboot it all over again. still no progress made. Please help

[@basicallyeditor]

Hi, I need to merge oldxl data and new xldata, label column is same in both xls I need to do outer join. If label value is same in both xls it should keep the row of new xl Can you pls help in this. Thank you

[@neuralNudge123]

hey, sooo useful tahnks. Could you please please please share the ntb?

[@inthemakinh]

if we want to merge the text in all the cells of a row or column into a single cell, or just keep the merged cell then how can we do that.

[@business_central]

Great video and love the visuals ! Just one question, any video/explanation on merge_asof techniques

[@noone-kg8ry]

what if the identifiers have the same values but they don't have the same titel for example: product_id and item_id. How do i merge these sort of dataframes?

[@sarthakshandilya8412]

hey, thanx for such a good explanation 😊😊

[@richarda1630]

fantastic tutorial! very well done. I was a bit confused by the "join" section starting at 10.00 It doesn't seem to be joining on the index. Could you explain that a bit more?

[@ChartExplorers]

Thanks Richard. If you delete the on='ID' argument then it should join on the index by default. I see that you are on discord so I'll explain in more detail there.

[@richarda1630]

@@ChartExplorers Thanks so much! LF to it! :)

[@jananisivasankar9784]

kzbin.info... It's a great explanation on this topic..i understand it very well.. I also got confused in join topic after 10.00.. it's given on = 'ID' but default left and also inner join not happened based on the common values of that ID column..need some explanation here to understand this .pls help

[@sarbanchakraborty2053]

Nice explanations, Thanx

[@NikhilSangle]

I have to combine 34 pyspark df to one df and all df has different no of rows and only one column. Pls any one can let me know......

[@kawsydaisy]

Helpful! Thank you!

[@jimmyhilman5612]

Cannot use df.append(), dataframe attribute append not found, any ideas ?

[@rhysbennett1667]

This Helped me out so much! Thank you!

[@ChartExplorers]

I'm so glad!😀

[@samt3825]

it was really well said

[@dishantshah4383]

Amazing Video.

[@vikram2105]

Very nicely explained

[@BlueSkyGoldSun]

So Join and merge are just the same in pandas and they are equivalent for SQL joins ?

[@Raja-cp8uo]

How to add two rows when column value same in pandas?

[@easydatascience2508]

you can watch mine. The playlists provide most of the fundamentals for Python and R.

[@bonefishboards]

Perfect. Thanks!

[@iliesboukhatem7803]

a great video, thank you so much

[@BridgeHK]

Hi Your explanation is so clear and easy to follow. Do you have a Github and people can run the code directly? Thanks

[@ChartExplorers]

Hi @Bridge2019, thank you. Here is a link to my GitHub with the code I used for the video. github.com/bvalgard/combine-dataframes

[@BridgeHK]

@@ChartExplorers Thanks! You may put the Github link under your description next time. People can access it immediately . Thanks anyway!

[@ChartExplorers]

@@BridgeHK Great idea. Thanks!

[@magi4k]

Amzing explanation to be honest

[@tarblood]

thanks Mr.

[@chilledvibes5700]

very efficient video.

[@freedoom4090]

I'm geting crazy... at 7:25 : (...) f = pd.merge([zero, a], how="right") (try to run and get:) TypeError: merge() missing 1 required positional argument: 'right' I don't know what else to do. Please, someone give me a light about were's my mistake, 'cos I'm about to broke my notebook in two :(

[@ChartExplorers]

It is looking for the second (i.e., right) dataframe. Because you added the brakets around the two dataframes it thinks it is one argument. Try it WITHOUT the brackets around your two dataframes. You want to use brakets [ ] with concat not merge. You will also need to specify what you are merging on f = pd.merge(zero, a, how='right')

[@freedoom4090]

@@ChartExplorers it worked! thanks! you saved me haha

[@ChartExplorers]

@@freedoom4090 Awesome, I'm glad it worked!

[@lala-jy4kz]

Is there any source code? I want to use it as lecture notes to review.

[@abegailherrera3337]

Thank you 😊

[@pankajgoikar4158]

You are superhero

[@bonnyongom1176]

Nice 👍

[@tindo0038]

amazing

[@injilaislam2275]

why does concat didn't match the value on basis of the column values in the inner join at 12.38

[@ChartExplorers]

I'm not sure what you are asking? Inner join on axis = 0 will match the columns that match between both dataframes. In this case that is the ID column and Col_4. It doesn't match with the original pd.concat([df1, df2)] because the original is doing an outer join, but the 12:38 example is doing an inner join. Does that address your concern?

[@injilaislam2275]

@@ChartExplorers hi thankyou for replying, my concern is regarding inner join, why here inner join is fetching values which are not common ,ie value from second index row, according to the law inner join only fetch common value , it more seems like outer join at 12.38

[@ChartExplorers]

Hi @Injila Islam, good critical thinking! What is happening here is that because axis is set to 0 we are grabbing the matching values based on columns not rows. You will notice when axis=1 and join='inner' (@12:30) what you are expecting is what happens. @12:38 however it is grabbing the similar columns (ID and Col_4) that are shared between the two dataframes.

[@ram-my6fl]

This is G.O.A.T shit

[@ChartExplorers]

Thanks!!!

[@hanyelshafey2722]

Thanks

[@ChartExplorers]

You're welcome!

[@shahzan525]

👍👍

[@ryansossou646]

note:: append going to be remove from future version of pandas

[@nandineeuma6659]

How to combine multiple row into single row separated with comma using python

[@ChartExplorers]

Could you provide more details about your specific situation? Do you really want to combine rows? Or are you trying to combine columns? combining multiple columns: df['combined_columns'] = [', '.join(str(x) for x in y) for y in map(list, df[['col_1', 'col_2','col_3']].values)] For this switch out df with the name of your dataframe (two places) switch out the column names with your column names (here i have three columns but you could use more or less depending on your needs) If you are combining rows are you then appending that new row to the dataframe? or do you just want a variable to store that information? combining multiple rows: rows_combined = dfm[:5]['col_1'].values

[@nandineeuma6659]

@@ChartExplorers Thanks you for the response want to combine rows only for example if i combine values in A1:A5 cells and the concatenated values should be in B1 Cell, sample input is A1:X0007DFG A2:X0006FGH A3:X0005HJK OUTPUT X0007DFG,X0006FGH,X0005HJK

[@richardnes1403]

👏🏻👏🏻👏🏻

[@user-nm8wf5dh2v]

It's possible a better video than this one?