Python Pandas - Loading Multiple files into DataFrame - DataMites Courses

[@ravinderreddi]

I was looking for a wildcard to pick all the file in a folder since 3 hrs, you have made my life easy. I have been through all the complex code available online nothing worked, your code is simple and does the job.

[@DataMites]

Thank you for your comment.

[@jobaute3984]

I was looking for this and didn't realise it was that simple! Thanks

[@DataMites]

Thank you and welcome

[@allieubisse470]

I need for of this in my data Science life. You make everything light and easy. One concept per video. Short and sweet.

[@ThankGodItsToday]

Glad it helped :-)

[@bhayj]

I checked many web-pages in google and nothing helped, but your video solved my issue. Thanks a lot!!!😀

[@DataMites]

Glad to hear that

[@ATHARVA89]

just what i was looking for in my work

[@VivianHarbison]

Thank you for such a succinct, and thorough video. Just what I needed. I followed along and got my files to all come together with one viewing of your video.

[@DataMites]

Glad it helped!

[@darkphysics4561]

Wow...you made it look so simple and saved me tons of time with this video. Many thanks.

[@DataMites]

You're welcome!

[@odeyemitemitope53]

This just helped solved the issue i am having. Thank you!

[@DataMites]

Glad to hear it! Keep Watching

[@MultiKasuku]

You Sir deserve 1 billion subscribers

[@DataMites]

Thank you. Please do subscribe and share with friends.

[@karinabrouwer2733]

Thank you for explaining this. Ive watch many tutorials on YT but nothing worked but yours did.

[@DataMites]

"Hi Karina Brouwer, thank you for your comment."

[@yichienchong7474]

Thank you sir! Your video saved me lots of time.

[@DataMites]

You're welcome!

[@judytsai9013]

Hello Sir! Thank you very much for sharing it. It helps a lot on the obstacles I’m having. I referenced this video to upload multiple .JSON, would you mind share suggestions if I want to flatten the nested node before concatenate files. Thank you!

[@DataMites]

Glad it was helpful!

[@Abel_The_]

thank you. you are a life saver

[@DataMites]

Glad it helped!

[@timothyhanson3694]

Is there a way to include a column which shows what file the data has been pulled from?

[@nikitamaru4198]

Thank you sir

[@harithameka8328]

thank you for the video sir i merge my multiple csv files into excel file output is it leaving one row when it printing the new file when it printing 10.csv it leaving 10rows then the data is going to print plz reply sir

[@manojkarthicksriramulu8639]

thanks for the concept ....the logic you explained very simple to understand for a beginners

[@DataMites]

Glad it was helpful!

[@manojkarthicksriramulu8639]

Can you explain..... Detailed concept of data cleaning null values, NaN, replace concept..... How to set data types.....

[@multitaskprueba1]

You are a genius! Thank you! What if the files share some columns but, we do not know which ones?

[@DataMites]

Can you please reframe your question?

[@GodX36999]

Yeah thank for your great videos. But dear sir, if a folder have too much more file but we just want to pick some of them and concatenate as horizontal . So how can we do sir

[@pallabiroy9091]

Very helpful. Thankyou

[@DataMites]

Glad it was helpful!

[@riyazbagban9190]

sir i have plain text how can i add these text file in each cell or i have to add each text file in each row of datastet how to do it

[@nidhi_singh9494]

thank you for wonderful explanation. my query is , I have .csv files which are needed to drop first 20 rows in every files , and after that I needed to concat those all . any suggestions?

[@DataMites]

Use simple for loop to remove first 20 records from every file and then concatenate those files

[@cynthiaphillips7143]

Thank you very much. so now how can i save the new cancate data in txt file?

[@DataMites]

Check with np.savetxt() or df_object.to_csv('filename.txt')

[@amolmzope]

Thank you so much sir

[@DataMites]

All the best

[@gulshankumar17]

I use dask data frame for reading multiple file. something like import dask.dataframe as dd df = dd.read_csv('*.txt').compute()

[@DataMites]

Good work

[@salamalsamman1314]

Thank you!

[@DataMites]

You're welcome!

[@mohammadtamimkashifi9570]

Great work!

[@DataMites]

Thanks!

[@chinmaya.6934]

Sir, can you tell how to add a new column of "source file name" in the dataframe? ) Purpose is like to know source of the data where it is coming from

[@ThankGodItsToday]

You can directly assign it using dataframe.loc it creates column if doesn't exist

[@chinmaya.6934]

@@ThankGodItsToday Thanks for the response. Will check it out!

[@dheerajkumark5602]

Sir plz make more pandas ,dask and scikit learn Videos

[@ThankGodItsToday]

Sure.. will take time for this.

[@yousfoss4367]

Please how to behave if you have three groups one of test file another of train file and the main file thks

[@DataMites]

Hi YOUS FOSS, you can manually set the location of files and read it using python.

[@yousfoss4367]

@@DataMites thks a lot

[@lilymaths4584]

Glob is such a useful but funny library!

[@DataMites]

"Hi Lily Maths, yes, glob is useful library."

[@pierreclaude1345]

Hi, thank you for great video. My question is how to include encoding='utf-8' ?

[@DataMites]

Using 'encoding' parameter in read_csv(). Kindly refer www.w3.org/International/questions/qa-what-is-encoding

[@aakash2402]

Can we follow same procedure for JSON files as well?

[@DataMites]

Hi Aakash Thorat, yes you can provided that all json has similar structure and it can be loaded with pandas as dataframe

[@satishkumar-ir9wy]

Sir can you help me to read multiple parquet files from miltiple directories

[@DataMites]

If we try to read more than one file the data gets overlapped. What we can do is to read different files to different variables and then concatenate them to a single variable.

[@NishantKumar-dw5er]

Awesome

[@DataMites]

Thank You!

[@tayseeruthman9525]

THANK YOU VERY MUCH

[@DataMites]

You're welcome!

[@sarahferko2257]

Thank you! What if you are trying to do something similar with PDF files?

[@DataMites]

Hi Sarah Ferko, to work with pdf file there are lots of other packages like textract, PyPDF2, etc.

[@hazelerturk9314]

Thanks a lot for this! I have two empty files in 207 files. I believe it gives an error for those files in between. Is there a solution for those cases?

[@DataMites]

Hi you can always check if your file is empty or not.

[@yousfoss4367]

thks a lot. what about multiple text files, please. thks

[@DataMites]

Hi YOUS FOSS, you can use for loop for txt files line a given directory and do processing like shown in video.

[@yousfoss4367]

@@DataMites thks a lot

[@GodX36999]

@@DataMites thank sir, but how can we do if we just wanna pick some of them with a list specified file name ( not all file in folder)

[@pennylaine8703]

Thank you! :)

[@DataMites]

You're welcome!

[@mariakosberg9244]

Hi, I really liked this video however i am struggling to do the same with my 120 log files. The reasons for this is that the columns are not defined in the files. They exist in the files but on the same row as a descriptor for the files. I have 3 rows with descriptors on their own but then on the 4th row the descriptor shares the row with the columns names. Also, all the data set is separated with spaces so I need to set the delim_whitespaces= True function but i can't do that with concat. I can fixed the file when I used the the delim_whitespaces functions and adding 'names' = each column name but i don't manage with the concat function. Is anyone able to assist?

[@DataMites]

"Hi Maria Kosberg, Based on your query, you can remove top 'n' lines from your file by opening and saving a new txt file without those lines (you can automate this if you need to remove the same number(n) of top lines that you don't want in your data frame). After that, you can go with the approach suggested in the video itself. "

[@sagarbhanage1832]

I tried it but my file showing empty.

[@DataMites]

Check the file path