Master Pandas: Boost Performance with These 3 Pro Tips

[@tirthasg]

Great content! Subbed :)

[@trentdoesmath]

What's your biggest challenge when working with large datasets in pandas? 🤔 Drop your questions or share your experiences in the comments.

[@totoarifiyanto8679]

Liked, subscribed, and shared. Another pro tips & tricks, please.

[@trentdoesmath]

Thank you for the support and encouragement! I'm on it :)

[@christianglennabarollo4060]

Can you explain the difference between vector processing versus chunking? thanks much for your super detailed videos

[@trentdoesmath]

Thanks so much 😎 To your question, the way I would think of it is a single column/series/array is a vector - vector processing is performing operations across an array e.g. [1,2,3] + [3,4,5] = [4, 6, 8], this would be vector addition - this is more efficient than writing a for loop and performing addition on the individual scalars. Chunking splits your data into a subset of rows - this is useful if you can't fit a full set of data in memory at a time. The problem is when you do aggregations (group bys) because assumably you want to group by an entire column, but a chunk limits the set of rows you operate on. Does this answer your question? I am making assumptions as to what you mean by 'vector processing'. Thanks again for your support 🙏

[@FIBONACCIVEGA]

Great content! subscribed 🫡My biggest challenge now is to know how to make a workflow in Airflow . I usually extracted the information from the data warehouse using SQL and when I finished my analysis I made my presentation in Tableau. But I never was asked at the same time automate it , to be integrated with my data analysis with Python. I had never been asked that before. I did not know how to work with large databases and how to automate so that it could be visualized. Of course I use Tableau but it was a mystery how to do all this in a single project. I keep reading to learn due in any interview they are asking me for same.

[@trentdoesmath]

I agree, the end to end workflow is challenging 🤔 it will take me a while but I will make content in this 👌