Process HUGE Data Sets in Pandas

Today we learn how to process huge data sets in Pandas, by using chunks.
◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾◾
📚 Programming Books & Merch 📚
🐍 The Python Bible Book: www.neuralnine...
💻 The Algorithm Bible Book: www.neuralnine...
👕 Programming Merch: www.neuralnine...
🌐 Social Media & Contact 🌐
📱 Website: www.neuralnine...
📷 Instagram: / neuralnine
🐦 Twitter: / neuralnine
🤵 LinkedIn: / neuralnine
📁 GitHub: github.com/Neu...
🎙 Discord: / discord
🎵 Outro Music From: www.bensound.com/

[@Open5to6]

I can't always follow everything he says, cause he moves pretty quick and throws a lot at you, but he's always straight to the point, no fluff, and innovative. I always glean more things to look up after hearing it from NeuralNine first.

[@DonaldRennie]

Very good! I'm a beginner, and this guy spent more time explaining this topic than DataCamp. The only thing I didn't understand was the "series" part.

[@thisoldproperty]

I like the simplicity. Wonder if a similar thing could be done with sql queries given they usually store incredibly large datasets.

[@jaysont5311]

I thought I read that you could, I could be wrong tho

[@mikecripps2011]

Yes, do it all day long. I read 2.5. billion records a new level for me this week on a wimpy PC. I chunk it by 200 K Rows normally.

[@nuhuhbruhbruh]

@@mikecripps2011 the whole point of SQL databases is that you can directly manipulate arbitrary amounts of data without having to load it all in memory though, so you don't need to do any chunking, just let the database run the query and retrieve the processed output

[@maloukemallouke9735]

thanks but how you deal with depending row like times series data or observations like text where context correletead to row?

[@mainak222]

I have the same question, do you have an answer?

[@leythecg]

wie immer top content perfekt präsentiert!

[@TomKnudsen]

Thank you.. Could you please make a tutorial on how you would stip out certain elements from a file that is not your typical "list", "csv" or "json".. Find this task to be the most confusing and difficult things you can do in Python. If needed, I can provide you with a text file which include information about airports such as runways, elevation, etc. Perhaps there are some way to clean such file up or even convert it to a json/excel/csv etc.

[@lilDaveist]

Can you explain what you mean? List is a data structure inside Python, csv is a file format (comma separated values), and json is also a file format (JavaScript Object Notation). If you have a file which incorporates many different ways of storing data you have either manually or in a script way copied a file line by line and pasted it in another file.

[@kavetisaikumar]

What kind of file are you referring to here?

[@Ngoc-KTVHCM]

In excel file, method "pd.read_excel" has no parameter "chunksize", how to handling the big data in many sheet in excel? Please help me!

[@aniv6346]

Thanks a ton ! This is very helpful !

[@goku-np5bk]

why would you use csv format instead of parquet or hdf5 for large datasets?

[@chrisl.9750]

pandas, for example, doesn't read parquet in chunks. CSV is still relevant for small, easy data transfers.

[@lakshay1168]

Your explanation is very good can you do a video on the Python project that else the position of an eye

[@hynguyen1794]

i'm a simple man, i see vim, i press like

[@csblueboy85]

Great video thanks

[@franklimmaciel]

Thanks!

[@wzqdhr]

The hard part is how to append the new feature back to the original dataset without loading them in one shot

[@siddheshphaple342]

How can I connect database in python, and how to optimise it if I have 60L+ records in it

[@wildchildhep]

it works! thanks!

[@uzeyirktk6732]

how we can further work on it. Suppose if want to use groupby function on column [ 'A '].

[@15handersson16]

By experimenting yourself

[@tcgvsocg1458]

i was litteraly watch a video when you post a new video...i like that!(8)

[@artabra1019]

OMG tnx im trying to open csv file with million data then my pc collapse so i find some i9 computer with 16gb ram to open it thanks now i can open big files using pandas.

[@FabioRBelotto]

Can we use each chunk to spawn a new process and do it in parallel?

[@Supercukr]

That would defeat the purpose of saving the RAM

[@ramaronin]

brilliant!

[@JuanCarlosMH]

Awesome!

[@tauseefmemon2331]

Why was the RAM increasing? should not it stop increasing once the data is loaded?

[@thisoldproperty]

It takes a while to load 4GB into memory. So the shown example was during the process load.

[@vishkerai9229]

is this faster than Dask?

[@MegaLukyBoy]

Is pickle better?

[@WilliamDean127]

Still would load all data at one time

[@hkpeaks]

Benchmark (Pandas vs Peaks vs Polars) kzbin.info/www/bejne/Z3zRZ2lrdqmGmc0

[@RidingWithGerdas]

Or with really huge datasets, use Koalas, interface is pretty much the same as pandas

[@Zonno5]

Provided you have access to scalable compute clusters. Recently Spark got a pandas API so koalas has sort of become unnecessary for that purpose.

[@RidingWithGerdas]

@@Zonno5 talking about pyspark?

[@pasqualegu]

all workеd

[@ashraf_isb]

1000th like 😀

[@imclowdy]

Awesome! First comment :D

[@driouichelmahdi]

Thank you