15 Python Libraries Every Data Engineer Needs!

[@mehdio]

What's your go-to Python Library for data engineering?

[@syedsaifuddintaj]

What are your opinions on great expectations library for data validation?

[@mehdio]

I consider these kinds to be 'full-fledged data quality frameworks.' They can be either open source (such as Great Expectations and Soda) or paid (like MonteCarlo and Bigeye). Thus, they go beyond mere libraries with functions to install and use. That being said, I find Great Expectations to be quite heavy, with many dependencies even for smaller tasks-I would definitely recommend giving Soda.io a try! :)

[@asnaeb-er3wz]

I'm hooked not only on data engineering topics but on your presentation style. Are there any other resources you would recommend to a beginner? Also, i just learned pandas, should I jump to spark or polars? Thank you Medih. Subscribed!

[@joneskiller8]

I did not know about Logger Guru, but now I know. I am also curious about the Ice Cream library.