The Struggles We Skipped: Data Engineering for the TikTok Generation [PyCon DE & PyData Berlin 2024]

🔊 Recorded at PyCon DE & PyData Berlin 2024, 24.04.2024
2024.pycon.de/...
🎓 Watch as two junior data engineers share their unique perspectives on navigating the complexities of data engineering and how Python libraries like dlt are revolutionizing the process, making it easier and more efficient for the TikTok generation.
Speakers:
Anuun, Hiba Jamal
Description:
Anuun, a Developer Relations professional at dltHub, and Hiba Jamal, a Data Science Working Student at the same company, presented a talk titled "The Struggles We Skipped: Data Engineering for the TikTok Generation." The talk highlighted the evolving landscape of data engineering, especially in the context of Python libraries like dlt making tasks more accessible to the current generation. They focused on the challenges faced by junior data engineers in constructing data pipelines and how tools like dlt simplify tasks such as data extraction, transformation, and loading.
The speakers shared their experiences, with Anuun discussing the importance of understanding the complexities of data pipeline engineering, while Hiba shared her journey of transitioning from manual data handling to leveraging automation tools like dlt. They showcased how dlt, an open-source Python library, streamlines the process of extracting, normalizing, and loading data from various sources in just a few lines of code. dlt's role in automating tasks that were traditionally cumbersome was emphasized, making it a valuable addition to any company's data stack.
The talk provided a step-by-step narrative on the challenges of building data pipelines and how dlt addresses these issues, enabling junior data engineers to take on tasks that were previously considered more senior-level. The shift towards greater levels of abstraction in data engineering, facilitated by tools like dlt's declarative incremental loading, was highlighted. An informative and engaging session, the talk underscored the importance of understanding the nuances of data engineering,
⭐️ About PyCon DE & PyData Berlin:
The PyCon DE & PyData conference unite the Python, AI, and data science communities, offering a unique platform for collaboration and innovation. The PyCon DE & PyData Berlin 2024 conference, hosted in partnership with the local Berlin PyData chapter, provided an exceptional experience, fostering deeper connections within the Python community while showcasing advancements in AI and data science. Attendees enjoyed a diverse and engaging program, solidifying the event as a highlight for Python and AI enthusiasts nationwide.
Follow us:
• LinkedIn: / 28908640
• X: www.x.com/pyconde
• X: www.x.com/pyda...
Links:
• Conference website: pycon.de
• Related sessions: 2024.pycon.de/p...
The conference is organized by
• Python Softwareverband e.V.: pysv.org
• NumFOCUS Inc.: numfocus.org
• Pioneers Hub gemeinnützige GmbH: pioneershub.org
If you enjoyed this session, please like, comment, and subscribe to our channel for more insightful talks and discussions.
Share this video with your network to spread the knowledge!
Hashtags:
#Python #PyConDE #PyData #OpenSource #AI #DataScience #MachineLearning #SoftwareDevelopment #LLMs #Community
Acknowledgements:
Special thanks to all the volunteers and sponsors who made this event possible.
About:
Python Softwareverband e.V.:
PySV is a non-profit that promotes the use and development of Python in Germany through events, education, and advocacy, fostering an open Python community.
NumFOCUS Inc.
supports open-source scientific computing by providing financial and logistical support to key projects like NumPy and Jupyter, promoting sustainable development and collaboration.
Pioneers Hub gemeinnützige GmbH:
is a non-profit fostering innovation in AI and tech by connecting experts and promoting knowledge exchange through events and collaborative initiatives.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.