What Tools Should Data Engineers Know In 2024 - 100 Days Of Data Engineering

What tools should a data engineer know?
Honestly this video is more of a list of tools that goes far beyond what most data engineers know but I wanted to create a video that shared a list of data engineering tools for the 100 days of data engineering video.
So here it is!
Also, if you're looking to check out some tools that I am advising for you can look at
Estuary for data ingestion
bit.ly/3Ed1RLe
And
Mage for workflow orchestration
www.mage.ai/
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/​​
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

[@avikchatterjee7854]

As an absolute clueless beginner, have to say you are doing an incredible job introducing this subject. No fancy jargon, just pure simple English for anyone to understand.

[@SeattleDataGuy]

Thank you! I am so glad to hear that, any thoughts on what the next few videos should be?

[@avikchatterjee7854]

@@SeattleDataGuy I really feel you are taking the channel in the right direction. You have cleared most of the general broader explanatory and introductory stuff and you are now creating in depth tutorials, how to actually use stuff; just keep on with that. Also maybe include some real life case studies and incidents similar to the style of that video where Hussein Nasser delved into that Uber shifting from Postgress to MySQL. And DE, being the tool/skill intensive and ever rapidly changing stream it is, you will never be short of content tbf atleast for the next 5 yrs.

[@raphaeldayan]

awesome video, thank you!

[@SeattleDataGuy]

Glad you liked it!

[@user-ww1rr5sb7w]

This video perfectly explains where to start with and where most of the focus should be.

[@SeattleDataGuy]

I am glad you found it helpful!

[@user-ww1rr5sb7w]

@@SeattleDataGuy Thanks Ben. I'm already on my 100 day challenge. Will mention you once I complete it. You are an inspiration!! ✨. Thank you. Can you please make an updated video on how to search for Data engineering jobs and what roles are good for beginners/ entry level

[@pankajtodkar7176]

On the Open source Big Data Platform Which best tools anyone can consider for data quality, data processing (Spark ), Data storage etc.?

[@Machiavelli698]

The sound at @15:57 is hilarious. But great video thank you.

[@SeattleDataGuy]

hahaha thanks i guess. Glad you enjoyed the video!

[@franzuaandrez5591]

what do you guys think about pentaho data integration ?

[@francisco_carvalho_]

I love this content. Thank you so much

[@SeattleDataGuy]

Glad you enjoy it! Any other content you'd like to see!

[@Boredom_Incarnate]

I'm not sure where else to ask this so here's my shot in the dark; I've been a Data Engineer for 2.5 years now and things are getting bad at work (typical management change issues, favoritism at work causing bad decisions etc) and trying to find another job even just at my own level is depressing. I already got my Azure Data Fundamentals certification and am working towards my DP-203 (Azure Data Engineering) but it doesn't seem to be doing anything for my call-back rate on applications. I feel like the big issue besides years of experience is that my current employer is very narrow-minded with their data so besides Data Factory and T-SQL we don't use much, so I can't list things like Synapse or Databricks on my resume in good faith. All this to ask; How do you get claimable experience with some of these things if your current employer doesn't use them?

[@danielsoto9876]

Maybe working on personal projects or looking around and seeing if you can contribute to open source stuff

[@firstshield9507]

I just started a job as trainee in data engineering, I just know Python (Numpy, Pandas, Matplotlib) and Advanced SQL and some concepts like ETL/ELT, EDA etc. I dont know anything about cloud, is it a good time to get a cloud certification as AZ-900 or it is more recommended to learn Spark before? (my company provide a voucher of 100% in Azure certifications)

[@sandeepkumarv7366]

Learning Spark on top of a cloud is the way!

[@omkarjadhav8482]

Can you share your LinkedIn please?

[@sahafhussain6751]

Where did you learn Advanced SQL? Any tips on it?

[@karszn]

Can you make a video on how dbt is used to make a software engineering type of transformations? Thank you and everything regarding it

[@SeattleDataGuy]

Thanks for the comment. I am curious, what do you mean by software engineering type?

[@karszn]

@@SeattleDataGuy I see many companies moving towards Dbt for ELT style of DE. From what I’ve seen online I know they follow software engineering principles to perform the transformations. What I couldn’t find was a good video on how they perform the end to end project . I’m reading this book called Data Engineering with Dbt by Roberto zagni to understand more on DBT. If you could make a video on that it would be great . I understand your explanations much better . Thank you again :)

[@SeattleDataGuy]

@@karszn Sounds like an interesting video! I often see dbt at mid-sized companies, i feel like larger companies tend to use more enterprise level tools

[@karszn]

@@SeattleDataGuy yes . Same and thank you again for your videos and content . Appreciate it

[@tommydang7682]

This is super helpful!!!

[@SeattleDataGuy]

Glad you think so! Let me know if there is any other content you'd like to see!

[@user-sd1hf1kj3i]

can you create a dedicated vedios on datawarehouse concepts,it will be helpfull for me!

[@SeattleDataGuy]

I have put together a few around data modeling, is that what you mean or do you mean like data warehouse technology?

[@SeattleDataGuy]

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

[@Syed-A-Rizvi]

so how much sql do I need? I know data science folks need expert level sql

[@pankajtodkar7176]

13:38 SSIS instead SSAS ??

[@Aliaseita]

Can you make video about analyst engineer

[@christiansanchez9880]

I've liked Prefect over Airflow

[@SeattleDataGuy]

I’d love to know why!

[@DANG3R0USS]

🎉🎉🎉🎉🎉

[@SeattleDataGuy]

Wooo hoo!

[@reddixiecrat]

Are you planning on covering Palantir anytime soon?

[@relaxinggospelmusic2421]

😗

[@richardtvtv]

Ok at this point would it be easier to list skills Data Engineers shouldn't be learning now 👀👀

[@SeattleDataGuy]

It’s getting close

[@sushrafshaikh7996]

He is using indian meme in his video...,🤣

[@SeattleDataGuy]

wait which one!

[@sushrafshaikh7996]

@1:31 He is a famous actor AKSHAY KUMAR (PHIR HERA PHERI) Movie scene It's comedy movie In india n numbers of memes are there for this movie.... People use this meme to tease someone who is in love or who did anything sexual...or any intimacy topics....which one you have used....

[@cestlachance7575]

Is this really a good video? i feel like he just namedrops every techs

[@DanielKamau-ku5cs]

Not clearly explained, just bs .

[@nadlev123]

You started nice, but then your moved to name dropping if all the latest buzz words in the industry. Snowflake, DBT, azure, etc. are not tools, those are just different type of technologies you might be using as part of your work