What Tools Should Data Engineers Know In 2024 - 100 Days Of Data Engineering

[@avikchatterjee7854]

As an absolute clueless beginner, have to say you are doing an incredible job introducing this subject. No fancy jargon, just pure simple English for anyone to understand.

[@SeattleDataGuy]

Thank you! I am so glad to hear that, any thoughts on what the next few videos should be?

[@avikchatterjee7854]

@@SeattleDataGuy I really feel you are taking the channel in the right direction. You have cleared most of the general broader explanatory and introductory stuff and you are now creating in depth tutorials, how to actually use stuff; just keep on with that. Also maybe include some real life case studies and incidents similar to the style of that video where Hussein Nasser delved into that Uber shifting from Postgress to MySQL. And DE, being the tool/skill intensive and ever rapidly changing stream it is, you will never be short of content tbf atleast for the next 5 yrs.

[@SeattleDataGuy]

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

[@NakkaAbhishek-u5l]

This video perfectly explains where to start with and where most of the focus should be.

[@SeattleDataGuy]

I am glad you found it helpful!

[@NakkaAbhishek-u5l]

@@SeattleDataGuy Thanks Ben. I'm already on my 100 day challenge. Will mention you once I complete it. You are an inspiration!! ✨. Thank you. Can you please make an updated video on how to search for Data engineering jobs and what roles are good for beginners/ entry level

[@oreballin]

Hey, fresh subscriber. I think it's gonna be 1 of my favorites. However I have to go refresh my SQL. Gotta find a free usable terminal now

[@Boredom_Incarnate]

I'm not sure where else to ask this so here's my shot in the dark; I've been a Data Engineer for 2.5 years now and things are getting bad at work (typical management change issues, favoritism at work causing bad decisions etc) and trying to find another job even just at my own level is depressing. I already got my Azure Data Fundamentals certification and am working towards my DP-203 (Azure Data Engineering) but it doesn't seem to be doing anything for my call-back rate on applications. I feel like the big issue besides years of experience is that my current employer is very narrow-minded with their data so besides Data Factory and T-SQL we don't use much, so I can't list things like Synapse or Databricks on my resume in good faith. All this to ask; How do you get claimable experience with some of these things if your current employer doesn't use them?

[@danielsoto9876]

Maybe working on personal projects or looking around and seeing if you can contribute to open source stuff

[@mustafakara7739]

You might change your job.

[@MerryMurrZ]

1:29 ....didn't expect this meme on this channel...lol.

[@Machiavelli698]

The sound at @15:57 is hilarious. But great video thank you.

[@SeattleDataGuy]

hahaha thanks i guess. Glad you enjoyed the video!

[@raphaeldayan]

awesome video, thank you!

[@SeattleDataGuy]

Glad you liked it!

[@karszn]

Can you make a video on how dbt is used to make a software engineering type of transformations? Thank you and everything regarding it

[@SeattleDataGuy]

Thanks for the comment. I am curious, what do you mean by software engineering type?

[@karszn]

@@SeattleDataGuy I see many companies moving towards Dbt for ELT style of DE. From what I’ve seen online I know they follow software engineering principles to perform the transformations. What I couldn’t find was a good video on how they perform the end to end project . I’m reading this book called Data Engineering with Dbt by Roberto zagni to understand more on DBT. If you could make a video on that it would be great . I understand your explanations much better . Thank you again :)

[@SeattleDataGuy]

@@karszn Sounds like an interesting video! I often see dbt at mid-sized companies, i feel like larger companies tend to use more enterprise level tools

[@karszn]

@@SeattleDataGuy yes . Same and thank you again for your videos and content . Appreciate it

[@franzuaandrez5591]

what do you guys think about pentaho data integration ?

[@SravaniReddy-f6b]

can you create a dedicated vedios on datawarehouse concepts,it will be helpfull for me!

[@SeattleDataGuy]

I have put together a few around data modeling, is that what you mean or do you mean like data warehouse technology?

[@francisco_carvalho_]

I love this content. Thank you so much

[@SeattleDataGuy]

Glad you enjoy it! Any other content you'd like to see!

[@x5HighDesert]

I've liked Prefect over Airflow

[@SeattleDataGuy]

I’d love to know why!

[@firstshield9507]

I just started a job as trainee in data engineering, I just know Python (Numpy, Pandas, Matplotlib) and Advanced SQL and some concepts like ETL/ELT, EDA etc. I dont know anything about cloud, is it a good time to get a cloud certification as AZ-900 or it is more recommended to learn Spark before? (my company provide a voucher of 100% in Azure certifications)

[@sandeepkumarv7366]

Learning Spark on top of a cloud is the way!

[@omkarjadhav8482]

Can you share your LinkedIn please?

[@sahafhussain6751]

Where did you learn Advanced SQL? Any tips on it?

[@tommydang7682]

This is super helpful!!!

[@SeattleDataGuy]

Glad you think so! Let me know if there is any other content you'd like to see!

[@Aliaseita]

Can you make video about analyst engineer

[@St0Ries-new-i2o]

On the Open source Big Data Platform Which best tools anyone can consider for data quality, data processing (Spark ), Data storage etc.?

[@geraldinechan1652]

What the Kimble means here when you mentioned how you find the difference between data warehouse and databases? Thanks a lot.

[@firasfekih6596]

What about microsoft fabric? Ur expectation for it!

[@namnguyenngoc1948]

Do u think that airflow just do orchestration work not relating to compute, and use other external compute tools to do that work, so it's not need to learn more about executor of airflow and implement something like kubernetes cluster to run airflow and scale airflow... because it just do orchestration work.

[@slayerindisguise]

what about r?

[@SeattleDataGuy]

r isn't really great for data engineering. I am sure some people use it, but it's not ideal and is more suited towards DS projects.

[@St0Ries-new-i2o]

13:38 SSIS instead SSAS ??

[@Syed-A-Rizvi]

so how much sql do I need? I know data science folks need expert level sql

[@DANG3R0USS]

🎉🎉🎉🎉🎉

[@SeattleDataGuy]

Wooo hoo!

[@sushrafshaikh7996]

He is using indian meme in his video...,🤣

[@SeattleDataGuy]

wait which one!

[@sushrafshaikh7996]

@1:31 He is a famous actor AKSHAY KUMAR (PHIR HERA PHERI) Movie scene It's comedy movie In india n numbers of memes are there for this movie.... People use this meme to tease someone who is in love or who did anything sexual...or any intimacy topics....which one you have used....

[@richardtvtv]

Ok at this point would it be easier to list skills Data Engineers shouldn't be learning now 👀👀

[@SeattleDataGuy]

It’s getting close

[@reddixiecrat]

Are you planning on covering Palantir anytime soon?

[@cestlachance7575]

Is this really a good video? i feel like he just namedrops every techs

[@DanielKamau-ku5cs]

Not clearly explained, just bs .

[@nadlev123]

You started nice, but then your moved to name dropping if all the latest buzz words in the industry. Snowflake, DBT, azure, etc. are not tools, those are just different type of technologies you might be using as part of your work