The Harsh Reality of Being a Data Engineer

[@SeattleDataGuy]

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

[@notjpengineer]

This is so true. I'm an Environmental Engineer who just succeed in the carrier transaction to become a Data Engineer (I've been a Data Engineer for 3 months only haha), and one of the things that I chat about with my friends is that the best thing in the technology world is that there are plenty to learn, but, at the same time, the worst thing it is also that there are plenty to learn. If you try to learn everything at once the anxiety will kick in and ruin your day. Got keep focus on the consistency, learning 1% every day is enough.

[@Neferfifi21]

Congrats on transitionning ! How long did it take you?

[@notjpengineer]

@@Neferfifi21 Thanks! It took me 3 months of pure studying data science, but during my bachelor's degree in Environmental Engineering I had already seen programming and done a data analysis project. Now I'm a Trainee Data Engineer, which allows me to study along the way, since the initial plan was to become a data scientist.

[@SeattleDataGuy]

Congrats on the new role! It is a double edged sword for sure. You always have something else to learn or read!

[@hotpeppermovie]

You have to accept the fact that you will not be able to learn everything. But the beauty of it is that most of your problem is just one Google away

[@JohnDoe19991]

Dealing with raw data is 1000 times better than dealing with business people who have -100 IQ when you are data Scientist/Analyst.

[@AhmedSayed-ki3fm]

Couldn’t agree more 👍

[@lauren185]

😂😂😂

[@tomastruchly9484]

I agree with you only partially concerning the old vs new technologies usage. Yes, you can stay working with Oracle or SQL Server for next X years, but the number of jobs will slowly drop and it will come time to switch. However is very different to switch when you're in your thirties or when in you're fifties. I switched from on premise world to Cloud solutions like Snowflake or AWS this year and even though I am 35 can feel it's not that easy as when I was 25 (I had no family, was ultra-motivated and literally craved for knowledge)

[@SeattleDataGuy]

Oh no...I am crossing 30 soon. Does it go downhill from here 😅

[@tomastruchly9484]

@@SeattleDataGuy For each person differently for sure. I am still pretty sharp and able to learn, however it goes a little bit slower as before. I know some very skilled developers, who were able to learn ton of new stuff after 40s, just it hurt them a lot :D However there is also a flip side to this coin Ben -> it is considerably easier to learn new stuff e.g. dbt (my case now) when you already are advanced in SQL and some scripting languages as a 20 years old noob :D

[@JL-pr4cw]

I'm 34 and have just switched over from being a MS SQL Developer to a Cloud Data Engineer. I have an opposite perspective as you, when I was 25, I had a good job and just wanted to have fun. Now, with having a family, I'm super motivated to learn, improve and provide for them.

[@alexanderpotts8425]

All the harsh realities you described are exactly why I love this job. I guess I'm starting to see why my old coworkers thought I was nuts when I said I really wanted to go hard into BI and modern data engineering... :) also, I'm glad you mentioned SWEs not caring about data. I had these same opinions on reddit when data contracts came up

[@SeattleDataGuy]

Yeah, some people love it others hate it! But data engineering always felt like a good fit for me!

[@deathbombs]

is BI powerBI? That's for data visualization right? What's difference between that and plotting on Excel?!

[@alexanderpotts8425]

@@deathbombs A lot of the workflows and terms you hear like ETL and data warehouses all started under the term "business intelligence". Long before the current modern landscape.

[@deathbombs]

@@alexanderpotts8425 ahh I see, thanks

[@overidon]

I love how there are two number '1' bullet points in the infographic cartoon at 5:15 too funny

[@sharu164]

I’m slowly starting to relate what you are talking about which is a good thing as I’m gaining so much of valuable information. Thank you so much, really appreciate what you are doing ❤

[@SeattleDataGuy]

Thanks for the comment! I am glad you found the video relatable.

[@boybi612]

Do some projects on Spark or Aws end to end pipeline that would be more interesting things in the eyes of recruiters. These days certs are something that everyone would have so choose to do something exclusively valuable.

[@antonkostov1691]

Brilliant as always. In my humble personal beginner's opinion . Due to Hana and cloud migration DE is getting lots of demand. I can say in Bulgaria, the market for DE jobs has grown radically last few months.

[@SeattleDataGuy]

wow thats really cool to hear that the demand for DEs are growing around the world!

[@adityagoyal6964]

Thank you for the video. I am a trainee data engineer and being a obsessive programmer I find it difficult to accept that I won't be a developer anymore but looking at this It seems like IT is much more than just writing codes in java and C++.

[@SeattleDataGuy]

Yes, more than likely if you go the strict DE route there will be less code but if you go more of a data platform engineer or software engineer, data, you can find lots of programming

[@rohit-pr]

Thank you for putting this out there for everyone. The problem that I have and I feel others also face as a DE is that the organization fails to understand the complexity there is with when working with data in general. Its not all green and breezy and a SELECT * does not do all the work for us. Pipelines maintainance and optimizing applications is a tedious work in general and someone(DE obviously) has to be on top of it. Removing DE is simply a recipe for hell and a ton of work for the DE who is going to join when everything falls apart.

[@frimatechnology]

thanks so much for this information it was delivered well with so much clearity.

[@flaviolanfranco]

This is a great resource for everyone, thanks for sharing it

[@SeattleDataGuy]

I am so glad you found this helpful!

[@zie3873]

When you said there is a lot of things to learn I almost cried because I spent the last 3 years acquiring a degree in Applied Mathematics and Computer Science which at this point I think is useless because it was too broad to a point where I do not know what to be a data engineer of. Even for entry level jobs the bar is so high you have to know a lot of things. I have an IBM Data Analyst certificate, AWS Cloud Practitioner and a couple of good projects but still can not even find an internship to at least put one foot through the door. At this point it's really stressful. 😢

[@tamalesolomon764]

Same here

[@irishuang5294]

Switch up your job hunting approach by meeting actual human beings. Reach out to people in the companies and or positions you are interested in and see what they say. You absolutely do not need to learn "everything", but at the same time, what Ben said is true, my first job was pretty much just Excel / automating Excel workload......I'm sure that's a very common starting point of many data engineers. You are never expected to know everything becuz even seasoned industry experts don't know everything :). Good luck out there!

[@wtfzalgo]

It's kinda hard to start as a data engineer. There are too many things to know about for there to exist a junior position in this field. I would suggest starting out in a parallel role, e.g. data analyst, data scientist, analytics engineer, and then laterally transition in the same company to data engineering.

[@bfkgod]

Why would they want to remove the very Engineers that are responsible for fixing the bottleneck? Do they think the bottleneck will just magically disappear? Here's an idea, you could hire people specifically to try and removing Data Engineering bottlenecks! Perhaps you could even call these people... Data Engineers.

[@SeattleDataGuy]

Its a wild world out there. I don't get it either.

[@irishuang5294]

They have a beautiful vision where data will always be self-evolving where business changes can be reflected on data automatically and downstream users like data scientists and analysts will magically understand changing data because they are that good. They also have a limited budget, so......hopefully at most just Fivetran/Stitch is enough to solve these ETL :p

[@cateclism316]

I was in software QA for nearly 15 years, and it was a trainwreck for me. I prefer working with databases, data files, and such. QA really didn't interest me, so I didn't study it like I should. I think data engineering is more to my liking, so I would be a lot more motivated.

[@CM_CM_]

I wanted to follow the CMU Intro to Database Systems KZbin series but the projects are written in C+ and I'm someone who's just learning python and can do some easy/medium stratascratch sql lol - I feel like I'm always going to be behind in learning DE compared to everyone else - Like you said is there's always something to learn, but it feels like the baseline for beginners is also always rising. I feel like I know nothing! Thanks for your insights, I really like that SQL poster, I just did a lot of googling haha

[@89kylestyle]

"Fix all these small problems that someone else produces"... Yep I'll pass

[@josepablolunasanchez1283]

I suspect that the concept of computer science will become a skill instead of an occupation in the future, just like using a typewriter was in a distant past.

[@DarshilParmar]

Another amazing video!

[@SeattleDataGuy]

Thank you!

[@dextermorgan6340]

Hi Ben, is topic of dimensionality reduction applicable to data engineering / future improvement of data engineering as well? I only know that it is for Machine learning. :o

[@SeattleDataGuy]

I feel like that tends to be more ML Engineer work. But sometimes there is cross over

[@MatthewReader]

Love going to Buc-ee's

[@SeattleDataGuy]

Glad you enjoyed the shirt!

[@jamestoneythethird699]

Hey Ben, although the gritty reality of being a DE doesn`t bother me that much, I`m still wondering whether or not it`s possible to find your first REMOTE job right after getting a Udemy/Coursera etc. certificate? Are DE jobs remote-friendly?🤔

[@egyptianqueen4007]

Most IT jobs are remote now. I'm an engineer and know a lot of Data Engineers that work remote. It's rare that someone in IT doesn't work remote now.

[@SeattleDataGuy]

I don't know if you can do it just with udemy and coursera. What is your background?

[@jamestoneythethird699]

@@SeattleDataGuy Well I`m a liberal arts major (lol). I`m taking IBM DE course on Coursera. Will this be enough to land my first job?

[@arunrajan1979]

Is it good to do a snowflake certification first or databricks data engineer associate certification first? Which one will you suggest

[@SeattleDataGuy]

I would say get a more general certificate first in DE

[@arunrajan1979]

@@SeattleDataGuy like Google data engineer certification ?

[@ABHITECHNOCRAT]

How is the work pressure in the job as compared to normal software engineering jobs?

[@SeattleDataGuy]

This will always depend on company and team. It can in some cases be stressful, but I'd say most tech jobs are

[@MomoDuality]

3:55 So what roles do you hypothesize data engineers will move/transition into after 10 yrs or so?

[@sirus312]

been in construction my whole life, data engineer seems like a very easy job compared to what I was doing and big salaries too. Might have to take this up as a side hustle on weekends. I use amazon for all my buying so AWS can't be that hard to get a hang of.

[@the_dented_NPC]

KEKW.. edit- OP is trolling 🤦

[@tomastruchly9484]

It's everything but not easy. I work in IT industry for 10 years and do data engineering like work last 3 and it's brutal (you need to have extremely high frustration limit & constantly willing to learn to handle some tasks). Good coins, but well deserved.

[@dandogamer]

Lol very naive in you thinking its easy

[@SeattleDataGuy]

That is my assumption but construction is difficult. As the son of a carpenter that grew up in Seattle...going onto the job site on a cold november morning and grabbing rebar or honestly holding anything while your fingers are soaked and cold suck.

[@MrJowisi]

Only 10 years of doing DE left??? Omg!

[@SeattleDataGuy]

More likely it'll just look really different in a decade. Tech is constantly changing!

[@antonkostov1691]

Thanks

[@SeattleDataGuy]

Thank you! you're tooo kind!

[@factspossible6670]

Hello man, you are doing great work! Can you please inform us all that if we strictly follow your medium article where you have given a roadmap to data engineering would we be able to land an entry level data engineer job/or able to get into this field professionally?? Your reply would help alot of us who are confused by plethora of information on internet!! 🙏

[@SeattleDataGuy]

Hey! too make this simpler I think Zach Wilsons recent tweet says it best twitter.com/EcZachly/status/1571587218833313793

[@painunstop5722]

You are awesome 😍

[@SeattleDataGuy]

Thank you so much 😀

[@psyche5184]

What is your take on snowflake vs databricks?

[@the_dented_NPC]

he already made a video

[@SeattleDataGuy]

I did already make a video on this . You can find it here! kzbin.info/www/bejne/jH3XomObmq2WZ5Y

[@dataArtists]

I think we may have a disconnect somewhere. Unstructured Data and Structured Query Language 🤷‍♂🤷‍♂🤷‍♂. Well at least we know two things for certain: 1. the technology will always be changing, 2. the data will always be changing. Perhaps DE will morph into CM (change management).😎

[@bettatheexplorer1480]

Hello

[@SeattleDataGuy]

Hello!

[@acostaisasmendi]

Hi

[@nevada_jinks]

Thanks again for another great video! Do you mind giving your feedback on this guy? kzbin.info/www/bejne/iJ6Vi2uqittmo6s He presents himself as kind of an "IT reality checker" lol and I find a lot of his videos compelling but as someone who's new to the space I'm not sure how credible his claims are... Also why do you think there's at least 10 years of life left for the data engineer role? How will companies deprecate the role?