Everyone's Data Infrastructure Is A Mess - The Truth About Working As A Data Engineer

[@SeattleDataGuy]

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

[@GoWmaster27]

Based on the thumbnail, I really expected this to be a 3 second video where you just record yourself saying “yes.”

[@SeattleDataGuy]

Hahaha - yes

[@urip_zukoharjo]

3 secs is too long to spit yes, 0.01 sec is all you need to say "Уe"

[@richardduncan3403]

I have noticed that automation needs quite a bit of manual maintenance

[@SeattleDataGuy]

hahaha...if you've ever had to backfill a table....

[@Yavin4]

There are other layers to this problem. E.g. regulatory compliance. Who can/cannot access the data? What and where can you access the data? Think GDPR. Other factors include external vs internal data. Is there a cost to accessing/collecting the data? Most companies are not even close to having good Data Governance fundamentals, and many of them may never meet a high standard given the constant turnover. The Data Engineer role will evolve into a greater one of overall Data Governance.

[@edwardmitchell6581]

I find this unlikely. Data engineers tend to be highly technical order takers. It’s in there interest to use technologies that lead to high salaries. It’s not in their interest to have data quality 4 quarters from now. On top of that, data governance is a topic that never goes below C Suite at most companies. I’ve seen job requirements follow the trend you predict, but I think that’s just about technical knowledge of metadata rather than business skills.

[@Yavin4]

@@edwardmitchell6581 You are describing the current state. I am talking about future state.

[@SuperLOLABC]

So with companies data governance being such a mess, would you say that the field of data engineering & governance still has a future for atleast a decade? Or will it all be automated since automation seems a huge part of Data Engineering already?

[@edwardmitchell6581]

How can you automate data strategy or data management?

[@SuperLOLABC]

@@edwardmitchell6581 Today in a data engineering team of 10, about 1-2 people take care of data strategy and data management. The remaining 8 build and maintain the solution. Their main job is to automate the engineering solution. If data engineering can be automated sufficiently then the total amount of DEs required will go down.

[@SeattleDataGuy]

There is plenty of work to do. I can't speak in decades, but 5 years, yeah probably

[@sigmapi1989]

Ha soooo True! Everywhere I've worked its been a mess!

[@SeattleDataGuy]

its just always a fight to try to bring it to some level of sanity

[@ceejay1353]

For those who want to get into consulting, assuming you're starting from 0 exerpince, how many years of experince would you say is good before you can reasonable make a living off of consulting?

[@KshitijPatil1]

Consulting is entertained mainly with the logic that someone with MORE experience than them is going to help solve an unsolvable problem. So if you have 0 experience, what in your opinion is it that you would be even offering to them?

[@ceejay1353]

@@KshitijPatil1 I think k you missunderstood my question, I'm asking howany years is good to start consulting in general

[@KshitijPatil1]

@@ceejay1353 My bad. So you're asking how many years does it take to make a living off of consulting gigs, should you leave your current job, right?

[@ceejay1353]

@@KshitijPatil1 Yeah!

[@KshitijPatil1]

​@@ceejay1353 Got it. So my assumtions about getting into these types of career paths is that you already have your first 2-3 clients when you start. This means the people you've worked with, trust and respect your contribution are happpy to commit their company's dollars on a weekly/monthly basis. This helps you to a) anchor your price and b) provide references for your potential clients to get social proof from. The reason point a) is important is so that you know how much is the max you can earn per month, and deduce the number of clients you need to juggle. Point b) helps you to go on an aggressive client acquition excercise, because till you get your schedule packed, there's no financial upside to this excercise.

[@sirus312]

Palantir seems to be the only solution

[@SeattleDataGuy]

we'll see! From a stock perspective I am still waiting to break even although i bought at like $18 so it was there a while back

[@kdgolden8463]

U look like drake n that’s y I clicked n YK Which drake.

[@SeattleDataGuy]

i don't know which drake hahaha