The Future Of Data Engineering | Trying To Predict Where Data Engineering Is Going in 2023

[@SeattleDataGuy]

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

[@jbejaran]

Great video. I'm a data architect specializing in data modeling and data warehouse design. My sincere hope is that data quality does begin to gain importance again. I've seen a sharp decline in organizations caring about data models at all. It's become much more "see requirement - meet requirement" which inevitably leads to a big heap of data marts that duplicate logic, and where that logic eventually diverges and you get two business units asking the same question and getting different answers. Whenever I've pushed back about the importance of having a sound data model, I've tended to get hand-wavy responses about how "we need to be agile." If you take just a small amount of time to understand the enterprise's entities and their granularities, you can build really good data models in a very agile way without sacrificing that data integrity, and future-proofing. Good data models aren't (any more) just about performance. They're about data quality and integrity. I really hope organizations rediscover this soon.

[@jithendrayenugula7137]

Amazing video as always - Ben ;) For like coming 5-7 years, I think companies require data engineers for correcting the existing data architecture more rather than starting from scratch. At the consulting firm that I'm working at - many clients are coming & asking to adjust/modify/correct backend / data architecture that they are currently having. They dont want to do major changes from the software side but they want the database to be optimized. Many small / new startups are focusing more on the frontend/end product and after 3-5 years they are realizing that the data quality sucks and also the data architecture sucks. I personally feel companies should hire enough data experts to take optimal decisions during the initial periof itself instead of thinking about this in future when something start to suck. opinions?

[@SeattleDataGuy]

Glad you're enjoying the content! Also thanks for sharing, its valuable to everyone watching!!

[@mochaspyce79]

This was very helpful for me. I am currently in a data analyst role and would like to move into a data engineer role but was kind of unsure if I should or where to start. 🙌🏾

[@robcaseygolf3400]

I just started in a Data Science Bootcamp and much of your channel has helped guide me in the direction I'm most interested in, and away from what I thought I was interested in. So thank you. Anyway, this is great insight and definitely not in the kool-aid we're being given at the moment. I look forward to seeing how Greg's carved comment and this content pans out. 👍

[@SeattleDataGuy]

Glad it helped! Yes, it'll be interesting to see what the future holds.

[@datag1199]

Thanks for sharing your perspective. As a data engineer, I've been thinking about how our profession will be impacted with some much going on in AI space, ChatGPT, etc. So true on the "garbage in, garbage out" example you brought up. Completely agree and see Data Quality becoming more of a requirement vs "need be basis" from our business partners. Exciting times we are in. All the best to you and all viewers of this channel

[@mustafakara7739]

Your videos are interesting. Can you also make a video about GIS or geospatial tools like GeoPandas, shapely?

[@herbertsousa7516]

With all the low budget and layoffs in different companies I believe some investments for instance in data quality will become even more restrict than already are. It is maybe the most important part after you have a stable data infrastructure and you are already feeding different company areas but I still think it will be a paradigm to break.

[@SeattleDataGuy]

Yeah, I feel for data quality companies..its a hard sell

[@SeattleDataGuy]

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/

[@alkaseram]

Great videos, Do you think learning hadoop eco system is still relevant nowadays in data engineering?

[@dang2395]

I just got an internship doing data engineering as an undergraduate, but in the long run I want to break into Data Science and Machine Learning. Currently I have a good general understanding of those topics, an even better understanding than I think I do of data engineering, but it seems like a lot of the data science roles are looking for people with graduate degrees. How do you think someone could move from data engineering into a more machine learning, mathematical position over the first few years of their career?

[@divyanshbhojane7977]

My company uses snowflake, so I just got snowpro core certified. Is it a good idea to move forward in that direction ?

[@SeattleDataGuy]

This is always hard to say. In 5 years, its not unlikely that we'll all be talking about some other tech. But that's the tech world. Constant learning.

[@PhilipJReed-db3zc]

Great survey of the landscape. In coming years, do you think data engineering and software engineering will grow closer, stay the same in relation to each other, or get farther apart?

[@technicalsyntax]

Amazing Bro!

[@SeattleDataGuy]

Thanks 🔥

[@Emanuel-yb3qk]

Do you think is important or mandatory work as backend/data administrator at least for a while before looking for the opportunity to works as data enginer? I read in some forums that tech companies do not use to hire fresher data engineer, they just hire experimented people from areas such previous mentioned or, people with two or more laboral data engineer experience/:…. I’d like to know your opinion please. 0:00

[@SeattleDataGuy]

A lot of people often start as a data analyst or SWE and then move into DE. So yeah usually you do need to have 1-2 years of experience in other fields to break in

[@tommynelson4795]

Do you think on average analytics engineers are better shielded from layoffs than data science jobs?

[@SeattleDataGuy]

I don't think anyone is safe this time around. 😭

[@sxxon751]

Nope. However boring jobs will most likely be more safe then fancy. Eg maintaining 10 year old spreadsheet etl pipelines in megacorp will continue but HOTPOtATO inc that started 2 months ago only using cloud services might not be so good.

[@bhanuteja4066]

Should data engineers need to master some analytics engineer skills like ERD, data modeling?

[@SeattleDataGuy]

Great question. Analytics engineer is technically new. Data modeling and ERD has always been part of Data Engineering work. Here is a video where I discussed some of the differences between these roles kzbin.info/www/bejne/jXSVaY1-fN56Y68

[@ChristianRRL]

Are there any conferences/conventions that are oriented towards data engineers?

[@opcon3155]

Will coding still be required for data engineering? Because I’m in it for the code 😅

[@baekimin21]

Oh i am the same..

[@willi1978]

future of data engineering is probably still excel

[@SeattleDataGuy]

cries....

[@sr-xd8jb]

Seeing the current layoffs, does getting MBA from India or some cheaper country make sense?

[@SeattleDataGuy]

Huh, assuming things eventually do swing back I imagine the US will likely..be fine...otherwise..we have far bigger problems

[@ravikrishna4839]

Hi bro my graduation done in 2021 i joined in one institute course as Java full stack I need to become data engineer is it better or not but i want do job as soon as possible

[@Vishal-pm3vv]

Don't go for. Data engineer in India it's down market very soon u will be laid off

[@gregap8282]

I give chatGPT and other AI's 5 to 10 years to surpass midlevel software programmers, carve this in stone, in less than a decade there's going to be a huge crisis related to software programming jobs all around the world.

[@SeattleDataGuy]

Alright! this comment is carved in some server somewhere and replicated several times over. All hail Roko's basilisk

[@DevAtul]

Tech changes exponentially. May be programmers programming what they programming now will be irrelevant. May be they will program more chatGPTs. 😁

[@gregap8282]

@@DevAtul I see the population of developers all around the world dwindling down to a small group that focuses in AI administration and maintenance. If a billionaire knows that investing in AI will cut jobs aka make more profit for him and the company, he will invest in that direction. For sure, in 5 years there's going to be a huge crisis.

[@gregap8282]

@@SeattleDataGuy all I have to say: Butlerian Jihad

[@DatabaseDiary]

if your saying that, you have not played with ChatGPT enough, the quality it produce is out dated, second you actually need to know software development to understand the code, otherwise the codes it produce is already in stackoverflow, so copying and pasting code from stack over flow won't work, you have to know what you doing. just because you'r software developer and understand the code it generates does not mean everyone understand it too. the only thing ChatGPT will do increase current software developers demand, as in 10 years university graduate won't be able to produce any working code as their homework is done by ChatGPT, this will be crisis as Developed countries need to import more programmer from under developed countries.