Will Data Engineering Exist In 5 Years - Is Data Engineering A Good Career Choice?

[@SeattleDataGuy]

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

[@justinballas7299]

I think a lot of the concerns are coming from students and people that haven’t entered the working world yet. When you see how much of a mess the vast majority of companies data infrastructure is you’ll realize data engineering won’t be going away any time soon.

[@SeattleDataGuy]

THIIIISSSS! Yes...I should do a whole live just on this topic. I remember being a student, yeah it felt like everything was solved and everyone was doing ML...I promise most companies still don't do ML now. They just write articles about how they are doing it to stay relevant.

[@idkwhy77]

@@SeattleDataGuy demand in EU is high and AI doesn't have scaling capabilities or problem solving the human data engineer has

[@ibexy]

Absolutely 😄

[@parlor3115]

That's for existing projects. But for fresh ones where AI or AI-assisted engineers would be responsible of developing the entire infrastructure, development will be conducted having a strong forsight and following good engineering approaches.

[@elkinhuertas2137]

@@parlor3115This will take years. Implementing such an infrastructure is crazy expensive. Until it can be offered as a service up and down and across industries, we’re good 👌

[@WisdomofHal]

I’ve been a backend developer. On recent interviews, hiring managers really asked a lot of questions about developing data platforms. There were technologies I wasn’t familiar with, which sucked because I was expecting a typical backend developer interview. The industry changes so fast.

[@WisdomofHal]

@@rouenyu Yes, I noticed this as well. It seems, there are plenty roles for dedicated frontend developers and in those roles frontend, design skills, knowledge of APIs and browser side scripting with js are what’s expected and, honestly, that’s no surprise. Backend developers, as of late, are expected to know the full stack and be able to contribute on every layer (backend, architecture, cloud, storage, frontend, CI/CD, getting coffee and so on). Just a few years ago, I was able to look for a plain old backend developer role and I was set with the regular old java skills for backend. The industry evolves so fast and, it seems, you must with it while always be learning. My advice would be to set short term and long term goals: what I did was study frontend (react/vue) for 6 months until I can build full stack project. In that time I also studied cloud and architecture. The next 6 months I studied databases (Relational and Non Relational). I’m currently on miscellaneous technologies like Kafka, Flink, and streaming technologies). Luckily I got plenty of experience on my first job with CI/CD with Jenkins and such. Also, keep in mind, if you’re < 5 years in like myself, it just takes time to get familiar with ALL these technologies. As of late, there just seems to be more requirements on software engineers and an explosion of technologies. I don’t see the pace slowing down anytime soon. If not, then just ignore what I said haha. You’re probably extremely proficient in the technologies you’ve been using and could use that as leverage when you learn fullstack. You’ll crush it man!

[@_muzzles]

I think your point at 7:52 is super relevant. There is so much tech debt and the requirements keep changing. Maybe the world moves more towards point and click, but the core of data engineering will still exist and there wont be less of it.

[@splashoui3760]

I don’t like you should set click bait video titles which you do not need. I see your channel as high quality channel. And it’s being disappointing this kind of titles.

[@SeattleDataGuy]

I think this is fair. I am thinking of how I might change the thumbnail. Whats kind of annoying is that usually my CTR is like 6% and this video is 13%

[@roncrudup2110]

@@SeattleDataGuy I personally don’t think the title is that bad. You have to make a title that pops for the largest audience.

[@SeattleDataGuy]

@@roncrudup2110 Did you see the old thumbnail? I just updated it...before it was like "It's Over For Data Engineers" with Fire. I think the Plain Bagel did an analysis and found those types of thumbnails got an average of like 3-4% more CTR which is massive.

[@splashoui3760]

@@SeattleDataGuy I think important thing is to create your channel community with robust, real subscribers. Not the ones coming with click bait titles. That’s what I think. Still I appreciate your videos and thank you for your effort 👌🏼

[@ulysseschang129]

I clicked through when I saw the new thumbnail title. I think if I had seen the original "clickbait" title, I probably wouldn't have bothered checking it out.

[@gamingcomputers7485]

Thanks for being positive about it just to remind people that while ai and llms exist such as chatgpt aka Generative pre-trained transformer it does not mean it'll replace jobs of people they only one who would replace you with a robots an ai alike is your boss no one else ai is not bad and nor good its natural and same thing for computers you can use them for good and bad things and the same for mkney you can use it for good and bad things such as increasing price of certain products is thr bad things you can do with it while the good thing that you can do with it is donating a charity for some poor or homeless and people with diseases etc.

[@Mari_Selalu_Berbuat_Kebaikan]

Let's always do alot of good ❤️

[@SeattleDataGuy]

yes!

[@sickair45]

now in 2024 would you recommend going to school for a masters in data science/analyst or bachelor's in computer science?

[@SeattleDataGuy]

Hmm, I have mixed feelings about masters in data science and computer science because i assume its better to just do..but i don't have a masters so maybe someone else would find it useful

[@collinsnnadozie5092]

I'm still wondering why a programmer will create an AI to take away our jobs

[@SeattleDataGuy]

Because the business who creates that tech will become very rich 😅

[@collinsnnadozie5092]

@@SeattleDataGuy indeed, but at the expense of taking away people's career 😭

[@Zuranthus]

@@collinsnnadozie5092 google "luddite"

[@collinsnnadozie5092]

@@Zuranthus Are you the developer of AI Chat GPT ?

[@Zuranthus]

@@collinsnnadozie5092 no, i'm just your average day rational individual that understands advancements will be made and you will find some other job to do. or do you propose we all go back to tilling the land and raising barns? cause you know, that would make sure there's plenty of work to go around for everyone

[@TomT-ds9vn]

Keep getting hyped about the lower pay scales coming due to less technical skills necessary to do the job and job loss.

[@shardulkatareaz9984]

Hi @SettaleDataGuy, I have been following your channel for quite some time now. Your channel is great! and really breaks down what Data Engineering looks like on a day to day basis. Currently, I am working on a Data-engineering project. I need to pull data and compile it in excel for Reporting The data comes from several databases - Athena, Oracle and SQL Database, and some from Excel. This data becomes available at different points in time as each is for a different country. Currently this process is done manually and takes about 1-2 days. I am thinking of building a datapipeline using Apache Airflow. Do you think this is a good solution? Any of guidance that you can provide will be really appreciated or any material/video that you have created in the past to reference. Thank you! And please keep doing what are doing

[@arbaazm3761]

Yes you should

[@hameethaabdul]

I am basically a Java developer I wanted to work as a data engineer I am getting a offer from different projects is this good to move as a data engineer from Java developer? Can you please give me some suggestions on this ! What future scope it holds and things ?

[@Nick-du9ss]

What do you think about data analyst and business analyst job role regarding next 5 to 10 years

[@MeLLiFLuOuSViDeOs]

The question is not that will humans will be required or not... The question is will they pay you 300K USD per year for something that the AI does completely and you are just being specific and looking over it? Humans will have the Job but those making 300k will be reduced to 100K

[@WisdomofHal]

Hmm. Good point, but have you thought of other alternatives. If AI can do the work, it’ll be more productive and produce more value (more money). Why reduce the salary of the human if more value is being produced? Like you said, human still needs to monitor: if no human is willing to monitor then what’s the use of AI that can have many mistakes? Therefore human can, still, demand very good pay. It will still take deep knowledge to know what to look for, mistakes, corrections, additions and so on. We reduce human labor. Also, human labor has always been added to any work ticket, the cost of that labor. If we are making labor cheap, this can also make other aspects of life cheap while producing more output. As computation power and storage becomes less expensive. I believe the overall goal is to give people more time to do other things, things we actually enjoy, we spend so much time at work. Wouldn’t you want to be with family, reading a book, skiing, or playing music? Basically, it’s why we wanted slaves in the first place, only exchanged for AI. Now we’re talking ethics… it’s a complicated chat, my friend. The idea of AI is still young; imagine a world with AI powered robots who can build without rest, who can clean without breaks, who can drive you from A to B, who can give you all the answers, cool your meal, read your kid a book. Would the cost of new construction be more or less expensive and who’s incurring the cost if we humans are not working? If robots can construct non-stop, where will all the material come from. Say construction becomes so cheap, countries decide to build homes for everyone. Is that feasible? Now, we’re talking policy. If we go another route, your route, and even highly paid jobs are now paying nickels and dimes while less paid jobs have already been automated to AI/Robots and we’re still expected to pay these enormous bills then how do we pay the bills? Or will government step in and develop more policies. It’s a very sticky situation. Now, what is the value of money? If money is no longer the god, then what becomes god for sheep? AI, a new god, or do people go back to spend more time with their one true God? Just something to think about. In my opinion, I don’t see salaries going down in our lifetime.

[@MeLLiFLuOuSViDeOs]

@@WisdomofHal That cannot work with capitalist system... And is polar opposite of current capitalist system

[@WisdomofHal]

@@MeLLiFLuOuSViDeOs That’s my point exactly 🙂 Don’t you think AI will relate a world like this? A world where we do less, if anything at all? Where monetary value is significantly less important or not important at all. The assumption is AI will automate and takeover most, if not all, jobs. Wouldn’t the next logical step be that government will morph into something completely different and have to establish more and more policies? I don’t know. There seems to be multiple ways this whole AI thing can go. I guess, we’ll just have to wait and see. Why would I have a human build my deck when government begins to produce machines that can do it perfectly in 5% of the time it takes a human to. What are the ramifications of government becoming more powerful? Why is it that, in capitalist societies, the idea of socialism and socialist policies are appealing to more people, how is that leaders that promote these policies in the most capitalist societies are being elected as leaders and what’s the effect on society? Why have countries in the West promoted, and some implementing, less working days (4 day weeks, 10 hour work days) and will 10 hrs a day slowly decrease back to 8? Officially reducing productivity? Or are we more productive because of AI and automation?

[@dekippiesip]

​@WisomofHal your market salary isn't really determined by your added value, but by the factors of supply & demand on the labor market. It is profitable to hire you if and only if your market salary is lower than your added value, as with all products. Added value is, therefore, more like a salary ceiling. It can be the case that new technology enables you to be more productive and simultaneously reduces the skill floor of anyone wanting to do your job. In that case your market value actually goes down even though your added value goes up. We associate added value and productivity with higher salaries, because often if a machine tips over 3 out of 4 workers, the remaining 1/4 has to do less manual work and needs to work at a higher abstraction level. THAT translates into more salary, not the added value in itself.

[@kunalmishra8326]

Very nicely explained, I am working as data engineer for around 7 years? Do i explore more into MLOPS career now, or further explore data engineering ?

[@gautam_505]

Hii brother I'm interested in data engineering will I get job at this role at first place and what do they expect in skill sets?? Currently I'm doing my intern where I'm learning to build ML models

[@adarshgupta9776]

​@@gautam_505hi gautam wanted to talk to you could u provide any social handles of yours?

[@khalidmuhammed3058]

After 1 year you still think data engineering will be relevant in the future or AI just gonna handle it?

[@mhn17]

I'm a freshly graduated, however I'm stuck finding my first DE job. I've tried to apply for many possible SQL/Python-related jobs, but it seems like no one is going to reply to my applications. The requirements for a freshly graduated are seemingly a lot. I've applied for many intern positions also (that means I'm willing to work at least 2 jobs, 1 for DE and 1 for making a living while gaining DE experiences). Furthermore, the market for which I'm currently seeking my first job is in Vietnam, which made I think that my English would be considered to be a fairly strong advantage. But, apparently, it's not. Do you have any recommendation for personal projects to put on my CV? I'm looking to gain more knowledge on building data pipelines, ETL, etc. I'm advanced in SQL, intermediate in Python and other languages such as Java, C++, web languages. I'm also familiar with data modelling, NoSQL, data visualizing (Power BI) and server-side data processing (NodeJS and PHP). How could I utilize the mentioned skills and tools? I do really need help now!

[@WisdomofHal]

I don’t know the market in Vietnamese, but I couldn’t imagine the competition being any less fierce than the competition in the US. I asked my mentor this question, his advice was to: 1. BUILD projects 2. Contribute to open source 3. Become more visible (blogging about popular technologies that many people use and will need help tutorial help with, blog about niche technologies you know that don’t have much resources, or blog about your own product and challenges you’ve faced, share those challenges because others might run into them). I wish you all the best, my friend!

[@sherwing9054]

There's too much supply of python DEs, what companies lack are framework/enterprise skilled DEs. Like example, they want to find people that excel in Talend, DataStage, Teradata, Informatica, Snowflake, to name a few. To land a job, use actual tools, not just python. For example, I only knew mysql back then, but landed an SSIS/SSAS/SSRS job. And with that, I landed an informatica and snowflake job. Now I get frequent calls for tools like DataStage and Talend which I don't use, but since I already have exp in other frameworks, they got no choice.

[@datpham9526]

Hu đọc được cmt của a em cũng thấy hơi hoang mang vì tình trạng tìm việc của DE fresher ở việt nam , a cho em chút lời khuyên được không ạ

[@rafayowais480]

Data Engineer vs Backend Developer in future?

[@idkwhy77]

data engineer.

[@WisdomofHal]

Backend developer is slowly becoming data engineer. Or is it other way ‘round?

[@coder7078]

@@idkwhy77why ? Any specific reason ?

[@SaveThatMoney411]

Please make a video on why you left Facebook. 😢

[@SeattleDataGuy]

Have you seen this video? kzbin.info/www/bejne/jKXRgqljeLOShKc

[@WisdomofHal]

Love your name. 💚

[@SaveThatMoney411]

@@SeattleDataGuy Thank you. You don’t discuss why you left in that video though…

[@SaveThatMoney411]

@@WisdomofHal Thanks. Someone called me aggressively NPC.

[@WisdomofHal]

@@SaveThatMoney411 Was there any reasoning?

[@Drakvod]

If many of the things that a data engineer was in charge of are now becoming a commodity, there might still be work, but paid significantly less.

[@SeattleDataGuy]

I don't know if that's true. In many cases, as we automate work, it increased productivity which increased salaries. If AI acts as a tool, then I imagine it will be the same. Of course the ICs will see a decent jump in their pay but the biggest winner is always the corporations.

[@quangvinhle7912]

Will be data engineering ok in the future? I want to transition from a backend developer to a data engineer.

[@SeattleDataGuy]

I think it'll change, I like what joe reis sad recently in his blog where he talked about DEs having to be more aware of data science and software so we can provide data for AI, data science, analytics, etc

[@Dheeraj7998]

Hi Ben, I'm a entry level into IT industry. Im more interested towards the field of data. Im confused to go with machine learning or data engineering.. please suggest . Thanks.

[@gavinkalaher7314]

entry level = data analyst. Microsoft Certified: Power BI Data Analyst Associate is the best course.

[@Dheeraj7998]

@@gavinkalaher7314 thanks

[@sharad3877]

your linkedIN?

[@SeattleDataGuy]

I am! You can look up Seattle Data Guy!

[@TomT-ds9vn]

It will shrink data engineering teams.

[@mahmoudelleithy6908]

Omg. All this time and didn't answer the question !! 👎👎

[@Jan_Be]

👍🏻

[@SeattleDataGuy]

😀