Why Is Data Engineering So HOT Right Now?

[@KenJee_ds]

Would you consider a career in data engineering? Let me know in the comments below! Thanks for watching everyone! For anyone wondering... Yes, I did make this entire video just so I could use the zoolander clip at the beginning.

[@TinaHuang1]

YES most companies think they need data scientists but what they actually need are data engineers. Awesome video!

[@KenJee_ds]

Thanks Tina!

[@piyushzope10]

Yes, true...

[@norbert4571]

Last year I did a Data Analyst programme and during last winter I started looking for Data Analyst/Data Scientist roles. I ended up getting a Data Engineer role with a local company and for a few months I thought I settled for less. But now, a year into the job, I realise how much better a Data Scientist I can be because of the Data Engineering skills that I have acquired here. I was myself blinded by the hype around Data Science, not knowing that without serious Data Engineering skills (to extract, transform/clean, load and most importantly, to understand the journey of data), all the effort of fine tuning ML models could be nullified by the poor quality of data.

[@KenJee_ds]

Thanks for sharing your experience. This is awesome!

[@SeattleDataGuy]

I was just enjoying the domain pun and then I heard a shoutout for my channel! Thank you so much for linking me! And great video as always.

[@KenJee_ds]

Thanks for making great educational content on the topic!

[@LeonLokk]

I like the analogy that the book Practical MLOps uses. If your home has no access to clean water, then even doing the most basic tasks becomes much more difficult and highly inefficient.

[@KenJee_ds]

Really great analogy!

[@SundasKhalid]

Ken, that's a solid point. I agree that companies are finally realizing that they have to hire data engineer before investing in a data scientist. Also, thanks for the shoutout. That was a nice surprise. I appreciate it!

[@KenJee_ds]

Thanks for putting an awesome resource on your experience out there!!

[@dejavu06007]

I worked in the oil and gas industry and what you said is very true. It is only realized after big AI/ML teams are built that: we don’t really have a big data to work with, or our data are not structured/ready, or our problems are not suitable to be solved by ML.

[@KenJee_ds]

💪

[@sidkapoor9085]

Those pizza making b-roll cracked me tf up xD

[@KenJee_ds]

I had some fun with it haha

[@benxneo]

Thanks for featuring my article Ken Jee!

[@KenJee_ds]

Thank you for writing it! Really good analysis!

[@piyushzope10]

Yes, I am considering data engineering as a carrer as it involves lots of learning about data handling which leads to get into other data science role as well... Chronology

[@KenJee_ds]

Yes!

[@learnwithanirudh4891]

Solid job, Ken! As a Data Scientist myself, I definitely agree with all the points you have mentioned in this video!

[@KenJee_ds]

Thanks for watching Anirudh!

[@jimcrowjoe451]

Thanks Ken. As a Data Scientist AND Engineer I agree with your video 100%. Most of my colleagues and myself wear both hats.

[@KenJee_ds]

Awesome! Thanks for sharing your experience!

[@jbsc97]

I studied data science to become a data scientist and I ended up becoming a data engineer, but I see a great long term future in it 👍

[@KenJee_ds]

💪

[@Avant402]

Yes, I would definitely go for a career in Data engineering. You get to see how pipelines are build, you get to understand the flow of data through those pipelines.

[@KenJee_ds]

💪

[@AssadJawaid]

I'd be interested in learning what they do and do some introductory course, but for the most part you've made pursuing Data Science a priority for me.

[@KenJee_ds]

Awesome!

[@DataProfessor]

Nice pizza b-roll, yummy 🍕. I also think that existing data scientists will significantly benefit from a little data engineering skills, perhaps the field may eventually move towards becoming a full stack data scientist?

[@KenJee_ds]

So true! Thanks for watching data prof! Congrats again on the new job!

[@DataProfessor]

@@KenJee_ds Thanks again for the kind regards 😆

[@letechnicaljames]

Also Ken, I was wondering your thoughts on algorithm questions for Data Engineers vs. Data Scientists. Maybe a cool idea is to go over different interview questions for the different roles?

[@KenJee_ds]

That is a really good idea! I'm actually not too sure about what the differences are. Will ask some Data Engineer friends!

[@letechnicaljames]

@@KenJee_ds Right on. May also help that one guy who has to prepare for both a Data Engineer and a Data Scientist interview within one week haha

[@_polyvoid]

Excellent video, cannot agree more. I think experience in a DE role will help you become more effective as a DS later on - you'll gain more knowledge about data infra and tooling that many DS lack.

[@KenJee_ds]

Thanks for watching Bart!

[@LukeBarousse]

2:26 TFW Ken Jee calls out the type of company you work for that has failed to invest in data engineering 😔😰😭... thanks ken....thanks Based on the pains I've seen, I'm SUPER optimistic about Data Engineering and think it's going to go "TO THE MOON" these next few years!!

[@KenJee_ds]

Papaya Paradise LLC would never fail to invest in data engineering.

[@LukeBarousse]

@@KenJee_ds You apparently haven't met my boss 😳#akward

[@YourMakingMeNervous]

Based on my experience on the job market, Data Engineering is competitive right now, but it's easier to find a place that is willing to train you on it compared to Data Science. I was an analyst for almost 3 years doing a lot of programming and just got a job as a Data Engineer. I was upfront about my relative lack of experience but showed them all the ways I have worked on bridging that gap, and they are willing to train me. I don't think I could find a company willing to do the same for Data Scientist, those usually ask for Masters or PhD at this point.

[@KenJee_ds]

Thanks for sharing your experience!!

[@rud1208]

That’s because you can learn data engineering on the go. It basically just learning your job tech stack and reading a few books on DWH and you will be fine. In Data Science you need an extensive knowledge of statistics, linear algebra and optimization that only school can provide.

[@YourMakingMeNervous]

@@rud1208 I don't think DE is quite as simple as you're making it out to be, the Kimball DWH series totals several thousand pages, and that's just one of the recommended series to read. Overall though, I do agree that you can learn it on the job. I think the real difference is that there are comprehensive resources for DE whereas there isn't any "Data Science Handbook" that I've seen

[@rud1208]

@@YourMakingMeNervous I was oversimplifying but you got the point

[@caioreis350]

@@YourMakingMeNervous what are the other recommended series besides Kimball?

[@kelline1979]

Great content, thanks. i just join a company as a data analyst and going through their general data bootcamp. halfway thru, there is a project and i needed to learn cloud and docker etc. i believe ML-Ops and Dev-Ops are also belonging to data engineering. Could you share more on them? Thanks

[@KenJee_ds]

I actually just made a video on ML Ops! Could be worth a watch!

[@SydneyRox1]

The pizza b-roll is too funny for no reason

[@KenJee_ds]

glad you enjoyed it!

[@ananyochatterjee2445]

coming for my interest in data science and curiosity about data engineering and recognising that ooni oven from a joshua weissman video in a tech channel makes me realise i watch too much youtube for which i am really proud of myself

[@KenJee_ds]

As long as it is in the name of learning!! I love Joshua Weissman's videos!

[@0x007A]

A career as a data engineer likely appeals to people, like myself, who enjoy building stuff that endures over time.

[@KenJee_ds]

A huge benefit of this track!

[@sams8502]

When I look at a lot of analysts jobs, there's a huge skills gap in what companies want now. They're either looking for an old school Excel guy w/ 10 years experience or an entry level Hadoop/Spark genius with some Tableau.

[@KenJee_ds]

Completely agree. I think this is a huge problem.

[@cosmingutu8256]

Hey Ken Jee, thanks for the great insights! Right now I'm working on a project that involves integrating data from multiple sources for other people to use, which is very similar to data engineering, but my goal is to be on the data analysis side and I'm hoping that in the future projects around that will arise. With that being said do you thing that doing this kind of work will pay off in the future, or should I be doing data analysis and data science instead? P.S. I have less than 6 months experience in the field so far. Thanks!

[@KenJee_ds]

I absolutely think it will pay off in the future. The natural extension to what you're doing now is the analysis. If you can understand how the data got to where it is, you are better informed as a data analyst or data scientist. I think this is a great start!

[@cosmingutu8256]

@@KenJee_ds Thank you for your answer, I think it will also help me to have the right mindset when starting working with data and understanding it's flaws and limitations. And to add to your point I had the opportunity to see how two fairly big companies store their data, and it's a big hassle indeed to investigate where the data is coming from, if it's the right data, why is what supposed to be the same data in more than one place and it doesn't 100% match, etc. And while working on it I was having the same thoughts, that the people who build these data lakes didn't think much on how it will be used, so yeah, someone who understands how the analysis is done should be part of the team building the data infrastructure.

[@svallam]

Hi Ken, Thank you for making great content on Data science and engineering in general. Do you have recommendation for Stocks or Options related Data science material?

[@KenJee_ds]

Thanks for watching! Honestly, this is outside of my depth. I generally recommend Jake Amaral for those who are interested in the quantitative side of finance

[@reno9549]

Hey! I'm considering data science as a career, and I was curious on your input as to what the best course of action would be, specifically regarding college. Should I consider majoring in a more general field like computer science, or would it be more beneficial to major in data science?

[@KenJee_ds]

I think it really depends on the program. As of right now, I generally recommend studying computer science & statistics double major over a data science program. I find that in the short term, a lot of universities have data science programs that are just watered down. I don't think this will be the case for a long time though. On the other hand, if a Uni is doing a good job and offering rigorous classes in their ds program I think it woudl definitely be worth it!

[@hiroxh6366]

Love your video. Now I'm hungry!

[@KenJee_ds]

Haha Thanks for watching!

[@mariopaul6505]

Solid video Ken! Should we expect more videos regarding Data Engineering? 👀

[@KenJee_ds]

Thanks Mario! Maybe!

[@dolusolana2127]

If we think of Data Science as a field of study rather than a role, then Data Engineers are technically data scientists.

[@fortuneitiola]

Great thoughts here, though technically engineers are not scientists. However, in today's world, words overlap in meanings. Again technically, engineer applies what scientist "makes" but it seems in the data industry/domain, the reverse is the case as data scientists applies what data engineers "make".

[@KenJee_ds]

Very good point!

[@chacmool2581]

@@fortuneitiola "Science is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the world. " If engineers are not scientists, who is?

[@fortuneitiola]

@@chacmool2581 Great and comprehensive definition here. I'm not arguing though, we just putting out points for all of us to learn. All I'm saying is even if engineers are scientists, not all scientists are engineers.

[@kepstein8888]

Why do I suddenly have a craving for burnt pizza?

[@KenJee_ds]

Lol I promise I don't usually burn it!

[@Ibraheem_ElAnsari]

I dunno if I tried to read too much into it or not but I think each Pizza shot was an analogy lol .. I absolutely love that ! Loved the video and the Pizza making skills 🌻 My long term dream is to open a restaurant after becoming a very successful professional 🍽🍕

[@KenJee_ds]

Yes haha :) The science is basically what goes on when we put it in the oven. A lot to do before then!

[@faizalshebli9520]

I'd stick to Data Scientist role coz I love to analyze better but I work closely with my Data Engineers to get the right data source for me to analyze further.

[@KenJee_ds]

Nice!

[@StefanMalic]

Idk, I'm super optimistic about data science in general. This is one job that has the potential to change the world for the better in ways we've never seen before.

[@KenJee_ds]

I'm very optimistic as well!

[@StefanMalic]

@@KenJee_ds aw man, it's so nice that you read and reply to comments! It feels like my opinion is being heard, even though my opinion means very little in the grand scheme of things. Sorry for sounding weird lol

[@KenJee_ds]

@@StefanMalic :)

[@decodewithabhishek]

This video is just making hungry 😅watching that pizza 🍕

[@KenJee_ds]

😆

[@timothyhong8962]

Is this pizza making tutorial or data engineering video? :D

[@KenJee_ds]

Why can't it be both :)

[@demohub]

Thanks for sharing

[@KenJee_ds]

Thanks for watching!

[@sebisamuel2605]

This video definitely triggered my jealousy by looking at your pizza making flex 🤡🤡 Well as they say, many people just go on for the sophisticated stuff without even clearing the base. I think same is the scenario with data. Many companies joined the data science hype but didn't pay attention to their data or it's architecture. So now as a damage control move, they're back to sorting out their basics and now we need data engineers. I was waiting for this topic to be touched and finally you did!! 🤠🤠🐢🐢🐢

[@KenJee_ds]

If you ever visit Hawaii, I'll make ya some!

[@trentonpearson7353]

What programs would you use to be an engineer? I am currently a data analytics grad student using tableau, python, SQL and R.

[@KenJee_ds]

A lot of SQL!

[@letechnicaljames]

Just realized why you chose to include baking pizza to compare how HOT Data Engineering is.

[@KenJee_ds]

💪

[@da_ta]

Hi Ken are you saying you recommend Data engineering than data science ?

[@KenJee_ds]

I am not saying that. At 5:24 I touch exactly on this. If you have the aptitudes for it and would enjoy the work, sure it could be worth it. If you really enjoy the data science work, you should 100% stick to that path. I hope this helps!

[@da_ta]

@@KenJee_ds thanks 👌

[@daAdiYogi]

I feel like ordering pizaa, after watching this video but anyway this has a good information why the DE is gaining so much popularity

[@KenJee_ds]

Lol glad you liked the b-roll

[@GuyThompsonFWTX]

I’m fairly new to data engineering, but I’m beginning to notice that there are some segments in Data Engineering approaches to solutions - custom code, implementation of tools, and a hybrid approach each with their own pros and cons. Custom code can be a good solution when there isn’t an out-of-the-box solution, the job is small, but this can take time. Prebuilt tools can get problems solved quickly, but may lack some functionality/features and can be expensive. The hybrid approach seems to be the most useful, but it takes a lot of knowledge to know how, where, and when to make these decisions for a business solution.

[@KenJee_ds]

Completely agree!

[@hiroxh6366]

What college education would you need for these roles? Like maths?

[@KenJee_ds]

Usually information systems or cs would make a lot of sense!

[@chacmool2581]

Moral of the story: you gotta harvest wheat, mill it and turn it into flour and make dough before and so that you can make pizzas. No wheat, no flour, no dough, no pizza.

[@KenJee_ds]

Amazing Analogy!!

[@yrysf777]

I wonder what will be the future, will data engineering fade away or boom ?

[@KenJee_ds]

I expect it will boom. There is more data every day than we know what to do with

[@pankajjoshi2547]

Any courses u suggest in data engineering?

[@KenJee_ds]

I will try to share some resources! I think there are a few decent ones on Udemy

[@davidporterrealestate]

so true

[@KenJee_ds]

💪

[@jordaneisenman4894]

Data science will always be more popular than data engineering because “science” is sexy and “engineering” is not.

[@KenJee_ds]

Don't tell the Machine Learning Engineers that lol

[@syhusada1130]

What's with the pizza?

[@KenJee_ds]

I just like making pizza and wanted to share 😅

[@javohirxusanov1229]

Why there is irrelevant food prep episodes? 🤣🤣🤣 Ps: actually, video and its point is to the point

[@KenJee_ds]

I just like food haha

[@andradaolteanu4598]

Great! Now I want a pizza 🤧

[@KenJee_ds]

Haha I'll make some if you ever visit Hawaii!!

[@miguelrios1997]

if i become data scientisit i become hot like you

[@KenJee_ds]

lol I have no clue how to respond to this question

[@shresthaditya2950]

1:54- 5:30-

[@shauryajain4851]

What is the future of data engineering?

[@KenJee_ds]

I think it will be intimately tied with data science. I expect both to continue to grow!

[@MrGamGamer]

Now I want a pizza

[@KenJee_ds]

Lol

[@Demojara]

dbt ftw

[@KenJee_ds]

💯