How I would learn Data Engineering (if I could start over)

In this video, I’ll share my step-by-step process on how I would learn Data Engineering if I could start over. Data Engineering is a fast emerging field within the Tech industry; where more and more people from traditional data science/software backgrounds are pivoting towards. We’ll cover the fundamentals of Data Engineering, and talk about some advanced topics you’ll need to learn in order to maximize your chances of success. Hope you enjoy this one!
Timestamps ⏰
0:00 Intro
1:45 Fundamentals of Data Engineering
4:51 Core Data Skills
7:30 Advanced Data Skills
10:07 Mindset
Notes from video 📝
bittersweet-mall-f00.notion.s...
bittersweet-mall-f00.notion.s...
bittersweet-mall-f00.notion.s...
Who am I? 🙋🏻‍♂️
I'm Jay, I love making videos about travel, self-help and tech. I currently work in New York City as a data engineer, but I grew up in Malaysia and lived in the UK when I was 19. Back then, I had no idea what life was about, moving to so many places, navigating career in Tech. Today, I've learned a lot and wanna share my perspective through filmmaking.
Socials 📱
linkedin: / jayzern
/ jayzern
Sub Count: 152

[@ekkyarmandi]

I'm a freelancer and also self-taught programmer who wrote a lot of Data Scraping scripts using Python for my clients. I have a loyal client who requires me to extract data for them every month. I repeated this manual process for about 3 months until I realized the scraped data was getting bigger in size and dimension. I thought it was not a data scraping job anymore, it was a data engineering job. After doing some research online, I figured out some useful tools I can use to manage all data collected by the data scraper. I start by storing all the data in an SQL database and automating the process using Apache Airflow. That was the first time I thought Data Engineering was the right direction for my career path as a developer. Thanks for your advice, Jay.

[@raffymcfee9846]

how long did you train the skills of Data Scraping and Python to make money out of it?

[@ajdinpipo08]

wow, thanks for sharing this, it's really inspiring me to learn more about data engineering

[@vuxuanhuy9079]

​@@ajdinpipo08 You can sharing resource you learn DE ?

[@longdatadevlog]

I love the way you split the career to basic and advanced skills. I’ve been keep doing that to become the data engineer. Thanks for sharing!

[@Scarletkillerxz]

Beginning my path here, great video with no filler!

[@triposat]

Best Roadmap ever. Just get started with it. Thanks a lot!

[@shubhamwaingade4144]

I wish I would have seen this sooner! Great and crisp content.

[@carlosleotaud3255]

really great and clear explanation ! please keep this way. I would recommend this video to anybody wanting for learn about data engineering.

[@gamersant7466]

What a gem. I am currently pursuing AI/ML. Getting data scientist job as a fresher without experience is almost impossible so had to start with either as data analyst or data engineer. Since now everyone is becoming data analyst so their is demand and supply issue in the job market. Thinking of expanding my domain knowledge to data engineering as well but as you stated “got lost” with the available resources. This video helped me to make a roadmap to pursue data engineering.

[@jayzern]

Thanks for sharing. I agree it's super difficult to get a DS job fresh out of college. Branching out to DE is a great way to "stand out" from the crowd.

[@christopherleader6517]

Thanks Jay for posting such an informative and insightful video!!!

[@DeltaXML_Ltd]

Very useful tips for those who are just starting out. Great content!

[@lucasmartinez4859]

Loved your video, it surprised me how you started the second section getting us into the book. It felt like already starting :) thank you!

[@yurialenccar]

Powerful tips! Thanks a lot for lightning the path!

[@bouihabdelouahed1579]

Thank you. Very kind of you to put out this valuable easy to digest and to follow content.

[@tolaseadegbite1027]

I am transitioning from web development to data. Fortunately for me, I learnt development skills from Learnenough. I have gone through their Commandline course and now on Python and Python for Data Science. I am excited to be doing this. Thank you for this content.

[@rakeshverma6867]

Very good explanation. Love it for the conclusion with the complete motivation.

[@amritasrivastva]

Amazing video with to the point information. Thank you Jay!

[@PuchaiM]

Thanks for the useful tips. Short to the points yet very effective !

[@lunariasophie]

Thank you for such an informative video, I'm a junior data engineer and your channel is an invaluable resource!

[@GoBlue402]

Great work on explaining the concepts 👍🏽

[@dereklei1746]

This is super super SUPER practical and insightful and helpful. Looking forward to your future content and successes!

[@jayzern]

Thanks for the support Derek

nice and well structured information, liking this!

[@lilbean_eth]

thanks for delivering a well produced video centered around DE

[@vuyilemagwaza3236]

I recently signed up for the data engineering course from Datacamp and I have covered pretty much most SQL courses but I'd love to go back on a few of them again, I work as a data analyst but I feel a career in Engineering would be great.

[@holmesketrick7]

We are in the same boat

[@Xavierpng]

​@@holmesketrick7mee to

[@kaizen9554]

Exactly. SQL can be voluminous to learn. It takes time and practice to let it sink in

[@claudiobarrera3186]

Hi, I’ve done the complete python track of data analyst, mostly of the sql courses of DataCamp. I’m from Chile, Latin America, do you have any advice for me to find remote jobs as data analyst in the US?

[@korands]

hi may I ask, does datacamp teach you how to normalize column tables in excel format to tables in sql / other database formats? thankyou

[@shreyashthengne]

This was really helpful. Thanks for sharing ❤️

[@pramodhkumar8771]

Great information. Looking for more videos from you. Keep doing .

[@ministerbo3025]

Hands done, the best video I have seen on the subject. It's no nonsense, practical and easy to digest (of course after you put in the work). I am just starting out in my journey to become a Azure Data Engineer so I thank you Jay. Thanks for sharing and being so transparent. I'll keep you posted on my many adventures. For what it's worth, I'm 70 years young and still dedicated to life long learning.

[@jayzern]

That's amazing! Love the attitude on life long learning

[@aidylweiss.]

Wow, that is so encouraging! Godspeed!

[@ajacosta6419]

Thank you for this video, it created more of a path for me.

[@grandmundi7107]

I have been working with python so I think a few very important things 1) pyspark 2) pick AWS and/or Azure to learn their data engineering stack 3) very solid SQL skills.

[@Sivx76]

Outstanding video! This video was remarkably well-explained and detailed. It is also structured very well and was easy to follow along with your reasoning. It was a pleasure to watch. I am a Senior iOS Engineer looking to dive deeper into ETL and related topics. This helped me organize what I want to learn. Definitely subscribing.

[@gustavojuantorena]

Best roadmap for Data Engineering on KZbin. New sub Here 😊. Greetings from Argentina 🇦🇷

[@ashish-blessings]

Thank you for sharing insights

[@avdulkhadyskhurmangazy2973]

Hey Jay thanks for sharing all those valuable information with us. I hope to you see soon in NYC! :D thanks again brother!

[@harshiniamarasena3616]

Nice briefing,waiting for more knowledge sharing on data engineering

[@Digital-Cloud-Tranformation]

It was amazing introduction and guide line. Thanks

[@artech4910]

Thanks for this video! I always loved SQL and manipulating data but was put in the software Eng camp. I hated programming so gave it up and moved to IT/admin now I realize data engineering may have been a fun career choice and this video has given me a lot of interest on things to get back into!

[@markkevnjflores]

Amazing and informative!!

[@fazerydevs]

Hello broo, your video was great! I'm an aspiring Data Engineer and what you said about being overwhelm with many ambigous word on the first year was true, sometimes I feel burn out too. Kinda sad but I know this career is great so I keep it up. This video helps me regain my motivation :))

[@albertowusu-banie154]

Very insightful video 👏👏👏

[@dianeavwenaghagha1193]

Eye opening, thanks

[@ZzZzZiggy]

Hugely useful! Subbed!

[@cooltrucly]

i'm a junior data scientist transitioning to data engineering/MLOps. i'm also based in nyc! i'm recently discovering my interest in filmmaking too! looks like i stumbled upon your content at just the right time. just subscribed!!!

[@jayzern]

No way! data engineering/MLOps and filmmaking is such a great niche

[@nagusameta366]

I am a fresh graduate in Information Systems and my internship had me working with the team's data lead using Python, Apache Airflow, and PostgreSQL. I was glad and became more confident from the experience but once I was job hunting, I kept finding data engineering job postings demanding tools like Azure, AWS, Spark, Hadoop, etc. I was discouraged as I only had the three I mentioned. But I was a bit encouraged when you talked about tools as getting more specific on what the task demands, and that foundation through concepts can be better.

[@pistachorios6796]

Is an excellent introduction I’m far away from being close to a data engineer. I’m just starting off in learning data analytics, but I enjoyed very much this introduction. I felt like it was well explained, and definitely sets the tone moving forward in my learning.

[@peekknuf]

Holy f, brother this is an amazing and quick overview. You def got potential in this, keep it up!

[@jayzern]

Thanks man! So much more to come

[@MaddyIndia]

Excellent information man

[@user-uw3mq4yb6m]

Thank you for this great video

[@mairios521]

Thank you, thank you a lot, Jay!! I recently graduated as a Data Scientist but when I see the options of a career path I think Data Engineer fits better with me than others. This video is helpful!! I'm a new subs!

[@surekhas8098]

Well developed and presented informative video. Thanks a lot 🙏👍

[@t2jhkt3b8adb5]

i was pretty much nervous and scared how to start, thank you for clearing things up :)

[@raneshguruparan]

Love the content bro, hope you keep growing. Its cool to see another Malaysian in the US whose also in the field of Data while making KZbin videos on the side.

[@jayzern]

Thanks bro! Awesome to see other Malaysians too

[@olivert.2241]

Awesome video Jay! I will personally be looking through the resources that you've recommended.

[@jayzern]

Thanks oliver, u the best

[@elizabeth00653]

Useful information thank you

[@LeweiHe-tc3rh]

Thank you so much!

[@rabihbadr54]

Great job with your career man, mixing technical and artistic paths, great balance.

[@jayzern]

Thanks for the kind words man!

[@jesseantwi6270]

Hello bro ,i completed my undergraduate in actuarial but failed miserable and i have been down,filled with regrets and lack motivation ever since but your has given something to believe and get me to regain my confidence . Today begin a new journey,my journey into teaching myself to become a data engineer hope to come back to this post after a uear to share my progress.

[@mondlistyleman9389]

you’ll do well, being a DE is thee best. and you’ll easily work worldwide

[@RNSS_0]

I am going back to your comment next year to check you. keep up 🫡

[@Odin.x.]

How has your journey been so far?

[@MajorDougx]

​@@Odin.x.He came bck to reality

[@Leibniz920]

No long ass intro no stupid music I love this video

[@alexperrine3497]

I've been a data analyst for a few years and I am just starting to learn the material to transition into the DE space.

[@educabo1]

thank for information and good video

[@user-jp9mg1gj5e]

Man, this was an awesome video. I'm currently a data analyst and my company has just offered me a junior data engineer role. I have a lot of learning to do and the path you have provided is helpful!

[@jayzern]

Amazing man, good luck in your new role!

[@ColeWhittington]

This is a super high quality video! Cant wait to see this channel grow!

[@jayzern]

Thanks man!

[@xERRRAHx]

I started / enrolled in courses on DataCamp on path to be Data Analyst then will take courses for Data Engineering. I’m a super excited to learn about Data and the career opportunities

[@Cxba222]

This is hands down the best high level introduction to data engineering I’ve come across. Thank you for making this amazing resource guide as well. I work as a BI analyst now and wanting to expand my skill set so I can truly become a data engineer. I appreciate your time and hard work!

[@jayzern]

Wow that means a lot! Keep working on expanding your skillsets. Learning is a non-stop journey

[@anishnehete]

​@@jayzernI'm a college student , I am currently doing the az900 and then I shall do the data bricks certification, are these enough to guarantee an entry level job in data engineering?

[@tomashorych394]

Thank you for a great introduction! I have been working in data science for quite some time, but over the years, I realized that the analysis part is much less interesting for me than the data part.

[@zeusanton]

Thank you Jayzern

[@chunporo]

This video helps me learn how to learn ML and process data.

[@msmoscow1859]

Hi Jay! Thank you so much for this video. Can you please make a video about DE vs Cloud Engineer. It seems that CE make more money but require more skills like AWS/Google Cloud; but a lot of skills overlap as well.

[@bilal.rk09]

Thankss very helpful..

[@mikehattias5837]

much appreciated

[@josephbui5912]

Thanks for the mindset part! I got a BS in DS, but shifted into DE because I enjoyed the backend of getting data for the analysis more than the analysis itself. Being new in the field, it’s definitely overwhelming with all the tools. However, Ill try to take your advice and start small by reading into a tool and doing instead of learning. I guess the best way to learn is to build and try? I always overcomplicate stuff and end up not following through because I try to tackle all these tools at once but will tackle it in smaller parts. From there, keep adding on to my project. Excited as this will be my first “DE” project. Thanks!

[@jayzern]

Amazing! I started out with a Bachelors in DS too

[@updateserver]

I do the same thing. It's a blessing and a curse. Scale back. Focus on one tool. Do a project. Join everything together like one complete Lego set. Blog about it. Reach out to recruiters. Wait for the 🎣 & snag a job

[@janricmalate6793]

Great content, keep up the good work!

[@jayzern]

Thank you!

[@DanielleBaylor]

About to dive into SQL this weekend. I'm just starting my journey into data analytics though. I don't think I have what it would take for an engineer 😅 but I still like watching videos like this cause I like looking at different things

[@jayzern]

Amazing! Good luck on your data analytics journey.

[@DanielleBaylor]

@@jayzern thank you! It's a little intimidating, but so far I'm hopeful ❤️. Kind of wish I'd known about this when I was younger though lol

[@taishamikell1438]

Sis!!! 🗣️YOU HAVE WHAT IT TAKES. Commit to putting in the work and you can become an engineer. Don’t let self-doubt keep you from reaching your potential. You got this!!

[@LisaEveShow]

How are you doing on this journey?

[@DanielleBaylor]

@@LisaEveShow considering I have ZERO technical background, it's going ok lol. I'm starting to get the basics of SQL and about to start Python. I kind of wish I'd spent more time on the hard skills this whole time

[@k.h.p.9862]

I appreciate this video. At 3:00 he mentions that "the objective is to learn about data concepts, not computer science." I'm literally a total novice (but getting deeper into this), and I always thought that computer science was partly focused on data concepts. Or, maybe he means that data concepts is more specifically focused, in contrast to computer science which is extremely broad.

[@krisnaaprileo4575]

Do I have opportunity to learn together with you mate? I also a total novice but getting deeper 😂

[@jasperhumpsman7151]

Starting Data Technologies course on September, for now I don't think about money, I just think how to survive and not become mad during education because of huge amount of information

[@atyt22]

This is awesome, thank you for sharing your knowledge. I'm a Software Engineer studying for my Master of Science in Data Analytics, maybe becoming a Data Engineer is in my future? Right now, focusing on my SWE job and learning SQL and Python through school have been my ways of learning and doing and while gaining experience (as well as coding projects on the side when I can).

[@jayzern]

Thanks for sharing Aaron. A lot of people are in the same boat and can totally relate. Enjoy the ride and good luck in your Masters degree.

[@atyt22]

@@jayzern thanks I subbed, will be tuning in for more videos!

[@jaydecanon1314]

you earned a subscription because of simplicity and being efficient on teaching!

[@igechasamuel3573]

This is super helpful. More of these please

[@obinnaanokam2760]

Great content.. the screen you have online with the contents where are u reading this from? The book or from a website?

[@lowayne5]

I'm currently transitioning from Data Analytics to Data Engineering. I think DE is only going to grow over the next decade as AI continues to blow up. Thanks for the amazing video jayzern!

[@Opq01939]

Why u transitioning from data analysis To data engineering? Is there any problem with data analysis?

[@evan9536]

I'm currently trying to sharpen my DE skills and make the transition as well, this video was incredibly helpful. I have an MS in Data Science, an undergrad in Data Analytics & Finance, so I'd love to add these skills to my toolbelt. Perhaps I'll look into Analytics Engineering roles!

[@DZONTRAH]

Thank you, its a good video

[@jayzern]

Thanks for the support!

[@yasmeenkhan3707]

Great information Thanks Jayzern . Could you make a video on HOW I WOULD LEARN SNOWFLAKE DEVELOPER [if i could start over ]

[@mohit9772]

its a very useful video thankyou

[@tatatadooooo11]

many thanks, i have confuse 2 things. Learning tools or Building concept . after watch your video. i have more specific look . kee sharing !

[@ngochongnhu7260]

thank you

[@spoonikle]

This stuff is low key fun though. ❤

[@gtizzle101]

Designing data intensive applications is another great book by O'Reilly

[@jayzern]

One of my fav books!

[@ridhirajput7779]

Hey, First of all thank you for making this video, Is there any way you could provide the doc you are referring to in the video? That will be so helpful.

[@erenyeager4452]

Nice Video. May be you can suggest a couple of courses for each section of the video, bcos that is where people have a hard time choosing and taking action.

[@Deiniol_1]

Exactly

[@jayzern]

Ooo thanks for the suggestion, will include that in future videos

[@RNSS_0]

that is the video I am looking to

[@Sensei__Nate]

Started my journey in learning, I'm currently a systems engineer and it suck lmao, requirements this, requirements that.... anyways, I feel like I found a good solid book to read and some videos on you tube to follow along with for postgreSQL

[@sham.saravai]

in the project youve mentioned in the core data skills section. are the resources that are involved free? as far as I know, airflow is open source. I have worked with databricks and the whole azure ecosystem, but it was my dads companies subscription.

[@StevulaDodoonline]

Hi Jay, is it recommended to have a good overview of certain areas of mathematics before starting to learn the individual stages of data engineering?? Like linear algebra, discrete math, and the like?

[@wendermezini2688]

Wow top video! could you give us a list of youtube channels + blogs for Data engineers? Take care!

[@RaNd0MiZeD]

Thanks

[@technicaltech8141]

It's lot's of help me JAY..

[@sizzusharma9138]

Thank you can you suggest demanding portfolio or personal project that helps in placing over best mnc

[@rebelgordo2339]

6:04 that throat clear was intense 😂

[@oluwatobipriscilla6956]

Hi thank you very much for this explanatory and helpful video, Please I keep trying the link to the notes from notion but it's not working, can you please help sort it out. I'll be very grateful 🙏. Thank again

[@tomasgimenez7480]

Hey so i’m trying to learn Data Engineering and currently following this amazing guide, do you have any recommendations on any alternative courses for those ones that are paid?

[@ustafatih]

Hi Jay, i have a question for data visualization. Do you advise power bi, for data visualization?

[@potatofarmer2099]

Hi, how do you suggest building and presenting a portfolio?