How To Become A Data Engineer in 2023 - From Coding To The Cloud

How do you become a data engineer in 2023?
There are a lot of paths to becoming a data engineer, and a lot of skills required.
In this video we will discuss the required skills and some of the courses you can take to help get you there.
Free Data Warehouse Book
aatinegar.com/wp-content/uplo...
Python
bit.ly/3FKp2x6
SQL
bit.ly/3jgcUMD
Docker And Kubernetes
bit.ly/3jbKBPj
Data Warehousing
bit.ly/3PFkj4e
AWS Cloud
bit.ly/3HLdV9M
GCP Cloud
bit.ly/3jkCCQg
0:00 Intro
3:01 Programming And Python
4:36 Programming Project Ideas
5:16 SQL
6:20 SQL Project Ideas
7:00 Data Warehousing
9:02 Cloud
11:15 Docker And Kubernetes
13:00 Other Things That Are Good To Know
14:00 Getting The Job
Looking to start you're own data engineering/analytics consulting company, then you should check out my new course here
courses.technicalfreelancerac... - and use the coupon code "deconsult" to get 50% off
If you enjoyed this video, check out some of my other top videos.
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/​​
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

[@monishagowda8928]

Hi! Thanks for the informative video! I admire that you answer the comments so patiently. I joined my company as a SE and was transitioned into a DE. Your videos are really helping me figure out my career

[@DarshilParmar]

Thanks for the projects shoutout from my channel, great video 🙌🏻

[@SeattleDataGuy]

Thanks for always putting out great project videos and finishing them!

[@mching4473]

✨ manifesting ✨ to get the last step done this 2023.

[@SeattleDataGuy]

We believe in you!

[@marcoshenriques3118]

Me too. I already have Python, SQL mastery and AWS Solutions Architect. I would add between step 1 and 2 Linux and networking

[@nicky_rads]

Nice video ! Very thorough break down of the career path. Totally agree that moving in DE is often a pivot from another data or technical field. Looking forward to 2023 content !

[@SeattleDataGuy]

There is so much content I want to put out!

[@Korudo]

Thanks for this video. I’m planning out my year and how to develop these skills further and this is super helpful.

[@FinanceNik]

Great Video! Love these roadmaps when figuring out whether or not I want to switch my professional focus

[@Roarrain]

Recently i just learned the basic knowledge of data analysis but i feel it not enough so i go to KZbin to search and i saw your video and i hope i will become what I want to be,thank you ben!

[@satyajitrajbanshi3620]

Thank you so much.. I'm a fresher in data engineering

[@pillar258]

Amazingly detailed video. Thanks for elaborate explanation. This is exactly what I needed now. THANKS again.

[@adalke2]

Fantastic advice, thank you, Ben!

[@SeattleDataGuy]

Glad you found it helpful!

[@LiJacob-gc4cy]

Thank you so much for your sharing!!! This is exactly what I am looking for. After watching your video, I am glad to notice that I might be closer to Data Engineer than I think I am. As a Data Analyst in a relatively small organization, I wear multiple hats: administrate the local database (weird as it sounds, this is the truth, it is only report-purposed, and does not make an effect in our product environment), created an SSIS flow to update the database every day, built data pipelines for data from different sources, and in this process, I applied Python to make API calls, and for sure, SQL is my daily routine, when everything is ready, all the data is loaded into Power BI. Though my boss mainly acknowledged my progress in data visualization, and indeed most of my time is spent on writing DAX queries, I try to challenge myself recently by automating some of the monthly reports myself. I am really happy that I found your video when searching the roadmap of becoming a data engineer.

[@delanescott7872]

did you go to college or did you take the self learned route?

[@timwebster85]

Thanks for this!

[@SeattleDataGuy]

Glad you found it helpful!

[@SIBUK]

It would be cool if you could do a video where you go in depth about the technical challenges they ask you to do during interviews.

[@notquitehim]

I am a software engineer with a background in CS, I've been trying to shift my career into more data centric engineering so this is much appreciated :)

[@SeattleDataGuy]

Glad you found it helpful!

[@oceanview3165]

I am on the same boat.

[@glindaiserrobo8746]

And did you succeed?

[@krishnadaily]

Great Video! This content is really helpful for those who are looking to move into a data engineering role in 2023.

[@SeattleDataGuy]

I am so glad you are finding the videos helpful!

[@user-cm6iy6uk2y]

Best channel😍

[@GamingJasper]

Really useful video, thank you! I am an analyst, slowly turning into a a data engineer role, and this gave me a great "roadmap" to fill in some knowledge gaps I have :)

[@SeattleDataGuy]

I am so glad to hear that! any quick question i can answer?

[@GamingJasper]

@@SeattleDataGuy You did a great job answering a lot of questions I have. I think there is a lack of simple data engineering "end-to-end" project examples to see how everything fits together. The ones I found are pretty advanced. I hope to create a few of my own to document my progress and help others!

[@SeattleDataGuy]

@@GamingJasper Hmm, what is too advanced. I think it's sometimes hard because once you have been doing this for a while you make a lot of assumptions on what people know. Plus in order to do a proper DE project you're likely using several services, maybe on the cloud, airflow, some data viz, etc.

[@musiliadebayo]

Hi Jasper, I am also trying to transition to a DE role. I currently work as a Data Analyst.

[@ramd2667]

Very informational. Quick question, what is the ide at 3.52 minutes?

[@chengyubai5330]

great video and very helpful！

[@SeattleDataGuy]

Glad you found it helpful!

[@SeattleDataGuy]

If you want to learn more about consulting/freelancing in data analytics fill out this form - forms.gle/oQeFNeetQXgRk73s7

[@sandeepsoni6628]

really usefull video , love from india

[@NguyenPhuong-nk4lm]

Your video is so useful with a fresher like me! Thank you

[@SeattleDataGuy]

So happy to hear you found it helpful!

[@MartykData]

I've been floating between data science and data engineering this last year within the business I work for, building ETL pipelines and DWH & Hive tables for the past year on top of my predictive modelling role. Looking at this, it seems that the Docker/Kubernetes is the only thing I'm kind of missing from this, which is good to know, but is containerisation a difficult skill to learn? I feel like I'm of more use to businesses as a data engineer, so wondering if there are any data scientists that have found they get more valuable output in DE rather than DS?

[@miguelvotrejr]

First, not even finished the video. But I appreciated the objectivity and pace you done this.

[@SeattleDataGuy]

Hello Miguel, Thanks for the comment. I appreciate it!

[@jawadalla11]

What are your thoughts on someone working in a leading ERP space for data integration and migration, to a data engineer? I would think there’s more variety in the work as a data engineer and better growth opportunities.

[@firefoxmetzger9063]

An often overlooked avenue to fast-track your coding skills is volunteering for (larger) open-source projects. You need some amount of coding skills (~3-6 months), git knowledge, and a friendly attitude. After that, it's a fast track because working on such OSS projects will expose you to all the best practices for writing maintainable code and the relevant tools for that. It's also extremely rewarding, because your work _will_ get used instead of succumbing to bit-rot somewhere on your external drive.

[@svenko333]

Great video, this kind of videos really mean a lot and thanks for them. I wanted to know one little thing that you meantioned in the video, but didn't expand on it. Where would you learn practical skills for Azzure? My company is in a process of going to cloud in the next 2 years, so I would like to learn Azzure for myself since we will be working on it.

[@SeattleDataGuy]

Thanks for the comment. There are several azure certs you can get. Heres a course that could help get you ready and provides practice on udemy www.udemy.com/course/az900-azure/

[@Ivan-qg5yl]

GOLD

[@mrsaurabhsha]

Kubernotes is one hell of a topic even the cloud professionals find tough

[@SeattleDataGuy]

knowing enough to be dangerous can be helpful. Plus sometimes there is no platform engineer

[@andrewmaxwell9399]

Hi! I've asked a few months ago about the question on how to switch from ETL to Data Engineer, with conditions that i don't have any previous of coding experiences. Now , I don't what to call this "lucky" or what lol I got a job in one of the biggest bank in my country, they're currently now only using "traditional tools" such as Pentaho and SQL Server,Postgresql and Mysql. I know it's not the "true Data Engineer" does, but after 1 week of in the job they said that they wanted me to be the one who started the "revolution" ,eventhough they had like 4 other members of "Data Engineer" which are more experienced in terms of years of experiences. While I'm still trying to grasp their development processes with the Pentaho tools, now I'm given another task to do a POC thing of Spark and PySpark on multi node. I'm not confident that I could do this, and what if I'm not able to pass their probation (3months) period because I didn't successfully implement Spark and PySpark for their pipelines? Should I put that in my Resume? What to do if I had to search for another job and they ask me "why did you fail your probation?"

[@SeattleDataGuy]

Hmm, thats interesting are they giving you any help? It's a big ask for you to have to take on a spark POC without prior experience. Have you already failed the probation period or are you just concerned?

[@andrewmaxwell9399]

@Seattle Data Guy i was asking the more experienced "data engineer" or should we call them ETL Developer, "Hi man, have you ever using these stuffs?" He said "nope, i thought you already understand 'that kind of tools'." So yeah pretty much i wouldn't likely to get much of a help maybe? Lol Still in my probation period, now i'm entering week 3, but i'm really concerned if i don't pass my probation period then what should i tell to the future HR lol

[@SeattleDataGuy]

Interesting. Do you have other job experiences? If you really don't do well in the role but you have other job experiences, its not the end of the world to drop it off your resume and extend your past role an extra month or two so people don't feel like you have a big job gap

[@andrewmaxwell9399]

@@SeattleDataGuy yes i have experiences in etl, thankyou for the advice :))

[@JorgeMunoz-gm2fr]

Any thoughts on the growing popularity of graph technology (Graph Databases, Graph Data Science, etc.)? Would love to get a DEs perspective on this.

[@SeattleDataGuy]

I personally have not seen anyone use graph databases for data engineering style work. Sure, I see plenty of SWEs use it but not DEs

[@Eriddoch]

If this is what data engineering is... I think my official job title is truly wrong 🤣. I hardly do anything with data modeling or ETL work. It's all infrastructure, REST APIs, and helping get models into products. MLOps is a fun space, too :D

[@SeattleDataGuy]

oh nice, more of an MLOps person then?

[@michelleandthemachine]

Thank you for another helpful roadmap to becoming a data engineer! I'm curious. You mentioned that you have a Udemy subscription to get any courses for skills that you need. Do you complete the courses in full or specifically look for the info you need? How do you determine if a course has "good" material? Thanks!

[@SeattleDataGuy]

Glad you enjoyed the video! I don't always complete entire courses. Sometimes I just go through the sections I need

[@matth1139]

Great video Ben. I’m curious what you think about Chatgpt and the hype around it. What do you think are the implications for data science/engineering jobs in the near future?

[@SeattleDataGuy]

Good question. I think Darshil has an interesting idea where he is going to try and see if he gives the right prompts to Chatgpt, if it can actually build some more complex queries.

[@wikiexportsindia3146]

very clear explanation - no college degree can give this

[@SeattleDataGuy]

Glad this was simple!

[@shermin2o9]

Hello. I am an aspiring data engineer and am interested in doing some data warehousing projects to add to my resume. I cannot figure out which cloud warehouse may be best (cheapest) for me to practice with as a non-business user. I am willing to use Snowflake but I have no idea how much that will cost me. What is your recommendation for beginners?

[@toddstrain3629]

The Udemy course you spoke about (MySQL for Data Analytics and Business Intelligence) is different than the one showed on screen (Data Warehouse - The Ultimate Guide) Can you clarify, please, which one you are recommending?

[@MrDarkplace22]

Hi, just a video suggestion, can you make a video on how to work with a data engineer, I am data analyst onboarding a data engineer and I want to help him more

[@SeattleDataGuy]

Let me add it to the list!

[@shaneboo1531]

Thanks for the informative video, I was wondering is Coursera still relevant for me to get into Data engineering, specifically the IBM prof course.

[@sankyeat]

Do you need a computer science degree for this or can you do it without the CS background?

[@mauludinrohman6177]

Hello sir, i just start a job as bi developer, iam from a fresh graduate college, and right now try to get data from apis, i am using python,postgresql, airflow ,i just learn many things in airflow like about executor, change from sequential to local, change sqlite to postgresql, next time gotta learn celery, or kube. And because the data from api change everytime, gotta switch from batch airflow, to realtime. Do u have any suggestion for me sir 🙏, what should i learn, kafka, pub sub, spark, airbyte, cdc ?

[@hubbiemid6209]

Could you recommend any project tutorials that implement all of this or at least the latter skills (cloud/containerization)?

[@SeattleDataGuy]

Oh darshil has some of these!

[@antoniomanuelfernandezcant823]

Is Kubernetes really important in the field of data engineering? Are you usually use this? Can you expose some examples to use it? Thanks a lot for your content!! Merry christmas.

[@SeattleDataGuy]

Yes and no. It usually dos depend a little bit. In theory you should have a data platform team that sets up say your airflow instances to run at scale and have them running and K8s. But sometimes thats not the case and you gotta do it.

[@SeattleDataGuy]

also Merry Christmas!!

[@pallavisrivastava2187]

Hi Ben, I was waiting for this video for past 1 month. I want to transition from Data Science to Data Engineering. I am wondering if you can suggest some resources to learn Spark. It will really appreciate it.

[@SeattleDataGuy]

Hello Pallavi, glad you're finding it helpful! This udemy course is classic www.udemy.com/course/taming-big-data-with-apache-spark-hands-on/

[@pallavisrivastava2187]

@@SeattleDataGuy Thanks Ben. I greatly appreciate all you do to motivate and guide us.

[@ukaszdugozima816]

Hello! Thank you for your invaluable video! I find it extremely useful for beginners! I would like to ask about one thing regarding this map. I learnt Pandas in terms of Data Wrangling and Transformation. Therefore, how about Pandas for Data Engineers? Is it useful tool for ETL/ELT transformations? Obviously, the next step will be PySpark, but I would like to start learninig Pandas. It seems it is a good path for the next one. What do you think about it ? I would appreciate it if you could share your views about it.

[@DataDevWorld]

Pandas for data wrangling definitely comes in handy when doing intermediate transformations for your pipelines. However, you do not want to use it for heavy transformations or when dealing with big data for performance reasons since it runs in-memory.

[@ukaszdugozima816]

@@DataDevWorld Hi! Thank you for your comment. I focus on PySpark in order to deal with big data. What do your think about this approach m

[@shubhamgupta997]

I started my career as Databases administration after my bachelor's degree in 2021 and now want to shift my career in data engineering. Will it be a little easy for me as I already have oracle database experience???

[@SeattleDataGuy]

It's a good baseline. If you add in some programming and data warehousing I am sure you can make the shift. Probably learn some cloud components too

[@MadaraAMVz]

I just graduated from comp sci, and ive been working on my general back end java skills applying for jobs, and i ended up getting a junior data engineer role haha

[@yuktibhatia7607]

I have worked as an azure iaas admin, and want to switch to data engineering.I have very basic understanding of python and SQL. There are soooo many resources on the internet that I keep getting confused which one to follow and end up completing nothing! Ideally,in how many months should I be able to make the switch?

[@SeattleDataGuy]

Thanks for the comment. How much time would you be able to dedicate to learning?

[@yuktibhatia7607]

@@SeattleDataGuy 2-3 hrs daily..

[@m04d10y1996]

What about big data and distributed compute

[@Himricks]

Hi, Thanks for the sharing the data engineer road map. I wanted to know if, after i complete all this projects and courses , will it be sufficient enough for me to land a job ? consider me as a support guy who wants to switch to data engineer role . I am currently struggling to get the job in it , cause i don't have prior work experience in data engineer. What would you suggest me.I am looking forward for a response from your end mate : )

[@bunmioke2697]

Ditto your question. Let's hope he gets to see this and respond.

[@edwardmitchell6581]

I've gotten in and have three Google certs. Now how do I increase my salary?

[@zaskens8083]

Can someone help me? I have doubts about which specialty to take in the data sector (engineer / scientist). I would appreciate any advice :)

[@Razi1357]

Hello! i have masters in data science. Now considering a 2nd masters in AI. Before my masters i have a degree from different domain. I have no working experience in data. Do you think my 2nd masters will broaden my job opportunities. For example i could still apply for a data engineer post, right? Should I go for that 2nd masters in Ai? Or maybe try another degree with a less defined niche.

[@SeattleDataGuy]

Generally getting work experience once you have a masters is better

[@sharifahmed1925]

Do you guys need on top of Python and SQL, would I need to learn Java or C#?

[@SeattleDataGuy]

Sometimes C# or Scala often come into play (java too) just depends

[@bjsk6109]

What about the "Data warehouse fundamentals for begginers" course? Do you still think its good?

[@SeattleDataGuy]

It's still a classic, I just happened to most recently look through Data Warehouse - The Ultimate Guide, its more expensive(at least when i go to the page). So you can always just take the beginners course + read kimball

[@huntwiller8231]

I didn't see you mention DSA. Does the DE need to dig deep into the DSA?

[@SeattleDataGuy]

You do need some DSA but not as much as a SWE in many cases. UNless you're a very low-level DE

[@Trihawk7]

what do you think of aws glue to learn?

[@SeattleDataGuy]

I can say that I am running a survey on what tools data teams are using and out of over 300 entries, 50+ have said they use AWS glue on their team. I do think its worth knowing but I don't think there is a rush to learn it (if that makes sense). First get the basics and once you start poking around with the cloud you can start testing out different services.

[@josesilesblanco9557]

Hi @Seattle Data Guy, how hadoop ecosystem and apache spark is not good to know as a data engineer?

[@SeattleDataGuy]

Thanks for your question. Are you asking if hadoop is also important?

[@josesilesblanco5944]

@@SeattleDataGuy hi again. Yes, kind of. I’ll be working as a Business analyst in Amazon and, in paralel, studying Big data. I’ve started with hadoop ecosystem (currently finishing it with apache Hive) and Then I’ll move to Hhbase, Scala and apache Spark, finishing with Airflow and doing a project in AWS. Please I would like to know if It’s a good path or I should go more straigth forward. Thanks for your time!

[@succeeder3044]

Please i want to switch from front end development to Data Engineering. I want to take IBM certification, is it advisable to take this path? Please your response would be greatly appreciated 🙏

[@SeattleDataGuy]

This video might help! kzbin.info/www/bejne/i6Gwc5ecnJeNqKc

[@ass2412]

I'm using SSIS. What are your thoughts on that? Is it in demand?

[@SeattleDataGuy]

SSIS is a great skill and it's where many people start. I do imagine you will want to add some cloud tools like ADF as well and libraries like airflow

[@ass2412]

@@SeattleDataGuy but as of now I'm not using adf with ssis, I use AWS Glue and pyspark as well. Hopefully I'll be able to move to senior position with those skills

[@gillallasaishekar174]

I am from India and I have simple question for. Can I join 6 months Full Stack Big Data Engineering bootcamp for learning Data engineering and do freshers get hired who is from Non CS background ?????? Thank you.

[@SeattleDataGuy]

Is it possible. Yes. Will it likely be harder until you get actual job experience. Also yes.

[@debajyotiroy6843]

​can you please share how to move from Big 4 (PwC) to Faang type companies. as a data engineer without no prior experience in data engineering industry.

[@SeattleDataGuy]

What experience do you have thus far? Data analyst?

[@debajyotiroy6843]

@@SeattleDataGuy I'm currently a fresher. Just graduated out of college and got a job at PWC. I'm not yet assigned for a team.

[@SeattleDataGuy]

If you're just getting started then I would say you should try to work in a data analyst, data engineering or programmer role or projects. That way its easy to transition

[@debajyotiroy6843]

@@SeattleDataGuy thanks will do that. Let's see what team i get picked for.

[@Kami.yato.]

is it realistic to go the self taught route with no degree? I have experience with Python and building apis

[@SeattleDataGuy]

It's possible, but many jobs require degrees. I once had to wait to get a full time role because I didn't have a degree

[@eugenefritz6669]

What's your view on chat gpt?

[@craig5716]

Good question, would love to hear his thoughts on it and/or to make a video about it

[@SeattleDataGuy]

It's pretty crazy to see what it's been able to handle. Perhaps at some point it'll be able to take human instructions and write the exact queries we need. But then people will have to get really good at giving the right prompts. I imagine we are a little bit away from it becoming any form of replacement

[@eugenefritz6669]

@@SeattleDataGuy thanks for your Answer Ben. I asked that because it seems that at Data Engineering there are loads of tasks that are quite repetitive. I want to become a Big Data Engineer but I want it to become a proper investment not a rubbish one of course. Do you have any suggestions on how to adapt to the situation, given the fact that you're quite experienced and more aware of the tasks that can be easily automated and how we future data engineers can envolve and be 'step ahead ' and stay relevant. Because loads of Corporations will want to get rid of many workers. Thank you for your service Ben, you're genuinely the best on KZbin on the matter. You truly add value to people lives and change it.

[@johnnycastillo9139]

So many technologies, so little time

[@khanhgia6817]

Do I need to learn about API like FastAPI?

[@SeattleDataGuy]

you'd don't have to, but it can be helpful

[@xavier2586]

System adminy

[@shadow6161]

--sql--1) postgre up and and running --linux scripting--1) linux bible and last but not list legendary book for data engineers "DATA ENGINEERING WITH PYTHON by PACKT PUBLICATION"

[@my_j.a.r.v.i.s.]

🤣🤣🤣

[@SeattleDataGuy]

why the laughs o.O

[@OGmediahub]

Cool vid ! Dumb question: can a regular guy become a data engineer by self studying ? And if its yes, how much time ?))

[@kmb96]

Do you think it’s feasible to get hired as a data scientist/engineer without a degree after a bootcamp (I’m starting the practicum data scientist bootcamp in a month) or do you think I’d have better luck as a SWE or data analyst and could later pivot into data scientist/engineer after a year or two of experience

[@SeattleDataGuy]

Can it happen, yes, will it be hard, also yes. But once you have work experience it'll be easy

[@kmb96]

@@SeattleDataGuy the first job will be the hardest one to land but I’m sure I can pull it off if my projects can land me a few interviews. My goals are DE/MLE and I’m sure that if someone can give me a shot I can excel in the space, thanks for the reply! I’m also thinking about doing the gcp and aws certificates for DE and MLE, hoping that can also help my case since I’d have knowledge in 2/3 of the big cloud services.

[@nolanaug]

You should consider that some people are new to this world and speak in simpler terms.