Data Engineering Road Map - How To Learn Data Engineering Quickly( By A FAANG Data Engineer)

How do you go from 0 to data engineer?
What is the road map to data engineering?
These are all valid questions that I will walk you through over the next 15 minutes.
The purpose of this video is to help you break down all the various skills we seem to need to know as data engineers in 2021 into manageable bites.
So you don’t feel overwhelmed, but instead have a clear path from where you are and where you need to go as a data engineer.
Also, here is a link to the data engineering roadmap I reference in the video. With links and everything!
drive.google.c...
Wherever you are in your data engineering journey. Good luck!
So good luck with your DE career. And here are some more resources you may enjoy:
What Skills Do Data Engineers Need?
• What Skills Do Data En...
Data Engineering Project Ideas
• 5 Data Sources for You...
If you want to learn more about data engineering, check out Google's Data Engineering Course
coursera.pxf.io/rn1rGv
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy...
Or check out my blog
www.theseattle...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consults on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

[@SeattleDataGuy]

If you want to learn more about what is going on in the data space, then sign up for my newsletter! seattledataguy.substack.com/

[@joshokello52]

sir what about the microsoft certified: Azure Data engineer associate certificate, would you also recommend it

[@socialmediademigod1496]

What is the highest paid tech salary?

[@Zappabain]

God bless you for the info on that Data Engineering Road Map about documentation and Courses.

[@Symmetryful]

I find it interesting that so many of these roadmaps focus so acutely on technical skills. Frankly, I’m not sure how valuable it is to learn AWS to ‘get into’ data engineering. You’ll learn this skill on the job when you get to it. Also this type of technology is highly dependent on what field you end up in. Lots of companies in the finance and insurance sector are nowhere near ramping up to AWS. I think learning the concepts of data storage, ETL pipelines and data warehousing - as well as having a general ability to write code in, say, Python - ought to suffice as a starting point. The hard part about technical roles like data engineering isn’t really writing the programs. The hard part is the softer skills involved and understanding the domain you’re working in.

[@deathbombs]

Agree but some tools are more general, like Kafka, Spark may help with initial getting job offers.

[@prof.mangabhai]

bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part

[@atwertsdfgsdfaasdfa]

@@prof.mangabhai There's such a wide range. SQL is a given. There are DEs that just do SQL and use low code tools. But also lots of DE positions focused on all of the other topics listed in this vid where SQL is a minimal part of the job.

[@atwertsdfgsdfaasdfa]

I moved from DS to DE. My main tool is Python and I really work more as a backend engineer, but with a focus on automating data analytics and some machine learning stuff.

[@ExecutionMods]

​@@atwertsdfgsdfaasdfa I am a cs new grad with some full stack experience. I see a lot of backend engineer jobs desiring desiring kafka, airflow, Cassandra, and even some ML familiarity. It seems like DE skills are the way to go regardless of specific role. Taking the IBM DE course on Coursera and then maybe an AWS specific one if needed; hopefully it'll help me in this entry job rat race.

[@kalydrae]

Don't worry... 15 years into my it career I still write hello world projects for new technologies and languages! Great video thanks.

[@SeattleDataGuy]

Glad you enjoyed it!

[@0x007A]

Learning about containerization from the outset is an important skill. Start by packaging an application such as an API and then perhaps containerize other components or processes such as data acquisition, cleaning, and loading into a database from which your API draws its data. Persistent data storage containers as early as possible will be useful.

[@nickarrigo5540]

great idea! sounds like you know your stuff

[@SD-ko4tz]

Huh? Never used API s in my 30 ywars

[@0x007A]

@@SD-ko4tz APIs have become the primary access mechanism for external data sources, at least for mobile / web apps and a lot of websites. There are other use cases but I won't enumerate them here.

[@prof.mangabhai]

bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part

[@DeezSkates]

Excellent video. The hardest thing about working in IT has been that every time I make a move to further my career, I find I have to build my own road map. Problem is, it’s impossible to know what all exists because there is so much. And every person you ask will give you some different list of things you need to learn, with no real concept of how it all fits together. As the technology expands, the need for roadmaps becomes so much more important than in previous years.

[@LukeBarousse]

Loved this road map that you shared! Also liked how it integrated learning, projects, and then building further. Solid! 🙌🏼

[@SeattleDataGuy]

Yeah! I am really hoping it helps people take time after learning to really apply things. Otherwise that information disappears into some weird part of our brain that seems locked up.

[@NP_Omar]

I just want to thank both of you for paving a roadmap for me into Data Analytics and Data Engineering with your content. You guys rock!

[@LukeBarousse]

@@NP_Omar Aww thanks for this Omar!!

[@axeld.santacruz4659]

@@LukeBarousse I agree with Omar, you and Ben are both amazing (long time follower here :D).

[@prof.mangabhai]

bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part

[@SeattleDataGuy]

If you enjoyed this video, then sign up for my substack seattledataguy.substack.com/

[@stream2learn]

Hey Seattle Data Guy, kzbin.info/www/bejne/e17baalogLpjgtE I have tried to explain stuff that's needed to become a data engineer in 2021 in the above link. Try checking that out as well. Hope it helps.

[@user-fi5oi9vy3s]

Thanks for the roadmap, im working as a developer for five years. Juz turn my mind to focus more on data engineering stuff starting 2022. It will be a good start to come across with the items mentioned in the map. Hope I can be a better man after the effort made. 2022 Wishes

[@SeattleDataGuy]

Glad this could be helpful! Good luck on your DE journey!

[@harveymitchell2113]

Thank you for this roadmap. I have a background in IT and I’m trying to make a lateral move into a jr data engineer role. I’ve been studying for 5 months now with UDACITY, and up until this video I still had no idea how to correctly maneuver into a DE role without feeling overwhelmed. Thanks 🙏🏽

[@SeattleDataGuy]

I do hope this helps you move into a data engineering role! Good luck

[@olajuwonokunubi1869]

Hey Harvey how are you finding the udacity de course , I'm thinking I would like to do it

[@anthonyphan1922]

This guy posted this for 6 days with nearly 400 likes and not a single disliked. That make me definitely give him a subscription

[@SeattleDataGuy]

Thank you so much! I appreciate every like, subscription, etc. I really want to help build the data engineering and data community so I will keep trying to put out great content.

[@CarlosMorenoV]

Amazing!. I am a Python software developer, so I have a background in making APIs and Unit Testing as well as some skills using Docker. But definitively I need to understand how to model Data Warehouses, and some cloud tools and play with Luigi or Airflow. I felt happy watching your video because I realized that I am on the correct path. I have a nice number of tools in my bag, but I have to keep working to get the other ones. Thank you for this great video and excellent road map, mate.

[@SeattleDataGuy]

Glad you found this helpful! It sounds like you're on the right path!

[@LayaWhite]

Thank you for this - it's exactly what I was looking for. I have some SQL skills and I work with relational databases, but I want to make a career shift into Data Engineering. I was starting to get overwhelmed by all the options!

[@SeattleDataGuy]

Glad this could help! Who knew that more knowledge and access to it..would make it more difficult. Hopefully this roadmap helps make the path clearer. good luck on your data engineering journey.

[@ModernPeopleAreEvil]

Have you been learning how to be a DE?

[@kinarapandya2988]

Please continue making videos on Data Engineering. They are very helpful !

[@SeattleDataGuy]

I will keep making data engineering videos!

[@42data]

00:02 Data engineering roadmap is overwhelming with too many skills and tools to learn at once. 01:46 Follow a structured roadmap for learning data engineering 03:24 Build a Flask API to learn how to utilize different skills 04:51 Focus on different paths within data engineering 06:28 Build a project to scrape and store data 07:57 Data engineers build data infrastructure for future use. 09:32 Learning about unit tests, integration tests, and workflow tools is important for data engineering. 11:01 Focus on learning cloud and NoSQL databases early on 12:27 Focus on learning basic databases before delving into streaming and distributed systems 14:04 Prepare for interviews and build a project 15:32 Focus on building a third project and document it. 16:57 Focus on building a solid base for interviews and the first job

[@MahmoudElaskare]

Thank you so much ,I was desperate navigating the diagram shown in the beginning (actually it was hanging on my wall giving me anxiety about actually being able to finish it)👍

[@SeattleDataGuy]

Hopefully it doesn't stress you out too much!

[@knightsavage647]

This is awesome, I'm currently a data analyst and I have been searching for a way to make the switch to data engineering. Thank you for compiling and breaking down the tons of information that is out there.

[@SeattleDataGuy]

I am glad you found this helpful! Good luck on your data engineering journey.

[@elimgarak3597]

Learning DE is like being trapped in vendor tool hell. There are so many tools it is really overwhelming. Thanks for this vid.

[@SeattleDataGuy]

It really is. Just build a strong foundation and then learn how to learn just enough of a tool when you need it. Or... Pick a tool that is not going anywhere soon and master it. Become a specific consultant and charge like $800 an hour. Kind of a joke.

[@essakhan1575]

@@SeattleDataGuy "Pick a tool that is not going anywhere soon and master it. Become a specific consultant and charge like $800 an hour. Kind of a joke." genuine question is this sarcastic? Could you kindly suggest which tool isn't going anywhere soon and to master? Damn I would be happy with $50 dollar an hour nevermind $800.

[@viswajitnayak1665]

Thanks ... i have been working towards transition to a data engineering role and I see this is the most concise path... Thanks seattle-guy

[@SeattleDataGuy]

Good luck, let me know how it goes!

[@DataProfessor]

Wow, this is golden! I think I'm at stage 4. Starting to learn about CI/CD. Yes indeed, writing a blog is a great way to learn! Like number 170 :)

[@SeattleDataGuy]

I am glad you found it helpful! CI/CD is so complex, there are so many layers. Are you focused a lot on deploying models?

[@DataProfessor]

@@SeattleDataGuy At the moment, it’s for updating a website’s content via GitHub. And yes also looking into model deployment as well.

[@SeattleDataGuy]

@@DataProfessor Awesome, for model deployment, are you looking into mlops tools?

[@DataProfessor]

@@SeattleDataGuy Yes definitely, but a newbie at it, learning a lot from your videos Ben 😆

[@SeattleDataGuy]

Thank you! I look forward to making more and learning a lot along the way.

[@PaulMisner]

Thanks. I got thrown into a data engineering role because of a unfilled vacancy. I'm trying to skill up for the position. This was a real help.

[@3dascension744]

I like your friends strategy of guessing what would be asked. That is very clever

[@SeattleDataGuy]

Yeah, it forces you think just one layer deeper. Like what types of questions should i expect? What questions have I seen in the past?

[@DemolaOkesola]

Best guide I have gotten so far. Helps to clear up the confusion

[@SeattleDataGuy]

Glad this data engineer guide helped clear up some of the noise!

[@kocermehmet]

Hi! I started a data engineer internship that week and I find myself here. Your video is great as I think It should help me. I didn't know where to start and how to use my time efficiently so a huge thanks for the video and the roadmap!

[@SeattleDataGuy]

Yeah! I am glad I could provide some signal in all the noise.

[@matthewgin4836]

Very informative. I'm taking udacity's data engineering nanodegree and we're doing a similar roadmap. Good job!

[@SeattleDataGuy]

Glad it was helpful!

[@kphorce]

Finally...you have come back with a.........plan *the rock voice*

[@SeattleDataGuy]

Hahaha! That sounds about right. Gotta alway approach things with a plan

[@moagarwal1999]

Was really looking forward to this. Great work man!

[@SeattleDataGuy]

Thank you! I am glad you enjoyed it!

[@carlobisda]

love this kinda videos, still relevant in 2023. i started a data science course but it wasn't really my jam and i tend to lean towards backend stuff anyways.

[@ridge_413]

Thanks man! I'm a HRIS and Reporting Specialist looking to make a switch to the other Data scene. I found a lot of your videos very down-to-earth and concise to learn more about Data Engineering.

[@SeattleDataGuy]

I am glad you are finding these videos helpful! Good luck on your data journey!

[@LearnAvecAmeen]

Thank you very much Ben, I learned that my ftp/tftp projects will serve for something, Anyway, great video, keep up the good work insh'Allah

[@SeattleDataGuy]

Happy to help!

[@muriloarruda]

Well detailed roadmap. I have 10 years of experience in modeling using Ralph Kimball's techniques and ETL (Sagent, Oracle ODI), where I stopped (2012) completely and moved on to the commercial area, selling technology with spin selling techniques (ERPs, BI, low-code...). Today I want to go back to my initial area, which is now something part of Data Engineering. Watching many of your videos, to understand how to get back to this area that I love so much (or once loved).

[@tushensu]

Thank you for this breakdown, I will update you when I get that Data Engineering job!

[@SeattleDataGuy]

Awesome! Good luck on your data engineering journey. #dataengineering

[@sebastianalvarez4239]

Great video Ben! I am definitely sure I want to pursue a career in Data Engineering but felt like I was all over the place when trying to learn the skills. This is definitely helpful, thank you!

[@SeattleDataGuy]

Thanks for the comment Sebastian! Yeah, it can feel super overwhelming. Luckily when I first started I didn't know as much about the entire field so I just learned things slowly without being overwhelmed. Now with all the courses, articles and tutorials its crazy. Hopefully this helps silence some of the noise.

[@__berichh5375]

Yes, I transitioned to cloud engineering and sooo overwhelming. I appreciate all maps lol

[@SeattleDataGuy]

It can be overwhelming. You just have to take it one step at a time.

[@iamsuzit96]

please more emphasis on the projects. I think it would truly help new learners. thank you!

[@SeattleDataGuy]

I am going to finally try to film the data engineering project video. I am mentally stuck and need to just do it.

[@shaleenarora3006]

One of the best video on data engineering I've come across. Great Work !!

[@SeattleDataGuy]

I am glad you enjoyed this data engineering video!

[@panasrat457]

Thank you so much! I just graduated and now I'm very interested in data engineering field. This is definitely the channel that I'm looking for. Keep uploading!

[@SeattleDataGuy]

Awesome! I appreciate all the support. I have plenty more videos coming

[@0x007A]

Adding those projects to the roadmap is much appreciated.

[@SeattleDataGuy]

For sure! I think its always great to stop and actually apply knowledge. Of course its a struggle to sometimes stop for projects since you will often feel like your learning rate will decreasing but you're picking up so many skills!

[@0x007A]

@@SeattleDataGuy I spent the last few days learning how to containerize and deploy a private git repository service using git and gitolite. It was the first time that I learned how to use volumes (not host file system volumes) but self-contained volumes independent of the host file system.

[@SeattleDataGuy]

That's awesome. Do you feel comfortable with them now?

[@0x007A]

@@SeattleDataGuy definitely. I am now converting the Docker configuration for use with Docker Compose to simplify the deployment of the 3 components. This experience should be useful for data engineering implementations too, but I won't know until I give it a try.

[@SeattleDataGuy]

Yeah it will be! There is a ton of DevOps style work we sometimes have to take on. So that should be helpful.

[@vb_294]

Thanks Ben! This is gonna help someone like me whose between step 1 and 2 a lot and I appreciate all your work 🙌🏻🙌🏻

[@SeattleDataGuy]

Glad to hear it was helpful. Good luck on your data engineering journey!

[@strategy_gal]

Wow this is awesome! You've clearly explained each step on the road map. Great video!

[@SeattleDataGuy]

I am glad it was clear! I need to check out your channel as well!

[@strategy_gal]

@@SeattleDataGuy Of course! See you in there :)

[@JCGarcia1]

Thanks for sharing this with everyone, super helpful as I go through this career transition!

[@SeattleDataGuy]

Good lcuk!

[@Kicsa]

Funny you referenced the first road maps since I saw those and those discouraged me, but your road map is very realistic and I'm excited to finally see something I want to do, thank you!

[@SeattleDataGuy]

Yeah! You don't need to learn everything. Just one step at a time. Good luck!

[@kanimozhimurugesan966]

I like the way you encourage your viewers in final part of your videos.

[@ramprksh555]

its a very informative. Also links given for other technologies to prepare is also much commendable. Thank you so much for sharing all those.

[@SeattleDataGuy]

I am glad you are enjoying the content!

[@AK-Star007]

I know most of the "things" , but it just that nothing is in order. Let me bring it in order. THANX for detailed video.

[@SeattleDataGuy]

Good luck my friend!

[@BBDoesTheThing]

Starting from near zero today!

[@Presssent]

Im a Data Analyst and considering Udemy's course but didn't know about Googles Cloud! Thank you. I applied for a local company and the only Data Engineer is leaving soon :(

[@SeattleDataGuy]

Sounds like a great opportunity to move into being a DE if you wan that.

[@christopheraustin2828]

This is the best and most detailed roadmap for DE that I've yet seen. My question is: Is this roadmap aiming to make you an absolute shoo-in candidate? Is it possible to apply for a job and get one before the end of the roadmap? I'm asking because it seems that the demand for engineer is high and the labor pool is smaller.

[@SeattleDataGuy]

I would say if you do everything on this roadmap you will have a good high level understand of most data engineering concepts. For most interviews I have found that a general understanding of technology(Programming, design, SQL, testing, and maybe a few other odds and ends) will be good enough. Especially if you are going for a more junior position. So You can probably get halfway through this roadmap and start putting out resumes.

[@rhythmcity2006]

Side note, I see The Alchemist on the side. Great book!

[@SeattleDataGuy]

yeah! solid book

[@nickarrigo5540]

awesome content! your channel is such a great resource. really thoughtful comment section too!

[@SeattleDataGuy]

Yeah we have a great community! Welcome to all. Data engineers, analysts, scientists and everyone else

[@TheMrNomadus]

As an UX Designer from IT-sphere I always look at that amount of knowledge I have to learn to be able to become a Data Engineer and become very sad that it's almost impossible to figure out where to start especially when you're 33.

[@SeattleDataGuy]

Hopefully this helped?

[@TheMrNomadus]

@@SeattleDataGuy of course!

[@nimish2568]

Wonderful content .Crisp, Clear & to the point. Thank you

[@SeattleDataGuy]

Thank you!

[@anglamthenhan6579]

Thanks for the roadmap! It helps me a lot. 🤧

[@SeattleDataGuy]

Glad to hear that! Good luck

[@gabyabed]

Thanks! I'm an experienced SQL DBA, but looking to progress my skills and talents, not necessarily change companies yet, but at least find a way to wrangle the data we handle at my current company.

[@SeattleDataGuy]

Thats interesting! Are you looking for specific tools or skills?

[@gabyabed]

@@SeattleDataGuy I guess to keep things specific to my company, we're an Azure shop mostly, and simple data flows to something that can be used by powerBI for example. I knew SSIS, so that's a big piece. I'm thinking the other major thing to learn is data warehousing, that's my missing piece it looks like.

[@SeattleDataGuy]

Makes sense. Yeah, data warehousing is generally that other component. Its funny how you can use tools like SSIS and not even realize they are connected to data engineering or data warehousing.

[@gabyabed]

@@SeattleDataGuy ETL is one of those in-between "glue" products that is not sexy, but when push comes to shove, is powerful. SSIS has a LOT of legs left in it, although I do know the writing is on the wall with Python and T-SQL combined.

[@SeattleDataGuy]

@@gabyabed Forget Python and T-SQL...Snowflake just has tasks that are basically just stored procedures you can put on a schedule + have dependencies...its tech..there is always new but there will always be companies that never switch.

[@shrikantprabhu1788]

Please make a video for interview preparation for data engineering , which sources to use for SQL practice (Leetcode SQL is very basic and might not be useful for data engineering SQL e.g. calculating active daily users etc) , practice data warehouse modeling or how to test a particular funnel etc.

[@SeattleDataGuy]

Interesting. I have a few other videos on my list, but I like the video idea.

[@peekguyy3194]

Fantastically concise video, thank you.

[@SeattleDataGuy]

Glad you liked it!

[@tech-n-data]

Really enjoying your channel!

[@SeattleDataGuy]

Thank you! I appreciate all your comments and support. Got to feed the YT algo.

[@shubhampathak8955]

I LOVE YOU SO MUCH DUDE!!

[@juang_]

If you add some networking to the roadmap you'll have a roadmap to a backend software engineer :D

[@gagangulati7417]

Thanks for sharing the roadmap!!..I have been thinking of moving to data science realm and this roadmap will definitely help. I watched some of your videos where you provided courses/resources for SQL, ETL, Data warehouse, Cloud etc..Could you please provide some courses for Python from Data Engineering perspective (not Data scientist)?

[@SeattleDataGuy]

I will put together a python focused video. I was recently checking out a few courses on sites like data quest and Udemy for data engineers. So I will try to see how these are and report back.

[@Mehrdadkh87]

Is data engineering for you, is awsome post.

[@chrischristi1818]

This is AWESOME ; I'm studying Python, C# and Java right now ; but , I have an unending urge to pursue Data Engineering. Would it be possible to gain employment in the Data Engineering field without any sort of college degree ? Thanks for everything !

[@SeattleDataGuy]

Sure you could get a data engineering degree without a college degree..but it will be hard. Because either you have experience or college to get that first job. If you can get experience without a degree then you will be set..but thats hard to do.

[@rezahamzeh3736]

@seattle Data Guy: It would be beneficial to introduce some materials (PDF, Link, KZbin channel) for each specific skill set. That is the most challenging part for learners

[@rezahamzeh3736]

Especially about networking, OS, containerization, ... that are too broad

[@kazimali07]

This was really helpful thank you so much!!!

[@SeattleDataGuy]

I am glad you enjoyed the video!

[@SeattleDataGuy]

EDIT: Wow...100 likes before 24 hours..thats a first

[@LukeBarousse]

Before 12 hours!

[@SeattleDataGuy]

@@LukeBarousse I can't wait until the day I am trying to get 10k before 24 hours....but for now...this is exciting!

[@SeattleDataGuy]

@@LukeBarousse Add to that...1k views before 12 hours! just barely. Also a first!!!

[@tanayk6146]

@@SeattleDataGuy look at u now!

[@minhson1411]

loved this roadmap

[@SeattleDataGuy]

Glad you liked the data engineering roadmap!

[@DeejayManiaNYC]

Great Roadmap!

[@SeattleDataGuy]

Glad you enjoyed it!

[@ataimebenson]

A good number of the Udemy courses you recommended in the pdf are quite Out-dated, that is a big learning challenge. It would be very appreciated if you review this video in your free time.

[@kingjude-enerie535]

Thanks @ Seattle. This is really explanatory and clear. Please i could'nt find the links for building the flask api or any of ther projects you listed.

[@SeattleDataGuy]

That I believe was just a suggestion. But try tech with tim kzbin.info/www/bejne/fX7Toax3hcd_j80

[@MohamedTarek-vt4lb]

Great outline!

[@SeattleDataGuy]

glad you found it helpful!

[@annaphamclouds]

this is such an amazing video. thank you.

[@SeattleDataGuy]

Glad you liked it!

[@madelineflores3534]

Thank you so much for this video!! 🙂

[@SeattleDataGuy]

Glad it helped!

[@Ravit20]

This is top class!!!!

[@SeattleDataGuy]

Thank you!

[@iakobkv271]

I think a video about the Data Mesh would be interesting from you

[@SeattleDataGuy]

I have thought about it for sure

[@bepractical9383]

looks like it's extremely hard to become a data engineer. kudos to ur efforts tho ;

[@SeattleDataGuy]

its totally possible!

[@viniciusluiz8833]

Great video!!

[@Viper-sn5cx]

Awesome video man thanks!! Do you have a specialization in any of those skills/tools?

[@jaysharma5822]

Hey Ben! Thank you for the vide. It's crisp and comprehensive at the same time. You mentioned that you'll be sharing resources/links to learn/explore flask api. But, I can't find the link anywhere. Could you please pinpoint the link? Thank you again!

[@brianbrophy1115]

It's in the Google drive link in the description

[@woobulbul]

Hey, thank you so much for this information, I have a few questions if I may: 1. I am a CS major, with no previous experience in this field, do I need to pursue my masters as well? If so, what master should I pick to suite data engineering? 2. As a fresh university grad, can I land entry level job in this field?

[@SeattleDataGuy]

In general, most technical roles like software or data engineering don't need masters degrees as long as you can pass the interviews. Generally, its a good idea to get an internship in a DE position and then get a returning offer. That way you can get experience. There are jr. positions, but I noticed...as per usually...most of them want 2 years of experience.

[@ministerboan]

Good information.... thanks a lot.

[@SeattleDataGuy]

you're welcome!

[@oluwadamilolajesutofunmi1358]

Thank you for this video. Is it important I get a laptop n which one should I get. I'm just getting started today

[@SeattleDataGuy]

i like my mac...but as long as your computer 8-16gb of ram you'll probably be fine..There are for sure some engineers out there running circles around me on 2 Gbs of ram in vim

[@montgomeryscottbrea2614]

Excellent content.

[@SeattleDataGuy]

Glad you enjoyed it!

[@tebogoawotunde4373]

Great road map. What's your take on the Udacity Nanodegree program: How to Become a Data Engineer. Is worth having a look at for someone with no prior experience?

[@ramielkady938]

Good point about building your own APIs so you would know how to use them. Do you recommend we build our own OSs, Web Servers, and our own Apache Sparks so we can get the hang of it as well ?

[@NVnun]

A good comprehensive methodology, have found this pretty useful as I strated out on my DE journey a while back. Can you please recommend a good way to develop strong sql skills ? I dont do a lot of data warehousing in my current DE role and also we dont have a traditional data lake so my prototypical ETL experience with SQL is low and I see most DE roles in the industry today test for intermediate to advanced sql.

[@SeattleDataGuy]

What do you do in your day to day?

[@gtang31]

can you make a video on different ways of building ETL infrastructure and scaling it

[@SeattleDataGuy]

Possibly, this would be far down the line though. I often recommend people try to look for services to scale their ETL infra. Like with Airflow, its probably better to use Cloud Composer or Astronomer vs try to manually scale.

[@nivretech2072]

Hi Ben! Thank you for the roadmap. I am torn in choosing whether to go with being a mobile engineer/developer or a data engineer. If you are in my place what will be your decision and which do you think will be more in demand in the future with these two? Thanks again!

[@SeattleDataGuy]

Thank you for the comment. I imagine both will have a similar demand. I would say you should take a few courses in both and build a mini-project and see which area you like more. Programming and data in general will be a necessary skill in the future. So I wouldn't worry about the perfect choice. Instead, focus on which area you enjoy.

[@mushroomvv]

The ending makes my cry…

[@SeattleDataGuy]

why :(

[@prof.mangabhai]

bro I like to code , is data engineering for me ? because I have heard that DE mostly write sql when it comes to programming , and have they might some automation scripts for aws with python but that's not very often and is not a large part

[@SeattleDataGuy]

It depends where you're working at what role. You might prefer a role like Software Engineering, Data these roles tend to be focused on building infra to handle big data

[@behrad9712]

Thank you very much!🙏

[@SeattleDataGuy]

You are welcome!

[@ukaszdugozima816]

Hello! Thank you for your invaluable video! I find it extremely useful for beginners! I would like to ask about one thing regarding this map. I learnt Pandas in terms of Data Wrangling and Transformation. Therefore, how about Pandas for Data Engineers? Is it useful tool for ETL/ELT transformations? Obviously, the next step will be PySpark, but I would like to start learninig Pandas. It seems it is a good path for the next one. What do you think about it ? I would appreciate it if you could share your views about it.

[@priyankpant2262]

Hi, Thank you for this video. How much time do you think it would take to complete all the steps mentioned here ? I work in IT but have mostly worked as a consultant on the ETL tools like Informatica etc. I know about Data but Data Engineering is somewhat new to me.

[@aag3298]

Do you think getting the AWS Data Analytics Certification is a good idea and it will help in this road to become a Data Engineer? Also, do you have any source that works as a guide to learn how to document or give proper structure to the projects in github that will serve as portfolio? Your content is great by the way, greetings from Costa Rica.

[@SeattleDataGuy]

I though I responded to this one. I would say focus on getting data engineering certificates. With AWS you could get the big data aws certificate. Of course, that also depends where you are in your technical journey. You can use some of felipe hoffas pieces of work as an example. I am working to get him onto the channel so he can talk about his projects. Here is one of his projects hoffa.medium.com/400-000-github-repositories-1-billion-files-14-terabytes-of-code-spaces-or-tabs-7cfe0b5dd7fd#.qm2s97y25

[@gunnersshishir11]

Thanks..this is amazing

[@SeattleDataGuy]

Thank you for the comment! This comment is amazing.

[@himayata9600]

Hey! I am currently going into my third-year of undergrad majoring in CS, found your page and now am interested in Data Engineering. I am beginning the UDemy A-Z SQL and Database course you recommended in another video to get a taste of data engineering. If I end up liking data engineering and want to take the next steps in this video, do you think I can complete at least up to step 7, 8, or 9 by the end of August if I keep my head down and grind? Currently have a Data Science/Analyst internship at a small company for the summer, but want to pivot and secure a big Data Engineering internship for next summer by being prepared for interviews in the Fall/Winter. Thoughts?

[@SeattleDataGuy]

If you have time perhaps. But truthfully it sounds like you have an internship and you should focus on that for the summer. Maybe you can intern at the company again next summer and ask them to let you try out DE.

[@simoneintech]

Thank you

[@SeattleDataGuy]

you're welcome!

[@angelocello9967]

legend, thank you :)

[@SeattleDataGuy]

Glad you liked it!

[@aleramos9149]

Excellent!

[@davidr9035]

Is it true that I *must* start w/ Front-end because companies seldom hire junior Back-end developers/ engineers.

[@SeattleDataGuy]

No, I know plenty of people who got back end jobs out of college. At least in the US

[@akshitmalik7973]

hi ,Can You share the latest tech stack whenever you are a little bit free. And also if possible, please share links to all the resources. that would be of great help . thanks.

[@RatFace_MonkeyEar_FishEye]

Hi! Thanks for the video. I am a beginner DE myself and would like to ask a question about the Kimball’s books you recommended. If you had to pick only one of the 3, which one would you get and why?