What do DATA ENGINEERS do? Is data engineering a good career choice in 2020?

***
My FREE guide on how to become a data engineer:
karolinasowinska.notion.site/...
***
I currently work as a data engineer and I'd like to tell you what data engineering is all about! It's commonly confused with machine learning, data science etc., when in fact data engineering is quite a distinct area of technology.
So, what do big data engineers do?
The job revolves around bringing new data to the company (ETL process if you like) and then maintaining the data flow!
Is data engineering a good career choice in 2020? Long story short - yes. As the amount of data we produce is growing rapidly, so is the demand for data professionals. However, my prediction is that many of the current data engineering tasks might get standardised, and eventually automated in the future. But hey, this is the XXI century! We'll have to learn new skills and adapt all the time!
-------
If you'd like to learn data engineering, I recommend following the 4 simple steps below to land you the first job interview:
In this long-awaited Airflow for Beginners video I'm showing you how to install Airflow from scratch, and how to schedule your first ETL job in Airflow!
We will run the Spotify ETL job that we created in my Data Engineering Course for Beginners. But if you have your own code to schedule in Airflow - feel free to use it instead!
In this Airflow tutorial I just wanted to get you started with Airflow, because I know that configuring Airflow can be non-trivial. I won't be going in-depth into all Airflow's features, so hopefully this tutorial will be as beginners friendly as it can get!
I'll show you how to fix some installation errors, how to set up your project folders, how to edit Airflow config file and where it sits, and how to write your first DAG file!
Get ready for a ride!
Oh, but grab some useful links first:
-github code repo: github.com/karolina-sowinska/...
-Airflow quick start guide: airflow.apache.org/docs/stabl...
-Spotify access token: developer.spotify.com/console...
Don't forget to subscribe if you're new here, more content is coming!
:)
---------
If you'd like to learn data engineering, I recommend following the 4 simple steps below to land you the first job interview:
1. Learn Python
I recommend following the Python for Everybody specialization course on Coursera, which is one of the most popular courses there:
imp.i384100.net/x9gVO3
2. Learn SQL
SQL is still the lingua franca of data. I recommend going with Learn SQL Basics for Data Science course, because it contains some chapters which are very releavant to data engineering in partcular, e.g. distributed computing with Spark
imp.i384100.net/QOMZ09
3. Learn Bash scripting/Linux
I wouldn't take a full course on it, but at least read a good article.
if you do prefer to take a course/guided project, I think this one is short and good:
www.coursera.org/projects/com...
4. Learn how to develop on the cloud, e.g. on AWS
There are a few good courses around there, but I think the Coursera one is the most comprehensive
imp.i384100.net/P0MJBM
Discloure: The above contain affiliate links, meaning when you click the links and make a purchase, I receive a commission!
-------
Find me on Instagram: @karo_sowinska
And if you want to make my day with a cup of coffee... :)
ko-fi.com/karolina_sowinska

[@karolinasowinska]

I've just released my Free Data Engineering Course for Beginners on KZbin! Make sure to check it out if you're interested in a career in data engineering! :) kzbin.info/www/bejne/mqfZmnyLgNx7bdE

[@karolinasowinska]

@Profound Programing I don't, but you can also find me on Instagram: @karo_sowinska :)

[@karolinasowinska]

@Profound Programing I'll consider setting it up at one point! :)

[@annmaryjo3803]

U r really amazing to starters. NB: I am just studying in 9th standard 😅 just checking how it works so that I don't have to look for this in more big way😁

[@karolinasowinska]

@@annmaryjo3803 Thanks, and good luck on your journey! :)

[@giljav]

What is your background and how you got into data engineering?

[@patrickd4468]

I am Data Engineer and I describe myself as follows: "I am like a plumber in the data world and ensure data flows from one abstract system to another in a scalable, consistent and standardized manner to allow the business users to consume the data in a usable way" . My only advise is to ensure all your pipelines are robust and fault tolerance/notifications are built into your data layer.

[@karolinasowinska]

Absolutely, a data engineer should always consider as many edge cases as possible, because they happen more often than not! I'm glad to read a comment from a fellow data engineer! ;)

[@mauricewalshe8339]

And in finance make sure the pennies add up :-) I used to work on billing systems for telecoms multiple types of records multiple types of data sources - we used Map Reduce (back in the early 1980's ) with F77 and Pl/1G

[@sumittamang3548]

If you had to suggest an undergrad student between data science and data engineering, what would you suggest?

[@anyasyht]

As a data engineer I couldn’t agree more with you!

[@vsevapopov2]

So great to have such video lessons on youtube! I had an internship 2 and a half years ago where I did the data engineering task and now I want to continue my journey!! Now I study at university and I have much time to do that! Thank you!!!!!

[@peterndiforchu2622]

Love it! How comes I'm just discovering you. Love your sense of humor. I'm working on the path of becoming a data engineer. This is encouraging as all the 5 points are things that excite me. Keep sharing great content. We'll keep supporting.

[@victorborges5291]

Knowing full well each job title is like a passing show these days, without doubt, I'm inspired! 👍

[@dianerocher6705]

As a recruiter trying to understand job queries, this is gold ! Many thanks

[@karolinasowinska]

My pleasure! :)

[@MisterNorthernCanuck]

I just got myself a job as a data engineer as my new career challenge. I appreciate your content, it essentially confirms what I'm getting into :)

[@mokshgrover9517]

I wanted clarity on this profession for a while now. Nobody could explain this in an easy way. You did. Thank you! Very informative.

[@karolinasowinska]

My pleasure! :)

[@bhuvaneshrao6322]

This is one of the very good videos i have seen on data engineering, thanks.

[@karolinasowinska]

Thank you, that means a lot. The saga is to be continued very soon!

[@yooalejandro386]

That's amazing! Your speaking is very logic and concise, covering a lot of the most important concepts or sessions a data engineer does everyday. A very talented and graceful lady. Thanks for sharing and god bless you.

[@karolinasowinska]

Thank you so much! I'm glad you like my delivery! :)

[@lfrqz]

Your vlog feels real, honest and accurate. I am learning about data science, sql, python, etc. and I find your content very inspiring. Thanks for sharing your knowledge and experience. I am looking forward your future content. Greetings from Mexico.

[@karolinasowinska]

Thanks so much Luis, I'm glad that you're enjoying my videos! :) Good luck on your data science journey!

[@chrisv8341]

This is extremely clear and concise! Thank you for sharing Karolina :)

[@karolinasowinska]

Thank you! :)

[@Robi2009]

Komentarz dla zasięgu ;) Powodzenia w rozwoju kanału!

[@karolinasowinska]

Dzięki! :)

[@samguy7209]

Thank you for sharing your experience and the great description. I love your accent. You have helped me decide my future career. ❣️

[@karolinasowinska]

That's wonderful, I'm glad my video was so useful! ;))

[@ryunosukefuriya3748]

Super glad I came across your channel, Karolina! I'm currently a DA working to make some long-term transitional goals come true! -> data engineering -> data science. Thanks for your awesome insights! Your delivery is really engaging and easy to understand! Best wishes, Karolina!

[@karolinasowinska]

Thanks, I'm glad you enjoy the videos! :) And your career transition plan sounds great, good luck on your journey!

[@exxodduss2008]

Perfect description

[@hypebeastuchiha9229]

Thanks for this video I’m happy to see an actual data engineer give her opinion unlike bigger cs career channels that just read articles

[@md.shahriarrahmanrana6692]

You helped me a lot Karolina. Continue to share your experience, so that we can learn from you.

[@karolinasowinska]

I'm glad that the video was useful! :)

[@caassimbah2485]

This was so helpful and informative. Thank you!

[@claudiusandika5366]

wonderful insights!! keep creating :)

[@karolinasowinska]

Thank you, will do! ;)

[@CarlaAnneQ]

thank you for the insights! = )

[@kevinle5772]

I like your clip, very well articulated. It's like you are reading my minds.

[@karolinasowinska]

I am a mind reader indeed ;)

[@jmrodrig00]

Hi Karolina, thanks for sharing your thoughts about the subject, I starting a career as a data engineer and you video clarifies me and motivate me to do it as quickle as possible. Best regards.

[@karolinasowinska]

Good luck on your journey and I hope to see you around here! ;)

[@splashoui3760]

Was a good video. Thank you for this one!

[@weirdinventor]

thank you so much , keep up !

[@juanluisduquelafuentes1823]

Thanks for another interesting video Karolina! You know...as I was watching the video and hearing you talk about the different tasks data engineers have to perform, I kept thinking about the inevitability of all of those processes getting automated eventually. And of course you concluded with that interesting tidbit about having to learn new skills because you anticipated automation. Just goes to show, not only do you have to identify new opportunities and learn new skills in order to keep thriving, but you also have to develop a strategy to become as financially independent as possible so that you’re not caught all of a sudden in a situation where your skills just cannot support a decent wage anymore. If that doesn’t motivate someone to get off the daily grind for a few hours a night and just sit down, make new plans, pursue other interests, and just think about the meaning of it all, I don’t know what will!

[@karolinasowinska]

My thoughts exactly, Juan! What you said about financial independence in particular. Sadly, in the face of automation, it is those who are not financially independent, or whose skills are not sufficiently transferable, that will be hit the most.

[@Techkhamun]

Very interesting! Keep up the great work!

[@karolinasowinska]

Thanks so much! :)

[@ngee4925]

This was super helpful, thank you so much!

[@karolinasowinska]

My pleasure! :)

[@geoffreyreceveur4605]

Thank you for the video. Very clear

[@karolinasowinska]

My pleasure, I'm glad you found it useful! :)

[@vineetjainorg]

Really liked your video! I work as a Data Scientist and can totally relate to the hard work our data engineering team puts in. Thanks, keep making more videos!

[@norpriest521]

Can you let me know what is the career path for data engineer? Say I don't want to work as data scientist or machine learning engineer, I only like to do data engineering my my entire professional career. Can one go far just by working as data engineer only and doesn't move to other areas like ML or DS on the later stage?

[@karolinasowinska]

Hi, absolutely! Data Engineering career path, like all career paths in technology, can take many shapes. But in majority of cases, you'll start out as a analyst/developer (building, fixing data pipelines), and then you will move on to assuming more responsibility, e.g. designing the entire data architecture of a company or managing teams.

[@lovemaakhe]

lol the shade in calling data scientists 'not-technical folks', great video!

[@DataAnalyticsIreland]

Great explanation of this role, thanks for sharing!

[@karolinasowinska]

Thanks! :)

[@DataAnalyticsIreland]

@@karolinasowinska you're welcome!

[@skiddi_botti]

Welcome to my world! Best of luck 👍

[@karolinasowinska]

And same to you! ;)

[@kubasochacki9599]

Przydatny film dla studenta I roku Data Engineering!

[@karolinasowinska]

O, super! Fajnie, że znalazłeś taki konkretny kierunek, na większości uniwersytetów jest tylko Computer Science. Powodzenia! :)

[@newaccbub]

Super helpful!

[@qasimzaheer1140]

Very very good video! Awesome.

[@karolinasowinska]

Thanks! :)

[@codingevelynmiami4230]

Appreciate your explanation.

[@karolinasowinska]

I'm glad it was helpful! :)

[@matheusmeirelles4189]

Thx! That's worked very well to me, cause I'm already work with data but I didn't know what I was lol.

[@spideken123]

gorgeous as always miss karolina. very imformative.. thanks dear :)

[@karolinasowinska]

I'm glad you found it useful :)

[@adriang.4839]

Thank you!! I just entered university in the career of data engineer, your video helped me clarify certain doubts

[@karolinasowinska]

I'm glad you found the video useful! Good luck at university! ;)

[@divyanshshrivastav2469]

Hi, i also want to study data engg in an university but didn't knw there is any good universities and procedures for it.. If u can send me an email regarding that knwldge will be helpfull..... [e-mail]

[@longinteger]

Just started my career as a data engineer in a professional environment as well and I can align with everything you say, Karolina! It's also very interesting because many people think Data Science is just that one single field but it can be divided in so many sub ones. I think you make the role of a DE clearer in this video! Great work. I am very excited to get to the same point and share the same experience with you.

[@karolinasowinska]

Hey, thanks so much! I'm glad to hear that you've embarked on your DE journey, I'll be curious to hear how you're finding it! :)

[@norpriest521]

How many tech stacks you need to have to become junior DE? Like you need to know Python, SQL or smth else? How many prerequisites should we have before go on to become junior DE?

[@sajarnitsa]

Thank you very much for this video. You could believe you are terrible helping other people installing software and things like that, but your videos are the most honest and down to earth description of the job of Data Engineering You are amazing.

[@karolinasowinska]

I'm so happy that I could help some people with the videos! Thanks!

[@JorgeEstradaMty]

Hello, I come from the @CodinEric channel. Your content is very interesting too, thanks for sharing

[@karolinasowinska]

Thanks and welcome! ;) Sorry for the delay in responding, I've just noticed your comment!

[@jaimecortesguevara5908]

I really felt that part when you mention to "adapt" for the next 5-10 years, because that's when you need to keep investing in your self. Thanks for sharing your opinion

[@karolinasowinska]

Absolutely, we will change jobs more frequently than people did in the past, so one needs to expect that and learn to be more adaptable! :)

[@ahsanmohammed1]

Great video! Good info! Sincerely done! Appreciated! Thank you! Liked! Subscribed!

[@karolinasowinska]

I'm glad you enjoyed watching! :)

[@mytechchannel2519]

Useful information, thanks🌹

[@karolinasowinska]

I'm glad it was helpful! ;)

[@epicwhat001]

1. Bring Data in from different vendors. Check 2. Data pipeline , data warehousing , and aws. Check 3. Table-valued function , Procs and Views. Check 4. Analytics, debug and Monitoring. Check 5. I will Help but Not going to be nice about it. Check Thank you this gives me confidence to change my career to data engineering . I Am not a Data Engineer but a BI developer. my company is kind of like a start up so we don't have a data engineer so I ended up doing all the pipeline to create my SSRS and Tableau report.

[@karolinasowinska]

Sounds like you're already doing data engineering! Awesome stuff ;)

[@ReenanOFC]

Awesome video.

[@karolinasowinska]

Thank you! More videos on data engineering are coming soon :)

[@yash1152]

11:46 - 12:19 - 12:37 - 12:57 this part of the vid is the one i liked the most. not that it's something i already didnt had the idea of, but it felt nice to get reminded of the upcoming highs.

[@buithanhlam3726]

This is very helpful!!!

[@karolinasowinska]

I'm glad you found this helpful! ;)

[@fbalseiro]

Great content! It's funny you said that the thing that you like the least is to help others and here you're creating videos to help others on how to become a data engineer. Plot twist. 😅 Nice job Karolina 👏

[@roopaljain1390]

From a women in tech to a women in tech, great vdo. I can see the spark in your eyes mentioning your fav part.

[@karolinasowinska]

Thanks! Great to hear from a fellow woman in tech! :)

[@ssbrunocode]

Thank you for the good content! You are awesome o/

[@karolinasowinska]

I appreciate that!

[@farwanqv]

A good one to explain what data engineers are supposed to do

[@karolinasowinska]

Tried my best! :)

[@luisalamo2658]

Thanks a lot from these kind of contents about Data Engineering, I'm a Java developer and I'm looking for conquer a new boundary because I'm a little bored doing micro-services and I want to try some related new topic. Greetings from Chile.

[@karolinasowinska]

I'm glad this was helpful, and best of luck! ;)

[@vitelinha123]

Apache Airflow looks like a good fit for this sort of pipeline, really powerfull

[@shaileshxjain]

Karolina, You are a pure heart 💓. (Evidence at 11:18) Keep uploading videos and share us your experience about Data engineering and it's prospects. May you achieve a Million Subscribers soon. Kudos, from India ✌️✌️

[@karolinasowinska]

Aw, I was just honest! Thanks so much! :)

[@victorpinasarnault9135]

2021 and still worthy.

[@karolinasowinska]

Not much has changed since I recorded this! :) Perhaps the accelerated move to the cloud!

[@muhammadsaad907]

Thank you so much :)

[@statisticianwordnpl4071]

Really interesting video! I Don't Know why in my country there aren't girl that talk about it carrers. 😅 Cya from Italy!

[@karolinasowinska]

Thank you! Tech careers seem to attract guys a bit more than girls, but I think it's slowly changing now, and many girls do coding! :) Ciao!

[@harshavardhan4523]

Wow what a great video.....It was funny even thank you

[@karolinasowinska]

Thanks!:)

[@bryanortiz9753]

Beautiful, funny, smart and data engineer like me! IM IN LOVE!!🥰

[@TheCaesius29]

i'm thinking of a career shift to a data engineer. thank you for this :)

[@karolinasowinska]

My pleasure, and good luck! :)

[@freemind.d2714]

Always pleased to see experienced people talk about they work and life. According to this video, I guess your work is much more focus on the pre-process the data rather than implementing and training the model right?

[@karolinasowinska]

100% true :)

[@Yok685]

hahah "each data vendor, they are not nice people, they want data in certain format" hahahah true! I can relate so much

[@karolinasowinska]

Totally! haha

[@badass_omelette5166]

I have been enjoying data engineering and cloud engineering over data science recently

[@yorkstbd]

I'm as a junior BI developer, studying and working with tableau, and I'm missing the programming. I'm going to start working on some projects on my own and see if that's really what I want to do, because of everything you said I liked it so much! The bad part is that where I live there is only Senior vacancy for data engineering

[@karolinasowinska]

Maybe some junior positions will pop up soon! Definitely go for what you enjoy!

[@yorkstbd]

@@karolinasowinska

[@markslima1557]

very cool thanks for posting - i'm learning about data analytics/python/sql wondering if data engineering is a better fit with my IT background

[@gracechan3039]

I work for an IT consultancy that primarily migrates businesses databases to the cloud. The quantity of work seems endless.

[@karolinasowinska]

100%, I can imagine that, it's a very trendy thing to do at the moment!

[@akeemadedigba8279]

working on cloud computing makes life easy and flow

[@denismerigold486]

being a data engineer is cool. But deep learning is much more interesting. Thank you!

[@karolinasowinska]

I agree that deep learning is more intelectually challenging!

[@norpriest521]

Can one work as DE the entire career without having to move onto other relevant areas like MLE or DS later on? Say I love DE job so much, can I stick to it forever? What will be career path for DE? I don't like machine learning and data scientist role 😂

[@karolinasowinska]

@@norpriest521 Hi, yes, you can! Typically the career progression is such that you eventually become the person responsible for the overall data flow in an organisation, eg. the Head of Data Engineering.

[@seb6226]

Is it possible to create a video(s) about data scientist vs data analyst vs data engineer? Or data jobs vs machine/deep learning/ai vs computer science jobs maybe a series of videos? Could u do a video about how much ai/machine/deep learning could impact jobs and which jobs will likely become automated or disappear vs which jobs will remain strong in the next 5-10 years? Also based on your last name, its great seeing someone else from a Polish or European background!!

[@karolinasowinska]

Awesome suggestions - noted, they land on my list! And yes, you're right, I'm Polish! :)

[@daringvishalverma7983]

Thanks a lot. You explained everything very well.I'm starting my career as a Data Engineer, so wish me luck.

[@karolinasowinska]

That's amazing, good luck! :)

[@logic9948]

9:40 - that smile hit me, you are so beautiful ❤

[@ihgnmah]

Hi Karolina, your content is amazing and informative. I just curious that what to learn to get started with data engineering. Thank you

[@karolinasowinska]

Thank you so much! I'm soon going to release a series of videos on data engineering, so stay tuned! :)

[@ihgnmah]

Karolina Sowinska That sounds interesting, Thank you

[@Anandhusreekumar]

Thanks for this video.entire KZbin is full of data science videos.im looking for data engineering channels

[@karolinasowinska]

Here is one! ;) Thanks for stepping by, and if you've got any video suggestions, please do let me know!

[@blah-blah-blah4715]

To your last point about standardization and automation, I'd honestly love if Data Engineering and ML Engineering came closer once this happened. DE's already have to deal with all the requests from MLE's, might as well just put them in the same room. I see both getting closer to being abstracted away to standardized tools within the broader software engineering ecosystem. More successful and matured design patterns just need matured first. But once this happens, I very well could just see software engineering jobs focused on ML and/or Data experience being the new norm.

[@medisettycharan9152]

Many thanks for the information, it’s really helpful. Can I please ask you with whom will the data engineers will be interacting most of the time. And what kind of information will be exchanged.

[@karolinasowinska]

My pleasure! Mostly with data scientists and data vendors:)

[@ErickHernandez-ri2ft]

It's great to know more about my degree (I am data enginerring student, I am in sixth quarter) and I think, oh my gosh, I need to learn a lot of topics because my teachers don't teach me but I am learning on my own. So, the question is: What are the main topics as a data engineer needs to know? and can you recommend me any page to learn about data engineering? More specific topics such as: Big data, distributed systems and data mining. Greetings from Mexico!

[@DAJakaRedAries]

Thank you for th enice video. Subscribed :) I'm wondering if data engineering job is more demanded comparing to data analyst jobs? Do you need to learn Scala in order to manage ETL pipelines? (I know Python/SQL already)

[@karolinasowinska]

I would say that data engineering is in more demand. You don't necessarily need Scala - the particular technologies depend on the company that you'd be working for. Python + SQL is definitely a base. I'd recommend learning about data modelling rather than Scala, because it's more universal :)

[@irishuang5294]

Thanks for the video and it's rly nice to see another female data engineer voicing their thoughts. I've been a data engineer for officially, I'd say 1.5 years and I really enjoyed the net result of ups and downs. Data engineers understand business better than software engineers, and understand tech better than data analysts/scientists. So to me, when I look at it as a role in translating data for business, I find it quite impossible to be obsolete in the next 5-10 years UNLESS the average data literacy within a company has increased (making better and more careful choices on data inputs) or people turn to machine-generated data more than human-generated data (or when NLP and Deep Learning married for good to successfully "guess" the right data). Otherwise the business would always need to pull in a data engineer to scale their data while preserving its correctness.

[@karolinasowinska]

Hi there, nice to meet a fellow female data engineer! :) That's a very interesting and convincing perspective - perhaps you're right (and I definitely hope so!)

[@davidthewise6617]

@@karolinasowinska You look beautiful 😍😍😍

[@sligon00]

@irishuang A very insightful comment , thanks for being so clear and concise about what goes on behind the scenes in the world of data.

[@cosmosbrasil5403]

Half a year? You know so much apparently for that period hahah. Anyways, I recently got my Physics degree and worked with Python in a few coursework and final year project. Next, I'll do a Masters in London but intend to start my career in the 'big data world' and I definitely consider data engineering (but mostly data science :D ). That video was quite helpful (although I don't know 10% of all programs you've mentioned lol ). Anyways, thumbs up! Thanks Karol. Ah, I hope I never need your help to install a driver or smth ahhaha

[@karolinasowinska]

Physics, nice! Awesome foundation for data science! Oh, I also did a Masters in London - have you started looking at which unis you'd want to apply to? P.S. Data Science is probably more interesting (if you like statistics more than technology/coding) P.S.2 I also hope you won't need me to install drivers 😂

[@cosmosbrasil5403]

@@karolinasowinska I already applied to 3 unis, I was lucky enough to have 3 offers from Queen Mary, Imperial and UCL. I'll think I'll stick with UCL, although there's still a small chance to change my mind. I'm glad to know that Physics is a good foundation. I love almost everything related to mathematics, coding, statistics, etc (a true geek, ahah), but have no experience in data industry : / a bit scared of not finding any job. About P.S.2: if I do, I prefer to ask you to make a video about it ahahah. Thanks Karol.

[@karolinasowinska]

Wow, congratulations, that's a lovely set of offers! I'm an Imperial graduate myself - if I can give you my biased opinion, it was a very good course (but also very, very tough). Don't worry, the industry loves geeks haha, you'll surely find a great job! A video about drivers you're saying... I'd give this one a long think! 😅

[@cosmosbrasil5403]

@@karolinasowinska ahahaha is better doing vids about your relationship with Apple ahhah. Imperial graduate, uow. Congrats!! I hope to go well through this toughness and start working soon (eager to learn :D ). Have an amazing evening. :D :D

[@RobertoMartinez-sx6ih]

You look a lot like Jodie Foster. thanks for the video, it was great.

[@karolinasowinska]

Haha I've heard that before (many times!) Thanks :)

[@troyfrericks-wti2510]

Would love to hear your opinion about what "Data Architects" do.

[@karolinasowinska]

Very similar job to what Data Engineers do, but I guess Data Architects are typically higher up in the hierarchy/they're more experienced, so they can decide on the architecture, and not just build pipelines

[@candyfloss184]

Suppose a company wants to have an erp from oracle to sap, now past data is in oracle db, future will be in sap db, now CEO of the company would like to have all data in same format report to make a year on year comparison, then you would need data engineers.

[@karolinasowinska]

Definitely! Data engineers are useful people to have on board! :)

[@candyfloss184]

Try to learn sap BW if you are so much interested to work in data engineering. But you could gain lot more insight if you would understand how business works. But then you donot appreciate economics much. So I think technical role is better for you.

[@karolinasowinska]

@@candyfloss184 Economics has little bearing on reality. Thanks for your suggestions though.

[@candyfloss184]

Karolina Sowinska I have always like macro economics. Microeconomics sounded a bit technical and it was difficult. Development economics and international economics too were interesting to me.

[@datawolf2318]

I'm an unspoken data engineer lol. I didn't know what ETL was...read about it and realized I have experience in it lol.

[@karolinasowinska]

Haha I can completely relate - I wrote my Master's thesis on this topic, and only later did I realise that was ETL!

[@karolinasowinska]

@Jared Dudley or data engineering specifically! ;)

[@jae11011]

Thanks a lot for the intro! It helps a lot for me as I am thinking of making data engineering as my career goal. For the last part, if data engineer becomes less demanding in the future, may I know personally which way would you recommend a data engineer to go, software engineer or data science?

[@karolinasowinska]

I believe you should go for what's most suited to your personality and interest :) Engineering vs science/research is the key question here.

[@sumittamang3548]

If you had to suggest an undergrad student between data science and data engineering, what would you suggest?

[@ofekezra7567]

Amazing video. simple and straight to the point. Karolina can you please explain if degree is needed to became a data engineer ? and if yes - what can i learn in home before (which skills)

[@karolinasowinska]

Hi, thanks so much! A degree in a technical subject is a plus, but not a strict prerequisite. I've just released a free data engineering course here on youtube, so if you want to learn some relevant skills, check it out :) kzbin.info/www/bejne/mqfZmnyLgNx7bdE

[@ofekezra7567]

@@karolinasowinska thank you so much

[@FrancoisLeCoguiec]

Karolina: Subscribed! Great video, really nice and simple down-to-earth explanations! One point though: I think at one time you talk of HADOOP and you seem to imply that HADOOP has to be on premise? Technically I think that is not true. So, your company might have their own on-premise HADOOP cluster(s), but it is also possible to get "HADOOP as a service" in the cloud, on AWS or other cloud providers. Still, that makes no big difference is what you explained, since the activities you described apply no matter if on premise or in the cloud, but I thought I would point it out.

[@karolinasowinska]

Yes, you're right, I only later realised that Hadoop might as well be done in the cloud! Thanks! :)

[@irishuang5294]

Is there an increasing or existing trend of hosting Hadoop on cloud? I also feel Hadoop tends to be an on-premise thing. My company actually has been trying to migrate from on-premise Hadoop to Azure

[@Fruitslycer]

Probably gonna have to merge Data Analysis, Machine Learning theory wise like MSc paper reading, implementation wise, like X amount of frameworks, C++ production code, and Data Engineering Pipeline/MLOps, into a single title, and know all those stuff, to have a job in the future. I feel like the market has been a lot softer on older guys, and us young new ones are gonna need to raise 3x of their for same salary (or less). Also older guys are far away from being nice to young ones imho, and the technical debt will become greater and greater too!

[@karolinasowinska]

Perhaps that's true. One benefit that the younger generation of developers have is knowledge/information availability!

[@omkarbhiwande7723]

Can you make a video on how should one go about learning data engineering?

[@karolinasowinska]

That's a highly requested video - I think I'm going to make it very soon! Stay tuned! :)

[@YukoFM]

An apology, I don't speak English. I found the video interesting, the explanation and finally reflection that although the career is currently well paid, things may change in the future.

[@aneeqbokhari4611]

Hi, I just finished my MSc course in Software development and I'm thinking starting a career in Data Engineering. Could you give any suggestions how to find a junior position in data engineering (No Experience ) and any online course I could take? Dataquest has an interesting course in. Data engineering.

[@karolinasowinska]

Hi Aneeq, great question! I will soon make a seaprate video on how to become a data engineer - hopefully that will be useful to you!

[@lucascorazza9792]

Hi @@karolinasowinska. Does it make sense to learn both data engineer and data science?

[@esm2000]

Thanks for the video and the insight. Do you think becoming a data engineer is feasible for someone coming from undergrad who's done two data science-oriented internships. I want to make a transition from data science to data engineering and the upcoming school year is my last before I earn my degree in CS.

[@karolinasowinska]

Yes, absolutely, you can! That's exactly what I did. I had an internship in data science first, and after that I transitioned to my field :)

[@mojloginjuzzajety4071]

It is really nice to listen to You, such a distance and so nice reflections. More over, You are just beautiful.

[@captainphysix]

Hello, just subscribed, AMAZING content! Very helpful! May I ask which country you are from, because sometimes you sound british or french or even russian!

[@karolinasowinska]

Wow, I sound like I'm a citizen of the world! Which is what I like to think of myself! But I'm originally from Poland :)

[@derekx3699]

Which country are you working at? Your video is the clearest I've ever watched on DE role. I am a full stack developer(junior, have just 1 year experience), so can I ask how do you see the career path between DE and full stack developer? Thank you

[@karolinasowinska]

Thanks so much! I live in the UK. I'd say DE is always a backend role, whereas full stack can involve some backend, and some frontend (like building GUIs). One more difference is that typically DE is responsible for bringing the data to a company for researchers to analyse. On the other hand, if SE does some back end work, this is typically to deal with data that their application generates. Of course everything will depend on a specific job. :)

[@muratisik6956]

You make a boring subject sound interesting, to me, so my compliments! 👍🏼 Did you receive the MacBook Pro 16 inch yet? Like it? I’m waiting for a MacBook Pro with WiFi 6 and a much better camera.

[@karolinasowinska]

Thank you so much :) Yes, I've received it. So far, no issues, I'm liking it!

[@milantripathi720]

well in my country every single employee in most AI company is ML Engineer/data analyst/data engineer/Data scientist/dev ops....full stack data scientist

[@karolinasowinska]

I can imagine that! Where is that, by the way? :)

[@alejandrocarvajal9068]

Another great video, You seem to have a lot of experience (or at least that's what it looks like) on scalability systems. I have one question for you, how much do you charge for a tutoring session or a workshop meeting ?

[@karolinasowinska]

I don't have much experience with them, I'm afraid:(

[@alejandrocarvajal9068]

@@karolinasowinska Well, don't worry about it. I'll just hope success on your journey.