What do DATA ENGINEERS do? Is data engineering a good career choice in 2020?

[@karolinasowinska]

I've just released my Free Data Engineering Course for Beginners on KZbin! Make sure to check it out if you're interested in a career in data engineering! :) kzbin.info/www/bejne/mqfZmnyLgNx7bdE

[@karolinasowinska]

@Profound Programing I don't, but you can also find me on Instagram: @karo_sowinska :)

[@karolinasowinska]

@Profound Programing I'll consider setting it up at one point! :)

[@annmaryjo3803]

U r really amazing to starters. NB: I am just studying in 9th standard 😅 just checking how it works so that I don't have to look for this in more big way😁

[@karolinasowinska]

@@annmaryjo3803 Thanks, and good luck on your journey! :)

[@giljav]

What is your background and how you got into data engineering?

[@patrickd4468]

I am Data Engineer and I describe myself as follows: "I am like a plumber in the data world and ensure data flows from one abstract system to another in a scalable, consistent and standardized manner to allow the business users to consume the data in a usable way" . My only advise is to ensure all your pipelines are robust and fault tolerance/notifications are built into your data layer.

[@karolinasowinska]

Absolutely, a data engineer should always consider as many edge cases as possible, because they happen more often than not! I'm glad to read a comment from a fellow data engineer! ;)

[@mauricewalshe8339]

And in finance make sure the pennies add up :-) I used to work on billing systems for telecoms multiple types of records multiple types of data sources - we used Map Reduce (back in the early 1980's ) with F77 and Pl/1G

[@sumittamang3548]

If you had to suggest an undergrad student between data science and data engineering, what would you suggest?

[@dianerocher6705]

As a recruiter trying to understand job queries, this is gold ! Many thanks

[@karolinasowinska]

My pleasure! :)

[@MisterNorthernCanuck]

I just got myself a job as a data engineer as my new career challenge. I appreciate your content, it essentially confirms what I'm getting into :)

[@anyasyht]

As a data engineer I couldn’t agree more with you!

[@yooalejandro386]

That's amazing! Your speaking is very logic and concise, covering a lot of the most important concepts or sessions a data engineer does everyday. A very talented and graceful lady. Thanks for sharing and god bless you.

[@karolinasowinska]

Thank you so much! I'm glad you like my delivery! :)

[@mokshgrover9517]

I wanted clarity on this profession for a while now. Nobody could explain this in an easy way. You did. Thank you! Very informative.

[@karolinasowinska]

My pleasure! :)

[@lfrqz]

Your vlog feels real, honest and accurate. I am learning about data science, sql, python, etc. and I find your content very inspiring. Thanks for sharing your knowledge and experience. I am looking forward your future content. Greetings from Mexico.

[@karolinasowinska]

Thanks so much Luis, I'm glad that you're enjoying my videos! :) Good luck on your data science journey!

[@epicwhat001]

1. Bring Data in from different vendors. Check 2. Data pipeline , data warehousing , and aws. Check 3. Table-valued function , Procs and Views. Check 4. Analytics, debug and Monitoring. Check 5. I will Help but Not going to be nice about it. Check Thank you this gives me confidence to change my career to data engineering . I Am not a Data Engineer but a BI developer. my company is kind of like a start up so we don't have a data engineer so I ended up doing all the pipeline to create my SSRS and Tableau report.

[@karolinasowinska]

Sounds like you're already doing data engineering! Awesome stuff ;)

[@roopaljain1390]

From a women in tech to a women in tech, great vdo. I can see the spark in your eyes mentioning your fav part.

[@karolinasowinska]

Thanks! Great to hear from a fellow woman in tech! :)

[@jaimecortesguevara5908]

I really felt that part when you mention to "adapt" for the next 5-10 years, because that's when you need to keep investing in your self. Thanks for sharing your opinion

[@karolinasowinska]

Absolutely, we will change jobs more frequently than people did in the past, so one needs to expect that and learn to be more adaptable! :)

[@Robi2009]

Komentarz dla zasięgu ;) Powodzenia w rozwoju kanału!

[@karolinasowinska]

Dzięki! :)

[@bhuvaneshrao6322]

This is one of the very good videos i have seen on data engineering, thanks.

[@karolinasowinska]

Thank you, that means a lot. The saga is to be continued very soon!

[@longinteger]

Just started my career as a data engineer in a professional environment as well and I can align with everything you say, Karolina! It's also very interesting because many people think Data Science is just that one single field but it can be divided in so many sub ones. I think you make the role of a DE clearer in this video! Great work. I am very excited to get to the same point and share the same experience with you.

[@karolinasowinska]

Hey, thanks so much! I'm glad to hear that you've embarked on your DE journey, I'll be curious to hear how you're finding it! :)

[@norpriest521]

How many tech stacks you need to have to become junior DE? Like you need to know Python, SQL or smth else? How many prerequisites should we have before go on to become junior DE?

[@peterndiforchu2622]

Love it! How comes I'm just discovering you. Love your sense of humor. I'm working on the path of becoming a data engineer. This is encouraging as all the 5 points are things that excite me. Keep sharing great content. We'll keep supporting.

[@kintarooe575]

Super glad I came across your channel, Karolina! I'm currently a DA working to make some long-term transitional goals come true! -> data engineering -> data science. Thanks for your awesome insights! Your delivery is really engaging and easy to understand! Best wishes, Karolina!

[@karolinasowinska]

Thanks, I'm glad you enjoy the videos! :) And your career transition plan sounds great, good luck on your journey!

[@sajarnitsa]

Thank you very much for this video. You could believe you are terrible helping other people installing software and things like that, but your videos are the most honest and down to earth description of the job of Data Engineering You are amazing.

[@karolinasowinska]

I'm so happy that I could help some people with the videos! Thanks!

[@jmrodrig00]

Hi Karolina, thanks for sharing your thoughts about the subject, I starting a career as a data engineer and you video clarifies me and motivate me to do it as quickle as possible. Best regards.

[@karolinasowinska]

Good luck on your journey and I hope to see you around here! ;)

[@vsevapopov2]

So great to have such video lessons on youtube! I had an internship 2 and a half years ago where I did the data engineering task and now I want to continue my journey!! Now I study at university and I have much time to do that! Thank you!!!!!

[@hypebeastuchiha9229]

Thanks for this video I’m happy to see an actual data engineer give her opinion unlike bigger cs career channels that just read articles

[@victorborges5291]

Knowing full well each job title is like a passing show these days, without doubt, I'm inspired! 👍

[@lovemaakhe]

lol the shade in calling data scientists 'not-technical folks', great video!

[@samguy7209]

Thank you for sharing your experience and the great description. I love your accent. You have helped me decide my future career. ❣️

[@karolinasowinska]

That's wonderful, I'm glad my video was so useful! ;))

[@juanluisduquelafuentes1823]

Thanks for another interesting video Karolina! You know...as I was watching the video and hearing you talk about the different tasks data engineers have to perform, I kept thinking about the inevitability of all of those processes getting automated eventually. And of course you concluded with that interesting tidbit about having to learn new skills because you anticipated automation. Just goes to show, not only do you have to identify new opportunities and learn new skills in order to keep thriving, but you also have to develop a strategy to become as financially independent as possible so that you’re not caught all of a sudden in a situation where your skills just cannot support a decent wage anymore. If that doesn’t motivate someone to get off the daily grind for a few hours a night and just sit down, make new plans, pursue other interests, and just think about the meaning of it all, I don’t know what will!

[@karolinasowinska]

My thoughts exactly, Juan! What you said about financial independence in particular. Sadly, in the face of automation, it is those who are not financially independent, or whose skills are not sufficiently transferable, that will be hit the most.

[@yash1152]

11:46 - 12:19 - 12:37 - 12:57 this part of the vid is the one i liked the most. not that it's something i already didnt had the idea of, but it felt nice to get reminded of the upcoming highs.

[@chrisv8341]

This is extremely clear and concise! Thank you for sharing Karolina :)

[@karolinasowinska]

Thank you! :)

[@kevinle5772]

I like your clip, very well articulated. It's like you are reading my minds.

[@karolinasowinska]

I am a mind reader indeed ;)

[@md.shahriarrahmanrana6692]

You helped me a lot Karolina. Continue to share your experience, so that we can learn from you.

[@karolinasowinska]

I'm glad that the video was useful! :)

[@shaileshxjain]

Karolina, You are a pure heart 💓. (Evidence at 11:18) Keep uploading videos and share us your experience about Data engineering and it's prospects. May you achieve a Million Subscribers soon. Kudos, from India ✌️✌️

[@karolinasowinska]

Aw, I was just honest! Thanks so much! :)

[@caassimbah2485]

This was so helpful and informative. Thank you!

[@adriang.4839]

Thank you!! I just entered university in the career of data engineer, your video helped me clarify certain doubts

[@karolinasowinska]

I'm glad you found the video useful! Good luck at university! ;)

[@divyanshshrivastav2469]

Hi, i also want to study data engg in an university but didn't knw there is any good universities and procedures for it.. If u can send me an email regarding that knwldge will be helpfull..... [e-mail]

[@kubasochacki9599]

Przydatny film dla studenta I roku Data Engineering!

[@karolinasowinska]

O, super! Fajnie, że znalazłeś taki konkretny kierunek, na większości uniwersytetów jest tylko Computer Science. Powodzenia! :)

[@vineetjainorg]

Really liked your video! I work as a Data Scientist and can totally relate to the hard work our data engineering team puts in. Thanks, keep making more videos!

[@norpriest521]

Can you let me know what is the career path for data engineer? Say I don't want to work as data scientist or machine learning engineer, I only like to do data engineering my my entire professional career. Can one go far just by working as data engineer only and doesn't move to other areas like ML or DS on the later stage?

[@karolinasowinska]

Hi, absolutely! Data Engineering career path, like all career paths in technology, can take many shapes. But in majority of cases, you'll start out as a analyst/developer (building, fixing data pipelines), and then you will move on to assuming more responsibility, e.g. designing the entire data architecture of a company or managing teams.

[@ngee4925]

This was super helpful, thank you so much!

[@karolinasowinska]

My pleasure! :)

[@luisalamo2658]

Thanks a lot from these kind of contents about Data Engineering, I'm a Java developer and I'm looking for conquer a new boundary because I'm a little bored doing micro-services and I want to try some related new topic. Greetings from Chile.

[@karolinasowinska]

I'm glad this was helpful, and best of luck! ;)

[@Randy_Loafers]

Welcome to my world! Best of luck 👍

[@karolinasowinska]

And same to you! ;)

[@spideken123]

gorgeous as always miss karolina. very imformative.. thanks dear :)

[@karolinasowinska]

I'm glad you found it useful :)

[@JorgeEstradaMty]

Hello, I come from the @CodinEric channel. Your content is very interesting too, thanks for sharing

[@karolinasowinska]

Thanks and welcome! ;) Sorry for the delay in responding, I've just noticed your comment!

[@fbalseiro]

Great content! It's funny you said that the thing that you like the least is to help others and here you're creating videos to help others on how to become a data engineer. Plot twist. 😅 Nice job Karolina 👏

[@irishuang5294]

Thanks for the video and it's rly nice to see another female data engineer voicing their thoughts. I've been a data engineer for officially, I'd say 1.5 years and I really enjoyed the net result of ups and downs. Data engineers understand business better than software engineers, and understand tech better than data analysts/scientists. So to me, when I look at it as a role in translating data for business, I find it quite impossible to be obsolete in the next 5-10 years UNLESS the average data literacy within a company has increased (making better and more careful choices on data inputs) or people turn to machine-generated data more than human-generated data (or when NLP and Deep Learning married for good to successfully "guess" the right data). Otherwise the business would always need to pull in a data engineer to scale their data while preserving its correctness.

[@karolinasowinska]

Hi there, nice to meet a fellow female data engineer! :) That's a very interesting and convincing perspective - perhaps you're right (and I definitely hope so!)

[@sligon00]

@irishuang A very insightful comment , thanks for being so clear and concise about what goes on behind the scenes in the world of data.

[@farwanqv]

A good one to explain what data engineers are supposed to do

[@karolinasowinska]

Tried my best! :)

[@denismerigold486]

being a data engineer is cool. But deep learning is much more interesting. Thank you!

[@karolinasowinska]

I agree that deep learning is more intelectually challenging!

[@norpriest521]

Can one work as DE the entire career without having to move onto other relevant areas like MLE or DS later on? Say I love DE job so much, can I stick to it forever? What will be career path for DE? I don't like machine learning and data scientist role 😂

[@karolinasowinska]

@@norpriest521 Hi, yes, you can! Typically the career progression is such that you eventually become the person responsible for the overall data flow in an organisation, eg. the Head of Data Engineering.

[@exxodduss2008]

Perfect description

[@TheCaesius29]

i'm thinking of a career shift to a data engineer. thank you for this :)

[@karolinasowinska]

My pleasure, and good luck! :)

[@seb6226]

Is it possible to create a video(s) about data scientist vs data analyst vs data engineer? Or data jobs vs machine/deep learning/ai vs computer science jobs maybe a series of videos? Could u do a video about how much ai/machine/deep learning could impact jobs and which jobs will likely become automated or disappear vs which jobs will remain strong in the next 5-10 years? Also based on your last name, its great seeing someone else from a Polish or European background!!

[@karolinasowinska]

Awesome suggestions - noted, they land on my list! And yes, you're right, I'm Polish! :)

[@claudiusandika5366]

wonderful insights!! keep creating :)

[@karolinasowinska]

Thank you, will do! ;)

[@yorkstbd]

I'm as a junior BI developer, studying and working with tableau, and I'm missing the programming. I'm going to start working on some projects on my own and see if that's really what I want to do, because of everything you said I liked it so much! The bad part is that where I live there is only Senior vacancy for data engineering

[@karolinasowinska]

Maybe some junior positions will pop up soon! Definitely go for what you enjoy!

[@yorkstbd]

@@karolinasowinska

[@candyfloss184]

Suppose a company wants to have an erp from oracle to sap, now past data is in oracle db, future will be in sap db, now CEO of the company would like to have all data in same format report to make a year on year comparison, then you would need data engineers.

[@karolinasowinska]

Definitely! Data engineers are useful people to have on board! :)

[@candyfloss184]

Try to learn sap BW if you are so much interested to work in data engineering. But you could gain lot more insight if you would understand how business works. But then you donot appreciate economics much. So I think technical role is better for you.

[@karolinasowinska]

@@candyfloss184 Economics has little bearing on reality. Thanks for your suggestions though.

[@candyfloss184]

Karolina Sowinska I have always like macro economics. Microeconomics sounded a bit technical and it was difficult. Development economics and international economics too were interesting to me.

[@splashoui3760]

Was a good video. Thank you for this one!

[@CarlaAnneQ]

thank you for the insights! = )

[@victorpinasarnault9135]

2021 and still worthy.

[@karolinasowinska]

Not much has changed since I recorded this! :) Perhaps the accelerated move to the cloud!

[@geoffreyreceveur4605]

Thank you for the video. Very clear

[@karolinasowinska]

My pleasure, I'm glad you found it useful! :)

[@gracechan3039]

I work for an IT consultancy that primarily migrates businesses databases to the cloud. The quantity of work seems endless.

[@karolinasowinska]

100%, I can imagine that, it's a very trendy thing to do at the moment!

[@akeemadedigba8279]

working on cloud computing makes life easy and flow

[@Techkhamun]

Very interesting! Keep up the great work!

[@karolinasowinska]

Thanks so much! :)

[@qasimzaheer1140]

Very very good video! Awesome.

[@karolinasowinska]

Thanks! :)

[@ReenanOFC]

Awesome video.

[@karolinasowinska]

Thank you! More videos on data engineering are coming soon :)

[@matheusmeirelles4189]

Thx! That's worked very well to me, cause I'm already work with data but I didn't know what I was lol.

[@weirdinventor]

thank you so much , keep up !

[@blah-blah-blah4715]

To your last point about standardization and automation, I'd honestly love if Data Engineering and ML Engineering came closer once this happened. DE's already have to deal with all the requests from MLE's, might as well just put them in the same room. I see both getting closer to being abstracted away to standardized tools within the broader software engineering ecosystem. More successful and matured design patterns just need matured first. But once this happens, I very well could just see software engineering jobs focused on ML and/or Data experience being the new norm.

[@newaccbub]

Super helpful!

[@freemind.d2714]

Always pleased to see experienced people talk about they work and life. According to this video, I guess your work is much more focus on the pre-process the data rather than implementing and training the model right?

[@karolinasowinska]

100% true :)

[@codingevelynmiami4230]

Appreciate your explanation.

[@karolinasowinska]

I'm glad it was helpful! :)

[@vitelinha123]

Apache Airflow looks like a good fit for this sort of pipeline, really powerfull

[@markslima1557]

very cool thanks for posting - i'm learning about data analytics/python/sql wondering if data engineering is a better fit with my IT background

[@DataAnalyticsIreland]

Great explanation of this role, thanks for sharing!

[@karolinasowinska]

Thanks! :)

[@DataAnalyticsIreland]

@@karolinasowinska you're welcome!

[@troyfrericks-wti2510]

Would love to hear your opinion about what "Data Architects" do.

[@karolinasowinska]

Very similar job to what Data Engineers do, but I guess Data Architects are typically higher up in the hierarchy/they're more experienced, so they can decide on the architecture, and not just build pipelines

[@Yok685]

hahah "each data vendor, they are not nice people, they want data in certain format" hahahah true! I can relate so much

[@karolinasowinska]

Totally! haha

[@ssbrunocode]

Thank you for the good content! You are awesome o/

[@karolinasowinska]

I appreciate that!

[@ahsanmohammed1]

Great video! Good info! Sincerely done! Appreciated! Thank you! Liked! Subscribed!

[@karolinasowinska]

I'm glad you enjoyed watching! :)

[@ihgnmah]

Hi Karolina, your content is amazing and informative. I just curious that what to learn to get started with data engineering. Thank you

[@karolinasowinska]

Thank you so much! I'm soon going to release a series of videos on data engineering, so stay tuned! :)

[@ihgnmah]

Karolina Sowinska That sounds interesting, Thank you

[@eshwarpriky1571]

I don't know why but I started loving this girl. I am on a date with this girl through youtube videos. Keep posting more videos for your lovable fan.

[@karolinasowinska]

That's very flattering, thanks!:)

[@Fruitslycer]

Probably gonna have to merge Data Analysis, Machine Learning theory wise like MSc paper reading, implementation wise, like X amount of frameworks, C++ production code, and Data Engineering Pipeline/MLOps, into a single title, and know all those stuff, to have a job in the future. I feel like the market has been a lot softer on older guys, and us young new ones are gonna need to raise 3x of their for same salary (or less). Also older guys are far away from being nice to young ones imho, and the technical debt will become greater and greater too!

[@karolinasowinska]

Perhaps that's true. One benefit that the younger generation of developers have is knowledge/information availability!

[@buithanhlam3726]

This is very helpful!!!

[@karolinasowinska]

I'm glad you found this helpful! ;)

[@datawolf2318]

I'm an unspoken data engineer lol. I didn't know what ETL was...read about it and realized I have experience in it lol.

[@karolinasowinska]

Haha I can completely relate - I wrote my Master's thesis on this topic, and only later did I realise that was ETL!

[@karolinasowinska]

@Jared Dudley or data engineering specifically! ;)

[@bryanortiz9753]

Beautiful, funny, smart and data engineer like me! IM IN LOVE!!🥰

[@badass_omelette5166]

I have been enjoying data engineering and cloud engineering over data science recently

[@mytechchannel2519]

Useful information, thanks🌹

[@karolinasowinska]

I'm glad it was helpful! ;)

[@ofekezra7567]

Amazing video. simple and straight to the point. Karolina can you please explain if degree is needed to became a data engineer ? and if yes - what can i learn in home before (which skills)

[@karolinasowinska]

Hi, thanks so much! A degree in a technical subject is a plus, but not a strict prerequisite. I've just released a free data engineering course here on youtube, so if you want to learn some relevant skills, check it out :) kzbin.info/www/bejne/mqfZmnyLgNx7bdE

[@ofekezra7567]

@@karolinasowinska thank you so much

[@daringvishalverma7983]

Thanks a lot. You explained everything very well.I'm starting my career as a Data Engineer, so wish me luck.

[@karolinasowinska]

That's amazing, good luck! :)

[@harshavardhan4523]

Wow what a great video.....It was funny even thank you

[@karolinasowinska]

Thanks!:)

[@medisettycharan9152]

Many thanks for the information, it’s really helpful. Can I please ask you with whom will the data engineers will be interacting most of the time. And what kind of information will be exchanged.

[@karolinasowinska]

My pleasure! Mostly with data scientists and data vendors:)

[@cosmosbrasil5403]

Half a year? You know so much apparently for that period hahah. Anyways, I recently got my Physics degree and worked with Python in a few coursework and final year project. Next, I'll do a Masters in London but intend to start my career in the 'big data world' and I definitely consider data engineering (but mostly data science :D ). That video was quite helpful (although I don't know 10% of all programs you've mentioned lol ). Anyways, thumbs up! Thanks Karol. Ah, I hope I never need your help to install a driver or smth ahhaha

[@karolinasowinska]

Physics, nice! Awesome foundation for data science! Oh, I also did a Masters in London - have you started looking at which unis you'd want to apply to? P.S. Data Science is probably more interesting (if you like statistics more than technology/coding) P.S.2 I also hope you won't need me to install drivers 😂

[@cosmosbrasil5403]

@@karolinasowinska I already applied to 3 unis, I was lucky enough to have 3 offers from Queen Mary, Imperial and UCL. I'll think I'll stick with UCL, although there's still a small chance to change my mind. I'm glad to know that Physics is a good foundation. I love almost everything related to mathematics, coding, statistics, etc (a true geek, ahah), but have no experience in data industry : / a bit scared of not finding any job. About P.S.2: if I do, I prefer to ask you to make a video about it ahahah. Thanks Karol.

[@karolinasowinska]

Wow, congratulations, that's a lovely set of offers! I'm an Imperial graduate myself - if I can give you my biased opinion, it was a very good course (but also very, very tough). Don't worry, the industry loves geeks haha, you'll surely find a great job! A video about drivers you're saying... I'd give this one a long think! 😅

[@cosmosbrasil5403]

@@karolinasowinska ahahaha is better doing vids about your relationship with Apple ahhah. Imperial graduate, uow. Congrats!! I hope to go well through this toughness and start working soon (eager to learn :D ). Have an amazing evening. :D :D

[@DAJakaRedAries]

Thank you for th enice video. Subscribed :) I'm wondering if data engineering job is more demanded comparing to data analyst jobs? Do you need to learn Scala in order to manage ETL pipelines? (I know Python/SQL already)

[@karolinasowinska]

I would say that data engineering is in more demand. You don't necessarily need Scala - the particular technologies depend on the company that you'd be working for. Python + SQL is definitely a base. I'd recommend learning about data modelling rather than Scala, because it's more universal :)

[@pouinichan]

I'm a BI analyst who got approached for a data engineer job. I'm really hesitant about doing the move. I'm interested about learning more about the architecture and big data field but I'm afraid that I won't like it and won't be able to go back in an analyst position 🥺🤯

[@karolinasowinska]

If you enjoy being an analyst, don't necessarily jump at the opportunity!

[@jae11011]

Thanks a lot for the intro! It helps a lot for me as I am thinking of making data engineering as my career goal. For the last part, if data engineer becomes less demanding in the future, may I know personally which way would you recommend a data engineer to go, software engineer or data science?

[@karolinasowinska]

I believe you should go for what's most suited to your personality and interest :) Engineering vs science/research is the key question here.

[@sumittamang3548]

If you had to suggest an undergrad student between data science and data engineering, what would you suggest?

[@muhammadsaad907]

Thank you so much :)

[@MohamedHSafey]

Can you take a look on datacamp data engineer path and give us your feedback

[@karolinasowinska]

Sure, I'll have a look soon and let you know!

[@victor5265]

@@karolinasowinska could you take a look at data engineering path in dataquest as well?

[@karolinasowinska]

@@victor5265 sure, I'll try to talk about it soon!

[@captainphysix]

Hello, just subscribed, AMAZING content! Very helpful! May I ask which country you are from, because sometimes you sound british or french or even russian!

[@karolinasowinska]

Wow, I sound like I'm a citizen of the world! Which is what I like to think of myself! But I'm originally from Poland :)

[@omkarbhiwande7723]

Can you make a video on how should one go about learning data engineering?

[@karolinasowinska]

That's a highly requested video - I think I'm going to make it very soon! Stay tuned! :)

[@alejandrocarvajal9068]

Another great video, You seem to have a lot of experience (or at least that's what it looks like) on scalability systems. I have one question for you, how much do you charge for a tutoring session or a workshop meeting ?

[@karolinasowinska]

I don't have much experience with them, I'm afraid:(

[@alejandrocarvajal9068]

@@karolinasowinska Well, don't worry about it. I'll just hope success on your journey.

[@statisticianwordnpl4071]

Really interesting video! I Don't Know why in my country there aren't girl that talk about it carrers. 😅 Cya from Italy!

[@karolinasowinska]

Thank you! Tech careers seem to attract guys a bit more than girls, but I think it's slowly changing now, and many girls do coding! :) Ciao!

[@EmilianButoi2]

where are the data vendors getting their data from? and what job does data vendors officially are call? How does someone become a data vendor? It is some1 gathering buying data from multiple business as a third party consumers for exemple w/ xyz vars ? or is a compagny selling a service and recycling business by selling datas? Do data enginnerer learn to program to and data scientist learn more in a particular science and ask you as a shortcut to triage second hand clothings to more reflect on a particular science to draw conclusion for future/act improvement and problems? what are the use of data use for?

[@karolinasowinska]

Data vendors often *collect* the data. Let's say you're a government, and you record every time someone commits a crime. In the end, you end up having a huge database of crime incidents in your country. If you choose to make this data available (for purchase or free download) then you become a data vendor :) Of course this is just one example, but all other datasets get generated in this way - by recording and storing observations.

[@EmilianButoi2]

​@@karolinasowinska Thank you. Was nice dreaming of you and hugging u. Some guy not happy. But why Alina do some1 buy this? What they do with it. What's the ultimate goal. AI?

[@EmilianButoi2]

What are the type of firm who buy data? financial educational governement ect ? Why your employer buys data? Ob to make or try to make money with it but can you give exemples?

[@karolinasowinska]

@@EmilianButoi2 You need data to make well-informed decisions. For example, investment decisions or marketing decisions.

[@EmilianButoi2]

@@karolinasowinska carrer first. Do you know any firms who buys data ahead for the sole purpose of re. Obv. this will disappear because of AI. (I had this revelation once in a dream but am not ready). Do you know who is the biggest data collector? -EuropA lol

[@FrancoisLeCoguiec]

Karolina: Subscribed! Great video, really nice and simple down-to-earth explanations! One point though: I think at one time you talk of HADOOP and you seem to imply that HADOOP has to be on premise? Technically I think that is not true. So, your company might have their own on-premise HADOOP cluster(s), but it is also possible to get "HADOOP as a service" in the cloud, on AWS or other cloud providers. Still, that makes no big difference is what you explained, since the activities you described apply no matter if on premise or in the cloud, but I thought I would point it out.

[@karolinasowinska]

Yes, you're right, I only later realised that Hadoop might as well be done in the cloud! Thanks! :)

[@irishuang5294]

Is there an increasing or existing trend of hosting Hadoop on cloud? I also feel Hadoop tends to be an on-premise thing. My company actually has been trying to migrate from on-premise Hadoop to Azure

[@jesflo4]

Agh, I have a huge choice to either pick between becoming a Power BI developer or Data Engineer, I'm proficient with sql and understand databases and table structures, but I know 0 in Python. This video definitely moved me towards data engineering though... agh decision decisions doesn't help I have to choose by tomorrow)... *BTW fantastic video, thank you for sharing your experience 😀

[@karolinasowinska]

I'm glad it did! :) I also think that it is better to learn a broader, more sought after skills of data engineering. PowerBI is very useful, but sounds a bit narrow!

[@milantripathi720]

well in my country every single employee in most AI company is ML Engineer/data analyst/data engineer/Data scientist/dev ops....full stack data scientist

[@karolinasowinska]

I can imagine that! Where is that, by the way? :)

[@Anandhusreekumar]

Thanks for this video.entire KZbin is full of data science videos.im looking for data engineering channels

[@karolinasowinska]

Here is one! ;) Thanks for stepping by, and if you've got any video suggestions, please do let me know!

[@ErickHernandez-ri2ft]

It's great to know more about my degree (I am data enginerring student, I am in sixth quarter) and I think, oh my gosh, I need to learn a lot of topics because my teachers don't teach me but I am learning on my own. So, the question is: What are the main topics as a data engineer needs to know? and can you recommend me any page to learn about data engineering? More specific topics such as: Big data, distributed systems and data mining. Greetings from Mexico!

[@medotop330]

Should I learn to deploy big data in cloud ?? I saw the job requirements say aws or big query Google and azure ?? What about on-premisses ...

[@karolinasowinska]

Yes, cloud is a very popular choice for many companies, so it's worth learning AWS and the like

[@70shahin]

thanks Karolina. how we are gonna learn Hadoop? (if it is your favorite part :) )

[@karolinasowinska]

Good question, I'll look for a suitable course to recommend:)

[@70shahin]

​@@karolinasowinska Nice, wait for that. beside that, you can get affiliation from them.

[@pradhyumansinghmandloi8240]

Hello I found your very helpful but I have some questions can you please answer them..? 1.) As you mentioned that if vendor change the name of coulum then we can make a system where we can take query take dynamic names as column changes. 2.) As it is automation process then soon demand for data engineer will be less ? 3) is data engineer a safe career to choose without hesitation that it won't automate or or it's demand will be less 3.) Is data engineering jobs and demands are at stagination stage as many people are moving into it. Is their a enough supply of people to this field like when the supply will be matched then it won't be hot job ? Thanks waiting for your reply