The Harsh Reality of Being a Data Engineer

[@SundasKhalid]

Thanks for watching! You can download the Python ebook (free) 👉🏼 clickhubspot.com/bbd

[@AchuVlogs]

Im a data engineer with 4 years experience. Harsh reality of being a data engineer is: - 5% building new pipelines - 15% automating the pipelines - 20% adding data quality checks to the pipelines And 98% of the time fixing bugs in OLD pipelines. For people who like building new products, this is not the right role. Because most of the time you're only firefighting. And since you work in backend, you don't get much recognition.

[@xinpengliu461]

Exactly agreed

[@aarnavchauhan4788]

Can you please guide me with a roadmap for getting into data engineering if I am from a college with good on campus placements, and whether I should learn data engineering in college instead of other conventional paths like say web development.

[@baybars3138]

Software development is samething, most of the times you are just fixing bugs. No?

[@AchuVlogs]

@@baybars3138 i have been both a software engineer and a data engineer. I find software engineering to be much easier. Ofcourse fixing bugs is on both side. But in data engineering, i have observed it much frequent. Example data source may change, data quality issues, errors due to upstream systems, errors due to missing information etc.

[@baybars3138]

@@AchuVlogs which one you found most interesting? Sounds like Data Engineer always has a job when Software Engineer doesn't.

[@hegerwalter]

Well, everything new requires you to figure out your path. I studied to be a mathematician but needed to find a job. Salaries in programming supported a more affluent lifestyle, so just a few classes in CS helped get my foot in the door. I think that the guy who wrote a violent race car game had a history Ph. D. I was reading about a philosopher who worked as a glass lens grinder, who died from cumulatively inhaling glass fibers in his 30s. Like everything else, you might need to change careers or job titles. Being a Data Engineer and knowing programming will get you further.

[@MitchMcCall-bd4ul]

yah these videos are clickbait. Along with the fear mongering over being automated out of a job which is barely happening.

[@MMARavid]

@@MitchMcCall-bd4ul100% these KZbinrs enjoy fear mongering for clicks, as well as inflating their salaries to get you to buy their products. It’s actually insane

[@vishnuvrv]

There are two types of data engineers: Software Engineers who are also data engineers and ETL engineers. If you are an ETL engineer (building pipelines using python and SQL) then you are just considered a glorified data analyst and your pay always be less than SE. My advice for them is to take the next step and become a cloud engineer (snowflake, Azure, AWS) and see your pay increase significantly.

[@angelsub9184]

Sir, are you advising DE with ETL engineer to take cloud engineer course or take Software Engineer bootcamp? Which one is better?

[@sabyasachi426]

Insightfull

[@LOFI_WORLD_SONG]

Hello sir. I don't want to code , programm like dev team. But I can work in cloud. Where should I go. DE or devops one. Please help

[@jean4j_]

Yes! There are technical DE and BI DE. Two different types of DE. Totally different culture and minset.

[@mitchtherighteous]

The real question is how to get the education without wracking up stupid provider bills

[@joelkanamugire6400]

I have been working as a Data Engineer for 3 years and I think it depends on companies and how they define their data engineers. I always find myself building data pipelines, API design, Web applications, Data Encryption and Database Security

[@LOFI_WORLD_SONG]

Do you code ? Or program from scratch?

[@jsingh7810]

Hey, where do u work?

[@patrickchan2503]

we like that! well done

[@omotayosawyer9535]

stressing on the point of a software engineer being able to do what a data engineer does is absolutely correct. I work as a data analyst and in one of my previous roles, the software engineer was pretty much the data engineer completing all the data engineering task.

[@patrickchan2503]

I'm sure DE's can switch to SWE too 🙂

[@dragster100]

But I think Data Analyst jobs can get saturated more easily than DE, considering the entry barrier is lower and there are a lot more people applying for DA, and then dashboard creation, SQL and less-advanced ML skills can be done by AI nowadays, don't you think so?

[@SantoshKumarsp57]

Hi Sundas, I am a bit confused. You are initially saying that SEs can do DEs role and also, with advancements in AI and possible taking over core SE's role, even they would do DE's role. Then, at the end, you are saying that DE's role will still be in demand?

[@jean4j_]

Not everybody agrees with this (especially juniors) but Data Engineering is a subset of Software Engineering. There are two types of DE: - software oriented DE - BI oriented DE DE is SWE. Be very good in software engineering with a focus on Big Data. You will able to switch roles easily.

[@max1232]

just got an internship as a data engineer(while at college) for the city of ottawa(only 1 spot btw)...wont have done it without you. Appreciate you..

[@vikaschouhan6976]

Hi please can you guide me how to get initial internship for this role as well have just started to prepare what projects and techstacks would it be needing

[@michalms7218]

Thank you - there are couple interesting points. Most software engineers who I know miss the skill of data modeling. Data engineers speaks more with data users and they are closer to data analysts (at least this is what I do). SDE need to know more technologies and dependencies but DE need to know more about data she/he plays with. Point about career growth has something, however I meet some Data Engineering Managers. But probably that is it. Then maybe DE switch to DS job but it is horizontal move and not a promotion. One last point is about logic - if SDE earns more then they will not do DE job for less. If they would be forced to do DE but paid the same it would still mean for them losing skill and time for something not related to SDE. It would be harder for them to change a job. That is why there is probably still demand for DE.

[@dragster100]

I like the logic part in your comment, good take and that could be a good possibility.

[@j.amethyst5685]

Great review, but i cant agree career growth. From Data Engineer you become a solutions architect. Or bridge being a DE to the next sensible path.

[@rangho10]

The aspect you might not have covered is that as most Data Engineers progress, they don't remain DEs but more so SREs and System Software Engineers, or systems architect as since they have a clear idea of how the information (data) is flowing, they have a clear idea of the system flow.

[@abiola4710]

The roles of a software engineer and data engineer are totally different, even though they might use related tech stacks. Also a data scientist can't take the role of a data engineer because, apart from being technical, a data engineer is expected to communicate with stakeholders and build solutions that turn data into insigts/revenue; data scientists suck at this. So a good data engineer should be able to do the work of a data analysts effectively. According to a Gartner report, 80% of data science projects fail. The number reason is: Lack of understanding of business context and user needs.

[@olatunbosunoyindasola6756]

Hello Abiola, I'm working to becoming a data engineer and would appreciate a mentorship as it's quite a large field and would appreciate any help you can give. I promise to not disturb you much. Thanks as I look forward..

[@abidrahim7607]

Thank you for putting this out, appreciate it! 🙌🏼

[@neetsabhi416]

absolutely disagree that SE can easily do DE role, working with data is different than coding, I have seen so many SE struggle with structured data let alone distributed computing and unstructured data, understanding data modeling, performance, ETL, understanding multiple storage and DBs. This is the high ground that SE need to get off, this is the reason why leetcode hiring gets you a really bad data engineer who is good at coding vs getting a really good data engineer who is average at coding.

[@kimblevuo]

Just started Data Analysist course few weeks ago.

[@mura685]

How is it going so far?

[@yousefosman4558]

i was confused about that thank you i will focus on data science then i will convert to data engineer because i don't see a particular road map for data engineer

[@sergiocarmona7238]

Im agree with all except number 2 software engineer can make easily DE work?, i think no, many DE were DBAs in the past. DBA is complete different skill to software engineering. 3. i think there is growing for DE you can skill up into machine learning engineer more easily than other roles.

[@cateclism316]

Even I make less than a developer, it's still a helluva lot better than my customer service job!

[@РоманР-я5п]

Hello everyone! I'm considering a transition from a Scala software engineer to a Scala data engineer. I've found that being a Scala dev is very challenging for me , but I'm not ready to give up on Scala just yet. Can anyone share insights on whether working as Scala Data engineer (Sql ,Spark) might be a smoother path for me? Appreciate any advice!

[@shobhitagnihotri416]

AWS beta certification for data engineer , Please talk about it.

[@nitcooksgf]

Yes I also want to know about this

[@mohammedabdulla1496]

Nice, I came to know that data engineers paid lesser than the average software engineer Felt sad 😔

[@yogajyothisomu2908]

Data science or data enginering which is better based on opportunites as a freshers?

[@prar_th_.2364]

Hi I'm a high school senior right now and I want to pursue a career in data science job family..... specifically AIML.....it would be wonderful if you could do a video on AIMl because I feel like there isn't much on the internet about the same

[@moinzajam5117]

Ma'am I m confused between data engineer or data scientist, which one should I choose , kindly suggest me

[@kazerdira4704]

I am interested in this field i come from mobile developpement (coding) , i still dont understand , you are dealing with data and SQL ? And little of python , ? If anyone know a course to buy or path to follow please help me

[@dotpy7928]

Through the whole video I was really concerned she will accidentally cut me with her fingers...............

[@Tayyaba-Zahid]

Sundas name is very unique for me 💕

[@rockshankar]

seems you went with your default thumbnail :D

[@SundasKhalid]

sadly yes 😭

[@harshitagarwal7475]

Hi Sundas, i m writing this comment to share with you that i got fresher job as a Data Engineer in one of the startups. I believe if u have the knowledge of tech stack required for the role, u can get hired. In btw, i have worked as data science intern for 3 months before(as it is considered hot topic in the field of tech and i went for it initially but coudnt find good pay jobs for it).

[@DaRedT]

thank you very much 👍👏

[@wendykgosi149]

Lol my gosh there are so many harsh realities why would anyone wanna pursue this.

[@kevinrobbs9717]

I’m a Database Administrator/ Network Administrator/ Full Stack Developer making 60k…. I would love 155k damn!

[@patrickchan2503]

you are all that in one job... amazing

[@junaidamin]

Please make sundas khaild Urdu/hindi channel for south asia audience specially Pakistani audience

[@AchampetaEshwar]

Instead of making this useless video why don't you try to make something useful like the ROad map or path to become a data engineer and courses to learn

[@patrickchan2503]

good question

[@vasanthmukkala5940]

You've no idea how fabulous you look!! 😇🫠

[@moinzajam5117]

Ma'am I m confused between data engineer or data scientist, which one should I choose , kindly suggest me

[@girlanonymous]

Do your research.