Top 5 Data Engineering Tools currently in the Job Market

[@JashRadia]

Hope you enjoyed the video! Share your thoughts here and share it with people who might find it handy ♥ AI Interview tool, Vera by Board Infinity: bit.ly/3Z5MzSp

[@vinayak6685]

Can you please make a similar code to extract the keywords used in job description which we can add in CV, so our CV doesn't get discarded in the automatic filter

[@narasimharaodavuluri5321]

May i know why Azure data engineer tools were not added kn the search

[@seekers_life]

Can we expect more such examples in coming days as it will really help people like me, who are learning data engineering as it provides real life example to learn data engineering.

[@sirajansari2848]

Hardly I come across such videos where OP puts so much of their efforts. This is one of the most useful videos I saw today. Thanks for your hard work.!! :)

[@JashRadia]

Thanks a lot for the shoutout! 😊

[@VivekDesai]

This is genius. No one can make such quality content, I have found a gem! Kudos!

[@JashRadia]

Thank you! 😎😁

[@BillionDollerDream]

You never fell to amaze me...Thanks Jash

[@anishdasgupta8949]

Very helpful video, have been working with my company's DEs to figure out a DE framework for data handling, have been using your videos for references.

[@JashRadia]

Glad to know! Thanks a lot for watching 😊

[@aishwaryaradia]

I know the hardwork that you've put in this video... Proud of you Brother 👏🏼🎉

[@JashRadia]

Thank you! 😊

[@seekers_life]

Very well stated, thanks for your hardwork, it is really helpful, thanks a lot.🥳🥳

[@guruprasadashridharhegde6792]

Such an Amazing informative video till date. Thanks for sharing your valuable insights. Keep us updated..

[@JashRadia]

Thank you 😃

[@Faz-Ahmed]

This is amazing, i wish you all the success in the world, for your efforts

[@JashRadia]

Thank you for noticing! 😁

[@rshane1598]

Very nice job! Thanks for sharing.

[@saimahendrapatnaik1477]

Excellent way of approach

[@CareerCraftCo]

I run the program however came across a problem how to display the data as you have shown to see insights

[@deepam.g4558]

Why Apache Spark not there anywhere in a list ? Explanations please.

[@JashRadia]

It's there in top 10. Just not in top 5

[@CareerCraftCo]

I run the program however got a problem on creating insights like u did

[@artofheart2891]

Great Going Dear. Keep it up.. 👍🏻👍🏻👍🏻

[@JashRadia]

Thank you! ♥️😊

[@vinayak6685]

Can you please also share this code to us so, as the time changes we can run it ourselves and see the current top 5

[@JashRadia]

It's already there in description

[@ChampionsCorner12]

Great effort brother thanks for this

[@JashRadia]

Thank you for watching! 😊

[@amitpatel9670]

I am new to DE. I dont understand why spark is not there.

[@JashRadia]

As mentioned in the video. It's not a tool... Databricks, EMR, Dataproc etc are tools that run Spark.

[@sumedhpatil1474]

Great explanation.

[@MC-vw5le]

Hi Jash, to apply for intermediate level DE position at FAANG, do we need to be able to solve Dynamic Programming and Backtracking problems on Leetcode? Thanks!

[@JashRadia]

Not required

[@MC-vw5le]

@@JashRadia Thanks for getting back to me Jash! I assume linked lists, heap, queue and BST (DFS/BFS) are included?

[@saurabhgavande6728]

how can we change the location from usa to india

[@srirammanda9697]

U r awesome!

[@RahulPal1998]

Hi jash, I have completed python and sql , and basically pandas in python , having project done on web scraping , so what you suggest that can I now start with pyspark or anything else for data engineering.

[@JashRadia]

Yes, I think it is now good to start with pyspark and eventually you can learn real time data processing too down the line.

[@pmmodi3583]

LOVE YOU BHAI❤

[@JashRadia]

Love you too, Modi ji ♥️😅

[@prakhargupta4347]

How to decide between DE and SWE(backend), I am in 2nd year. I have dipped my hands into both and both seem feasible to me. I have to finalise and pick one thing eventually. How should I go about it. Please guide.

[@JashRadia]

Do you want to build apps or do you want to dig deeper into that app's data? What would you like to do more? There is no wrong option here and both roles are good. It depends on what do you like to do more.

[@prakhargupta4347]

​@@JashRadia I will have to spend some more time figuring out

[@aayushkumar561]

Please make a video to explain the setup and code you used to do the web scraping.

[@JashRadia]

After installing packages mentioned in the code, you can run it in any Python environment.

[@sathyamanikantabk4483]

Amazing and good information

[@JashRadia]

Thanks!

[@ritupornasengupta1391]

This is genuinely a very useful video and the work you have put to gather the stats is really recognized . I had one question -Are the stats that is pulled up is only for 'Canada' location or is it the complete list irrespective of the location ? While i get a reply on this would like to - Thank you for your videos I hope you keep making them and educating us .

[@JashRadia]

Glad you liked it! It is location agnostic

[@prof.mangabhai]

hello jash , my company has provided me with the tech stack as aws glue, snowflake and pyspark , is this the best tech stack (or the one which is in demand) , I am a fresher who has just completed my btech , I want to switch jobs within a year and a half, so do I need to get familiar with top 7 tools you mentioned or just sticking up to my current tech stack is well enough, becuase you said redshift,bigquery and snowflake falls under same category ,so would I be eligible for jobs where they demand some other tool like reshift or bigquery but I will have only experience in snowflake

[@JashRadia]

That is a pretty good tech stack. You'll learn a lot and it will be useful down the line, too. Good job!

[@pranavganesh1855]

Do ADF uses Apache airflow??

[@JashRadia]

No that's a different tool. It does what airflow does and some more like even transformations on its own.

[@bgmiinsider3379]

Azure Data factory not even in top 26 ? Strange.

[@JashRadia]

Yes that's what I thought, too. It is one of the main reasons people choose Azure 🤔

[@Onelife3468]

AWS vs azure which have more job and demand in market

[@JashRadia]

AWS is the market leader and GCP is the fastest growing platform

[@babluprasad813]

Hey Jash can you share how long you prepared and how did you prepare for a job switch? I am trying to job switch but i am confused where to start preparation. Also where do i find what to prepare for interviews for data engineer and science roles.

[@JashRadia]

If you're looking for a job switch then my roadmap video will be useful. For preparation, I prepared for a couple of months while working simultaneously.

[@pramodreddy8151]

Content ❤

[@mayurpatil8297]

Can you prepare save video for 2024?

[@JashRadia]

Sure, thanks for suggesting

[@mayurpatil8297]

@@JashRadia Can you share the code so that we can check it at our end?

[@Ghost-of-uchihaaa]

Why ADF didn’t make it to the list?

[@JashRadia]

No idea, surprising 🤷

[@Pavan_Prabhu]

hi jash ,what about snowplow? if i am giving a chance which one should i pick snowflake or snowplow

[@JashRadia]

It is very similar to snowflake but I think it's too new to make it into this list. Even if it is better, it would take time to adopt

[@SystemDesign-z2d]

#1 Snowflake #2 Kafka #3 Airflow #4 Tableau #5 Databricks #6 Redshift #7 BigQuery

[@moyeenshaikh4378]

Sir i have heard kafka if we don't get real time project then its better not to learn that as it's hard

[@JashRadia]

That is true. I myself didn't put a lot of time in learning Kafka like tools until I was working in a real time project but there are many roles out there that hire for this specific type of projects so learning it won't go to waste. I don't think it is hard though.

[@navaneethneeth6094]

Hi jash, Ho are you! In my company we have thing called academy by which we can move to technical domain, I chose Kafka and cleared its pre-assessment for further training. Will it be a good helping step or advantage for me by proceeding and working in that track(Kafka) to get into data engineering which is my ultimate goal? Or this kafka work experience won't be any added advantage for me to DE job? Please share your opinion, So I can decide to proceed or not with Kafka training :)

[@JashRadia]

Kafka training will definitely helpful but it will only help you with real time ot near real time analysis. Many DE projects are batch analysis as well. Eventually you will have to learn that, too. But Kafka is a good skill to in DE with. All the best!

[@prasadpg6189]

Thanks for sharing these information DO YOU HAVE ANY PLANS TO START GCP DATA ENGINEERING COURSE please let us know?

[@JashRadia]

Thanks for watching. And no, not anytime soon

[@mirli33]

How did you transitioned from Data engineer to AI engineer? What are the steps to be followed if one want to transition to AI engineer from Data engineer?

[@JashRadia]

I have made a separate video about this topic on this channel

[@sivaprasad-ji4fi]

How long will it take to transfer to USA location as a ai/ml engineer what is the way please tell me

[@JashRadia]

There is no definitive answer to this. Depends on company, projects and your skillset. Can take anywhere between 1 to 5 years if you really want to go.

[@SushantShambharkar]

What do you think about future of Flink?

[@JashRadia]

I have very little experience on Flink so I'm not sure I would be the right person to answer this. But I can say that I have never yet come across a project that requires Flink.

[@neerajrajput9977]

sir i am following your videos since couple of months,i love your hard work.apart from this i have a personal question.i have done my bsc(information technology) got passed in 2019 after that i prepeared for civil services exam for 4 year now i am thinking to study masters(data science) in germany .is this wise decision ?because study gap is very long so very difficult here to get a job

[@JashRadia]

Thank you and call me Jash 😊 If you're doing masters then this gap wouldn't be a problem at all. As long as you can justify why you took it. Because after masters you will be tested on the skills you learned in uni. Yes, they would consider you a fresher but gap won't be a problem in most cases.

[@ankitsharma8702]

Is learning a cloud necessary for data engineer as a fresher Or if I would learning a cloud what cloud should i learn as a fresher azure data engineering certification is good

[@JashRadia]

Yes learning cloud is a must. You can learn any of GCP, AWS or Azure. The knowledge is easily transferrable and they all have distinct features as well.

[@sivaprasad-ji4fi]

Hai jash tell me how u transfer to ai team do u need to learn ai and ml before shifting or after entering u learnt bcz it helps us when we also shift we do same as u did

[@JashRadia]

Yes you need to learn ML concepts and it would also help if you can show it with a certification like GCP PMLE

[@ninaddeshpande4427]

Can you also make a video on how to look for jobs in data, if someone is transitioning from Mechanical Engineering field, like how to create a good resume for data, which ceertifications will be helpful ?

[@JashRadia]

I'll try to address these topics thanks for suggestion

[@sivaprasad-ji4fi]

Hai jash this is siva i want to know how you prepare for dsa and please hell me resources

[@JashRadia]

There are only 2 resources needed for DSA. Geeksforgeeks for understanding concepts and leetcode for practice and looking at frequent questions.

[@sivaprasad-ji4fi]

@@JashRadiathanks jash in interview what is the level of difficulty in DSA they asked And how you changed u from data engineer to ai engineer 1.u need to learn the course before joining the ai team or after joining ai team u learning 2.how long will it take to transfer to USA location please reply

[@ramakrishnavanga1335]

In topmate please make price less na then it can reach then every one we are less financial

[@shivaramthallapally369]

Why can't you post data engineer full course on your channel.. It will be very helpful if you post.. Your way explanation is too good

[@JashRadia]

Thanks for the suggestion. I usually don't have the bandwidth to do this but if I do, I'll definitely consider this!

[@jhonsen9842]

Surprise where is Spark?

[@Nick-du9ss]

So we can learn mongodb over Casandra

[@JashRadia]

Both of their usecase are completely different. One is used for transactional but Cassandra is used as a columnar storage. More comparable to BigTable and DynamoDB. But anyway use of Cassandra is niche and s not an alternative to mongodb.

[@Nick-du9ss]

@@JashRadia these top 7 tools you mentioned it is trending tech stack means if we learn these skills chances of getting job high Or what ?

[@Nick-du9ss]

Hii jash good to see you back Visualisation tool is required for data engineering or not And i want to become azure data engineer so it is a good move for future If yes then which skills demands in market for azure data engineer

[@JashRadia]

Visualization tool experience is not a must for most roles but It's good to have. I would always prefer being a cloud agnostic data engineer. You can start with Azure if you really want to but I don't see a reason why you shouldn't explore other cloud platforms after some experience.

[@Nick-du9ss]

@@JashRadia i will explore other cloud but for right now I want to get entry in data engineering

[@krish0804]

Great content.. Keep doing more such videos. .

[@haaeee]

YNWA❤

[@jnana1985]

What happened to Spark? Dead already?

[@JashRadia]

It's not really a tool. It's a framework. Tools that execute spark are EMR, dataproc, databricks etc. All of them were included in this.

[@jnana1985]

​@@JashRadiaOk bro..How come redshift(dwh) postgres(db), mongodb(db) fall under tools?

[@A_yush_Panday]

Snowflake gaining so much momentum on every job profile related to DE

[@JashRadia]

💯💯