AWS Services You Should Learn as Data Engineer in 2023

[@DarshilParmar]

New setup? 👀 I am working on next GCP project, I will upload it this month and launch SQL for Data Engineering course too :D Don't forget to like this video :)

[@finnyar]

hi i am not able to get any internship in data field i have a ba hons degree i have made portfolio also what should i do

[@arpitajana7788]

Love the new setup! and very very thanks for this amazing info. Excited for SQL course 🤩🤩

[@endangeredcreator5050]

TL;DR 0:50 AWS S3 (Simple Storage Service ) To store any type of data 1:51 AWS Glue used for transformation scripts 2:29 AWS EMR compute over a distribued cluster 2:55 AWS Redshift data warehouse where extracted datas are loaded for further process 3:30 AWS Athena to directly querry data from S3 without storing it in warehouse 4:39 AWS Lambda compute service 5:26 AWS Kinesis to process data in real time 6:06 AWS DMS (Data Migration Service)

[@RAHULKUMAR-px8em]

7:09 AWS Lake Formation 7:13 Amazon Managed Workflow for Airflow 7:25 Amazon QuickSight 7:27 Amazon RDS 7:48 Amazon VPC 7:50 Amazon Ec2 7:56 Amazon SNS 7:58 Amazon CloudWatch 8:00 Amazon IAM

[@shafi123]

whats the difference between EMR and Lambda as both are for compute purpose?

[@haarishkhan61]

😊

[@gavinkalaher7314]

Great video Darshil, comprehensive explaination and good scope of DE processes. After I've completed the 2 main courses I'm currently studying, I'll switch focus to some AWS projects.

[@DarshilParmar]

Thanks Gavin, all the best 👍🏼

[@hritikapal683]

After completing your python course this particular video made lot of sense, still practicing and learning these cloud services thanks man! Btw setup seems nice🔥👏🏻

[@DarshilParmar]

Awesome Hritika, keep practising and you’ll understand it better

[@istwishmeister9368]

Hi Darshil your videos on Data Engineer are very interesting and to the point. I would like to make a request can you please make videos on Azure. For example showing, completea pipeline for any OLTP or OLAP project? Thanking you in advance

[@DarshilParmar]

I will put this in my list

[@amitjaiswal781]

Please also make video on GCP Data Engineer Roadmap.

[@sushantadnaik7548]

Great video brother ur all videos are very helpful for me..

[@amandeepsingh3143]

Thank you for this informative video

[@adaparachana7182]

I really loved your classes but I have one concern if you make any projects related to streaming data using technologies like Hadoop, Kafka, spark, hive, pig, and Flik related to hospital data it would be very helpful for me as a learner of your videos

[@VaibhavShewale]

wel that was very useull now i know everything!

[@rayyanabdulwajid7681]

Book My Show has a crazy ETL pipeline. Thanks for showing us.

[@hiteshramrakhyani6828]

Well explained Great stuff buddy ✌️

[@muskanjaiswar2590]

Keep it up! It helps a lot for students like us who want to get into data engineering

[@DarshilParmar]

Thank you Muskan

[@anuragsoni1509]

Hey Darshil, I was waiting for this video to understand the integration of cloud services with ETL process and fundamentals of these services. Thanks for the explanation !!

[@DarshilParmar]

Thank you Anurag

[@jacksmith7160]

Darshil in data engineering We should know a lot other than practice Or practice will make us stronger in data engineering

[@DarshilParmar]

Theory + Practice = Win

[@anildangol]

Great Video man!

[@NextGen_Tech_Hindi]

Thanks man

[@krightjrar1356]

Bro, What do you think what's the future of data engineering after this recent AI boom ? I joined an MNC as a fresher in for data integration role.

[@rajkumardubey5486]

Make video of dms

[@abhishekravi6975]

Great video bro❤

[@bilalgaming123]

very informative video and new setup is looking great

[@DarshilParmar]

Thank you Bilal

[@varunmahanot5766]

We can even modify data through Athena by using ICEBERG type table.

[@DarshilParmar]

Yes

[@mmp9371]

please make video and courses on azure Data engineering road map, thanks.

[@rakeshverma6867]

helpful and providing your insights.

[@DarshilParmar]

Glad it was helpful!

[@chandann7344]

Data engineer with AWS Book is best to study and understand AWS

[@khalidsk3457]

Hi, if u have this book please send me

[@chandann7344]

I don't have the pdf but i can tell u that orielly's website's providing 10 days free subscription u can read there

[@sidsaliyam2725]

Man you are a great deal !

[@DarshilParmar]

Hey thanks man

[@meetpatel.47]

Is aws glue and airflow is same?

[@yuvraj2913]

Hey Pls tell whether it is mandatory to give AWS certification exam to become cloud data engineer ?

[@call_of_duty1327]

i will add one more service, aws cloudformation for automating all these

[@bhaveshnayak6836]

If I start learning cloud, which cloud is good and which one can get a job quickly

[@mohdhammadkhan5570]

I used aws glue crawler. I am taking my mssql table to s3 and then through glue to athena so that i can query. In s3 i just checked the columns using objects action button available on bucket and then run a query. So there data is same as mssql. Data in columns where it should be. In athena some columns got mismatched and i don't know how to fix glue crawler now. Kindly help me out Basically crawler got confused because some cols has json type data in rows

[@Machaivelli]

Need your advice brother Between data analyst vs data engineering vs data science 1)Which role requires less coding ? I am a civil engineer who can do decent coding but don’t really enjoyed it like coding geeks do. So need your advice. People suggested me for data analysis profile as it has less doing and needs knowledge of data visualisation but would like to know your inputs 2)which role has more jobs in India ? Data scientist has jobs abroad but in India it’s still in less no and limited to folks from top colleges or brilliant coders, data engineering has decent jobs, no clue about data analyst job scenario Please help

[@aka_rony]

start with data analysis>after gaining some experience move to DE> then you will get a clear understanding of how an end2end business problem is solved. By that time you will know should I go with data science or stay in Data Engineering (which includes skills of data analyst too). hope this helps, happy learning : )

[@diwakarnrp2092]

Hii darshil pls do the project on azure

[@DarshilParmar]

I will put this on my list

[@bjoe2565]

Hello Darshil, Can you please bring some more real time projects? On Azure as well

[@DarshilParmar]

I am working on it, the thing is creating end-to-end projects takes around 3-4 weeks of efforts and requires lot of planning

[@udaybhaskarsavvana8049]

Hi Darshil, Need similar one on GCP

[@Nick-du9ss]

Darshil what do you think in future slowly slowly in data engineering cloud services will be used more and decrease open source tools like hive hbase etc.. means data engineering tech stack will go as much as possible on the cloud ?

[@DarshilParmar]

Open Source will never go away, most of the cloud services use Open Source projects and provide their own feature on top of it, like Managed Airflow, EMR (Hadoop MapReduce) Cloud is future but open is a base for everything

[@vaibhavverma1340]

Can I put on my resume the real time projects after completing your course ?

[@DarshilParmar]

Yes you can

[@s.211]

Bro bura mat maano but , I am following you since 1 year.. And aap ne last 1 saal main 10 baar roadmap pe video banayi aur wahi baatein repeat ki har video main.. wahi baatein pipeline creation main batate ho... Aap isse accha AWS ki pipelines wali ek ek service in depth padhao

[@DarshilParmar]

Last 1 year me 5 end-to-end projects bhi banai hai jisme sab service ka indepth explanation and hands-on bataya hai vo bhi dekhlo ;)

[@s.211]

@@DarshilParmar ok bro , thanks

[@venkatexplorermetturdam]

bro is this sufficient any free resources and shall i do cloud practisner certification im an aspiring data engineer , can u guide please

[@DarshilParmar]

This is enough but you can do AWS Data Analytics Certification

[@venkatexplorermetturdam]

@@DarshilParmar thanks bro

[@surajitpradhan7848]

​@@DarshilParmarfrom where we can do it ???

[@pratikmahajan6082]

What about gcp project darshil when he coming

[@DarshilParmar]

Working on it, will update once it is done

[@akashshaw9862]

Can't we do all these in Snowflake ? Why would anyone use these services?

[@DarshilParmar]

Because you can't do everything on snowflake, theoretically you can say (for marketing purpose) but real-world problems are not that straight forward you need custom solution based on requirements

[@endangeredcreator5050]

You got 6 more hours to make a aprils fools community post 😂

[@DarshilParmar]

😂😂

[@revanthbhuvanagiri9177]

helpful and informative

[@DarshilParmar]

Thank you Revanth