AWS Services You Should Learn as Data Engineer in 2023

Join Python for Data Engineering Course: datavidhya.com/courses
Learning AWS for Data Engineering can open doors for multiple opportunities, in this video, I will give you a simple explanation of all of the services used by Data Engineer on the AWS Cloud Platform.
Case Study on AWS - aws.amazon.com/blogs/big-data/
Timestamps
0:00 How to learn AWS Services
0:50 Service no 1
1:51 Service no 2
2:29 Service no 3
2:55 Service no 4
3:30 Service no 5
4:39 Service no 6
5:26 Service no 7
6:06 Service no 8
6:58 Other Important Services
8:00 Real-World Data Engineering Case Study
12:20 What did you learn? Summarize in the comment
👦🏻 My Linkedin - / darshil-parmar
📷 Instagram - / darshilparmarr
🎯Twitter - / parmardarshil07
🌟 Please leave a LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟
3 Books You Should Read
📈Principles: Life and Work: amzn.to/3HQJDyP
👀Deep Work: amzn.to/3IParkk
💼Rework: amzn.to/3HW981O
Tech I use every day
💻MacBook Pro M1: amzn.to/3CiFVwC
📺LG 22 Inch Monitor: amzn.to/3zk0Dts
🎥Sony ZV1: amzn.to/3hRpSMJ
🎙Maono AU-A04: amzn.to/3Bnu53n
⽴Tripod Stand: amzn.to/3tA7hu7
🔅Osaka Ring Light and Stand: amzn.to/3MtLAEG
🎧Sony WH-1000XM4 Headphone: amzn.to/3sM4sXS
🖱Zebronics Zeb-War Keyboard and Mouse: amzn.to/3zeF1yq
💺CELLBELL C104 Office Chair: amzn.to/3IRpiL2
👉Data Engineering Complete Roadmap: • Data Engineer Complete...
👉Data Engineering Project Series: • Data Engineering Proje...
👉Become Full-Time Freelancer: • Best Freelancer Series...
👉Data With Darshil Podcast: • Podcast Series - Data ...
✨ Tags ✨
✨ Hashtags ✨
#aws #dataengineering #projects

[@DarshilParmar]

New setup? 👀 I am working on next GCP project, I will upload it this month and launch SQL for Data Engineering course too :D Don't forget to like this video :)

[@finnyar]

hi i am not able to get any internship in data field i have a ba hons degree i have made portfolio also what should i do

[@arpitajana7788]

Love the new setup! and very very thanks for this amazing info. Excited for SQL course 🤩🤩

[@endangeredcreator5050]

TL;DR 0:50 AWS S3 (Simple Storage Service ) To store any type of data 1:51 AWS Glue used for transformation scripts 2:29 AWS EMR compute over a distribued cluster 2:55 AWS Redshift data warehouse where extracted datas are loaded for further process 3:30 AWS Athena to directly querry data from S3 without storing it in warehouse 4:39 AWS Lambda compute service 5:26 AWS Kinesis to process data in real time 6:06 AWS DMS (Data Migration Service)

[@RAHULKUMAR-px8em]

7:09 AWS Lake Formation 7:13 Amazon Managed Workflow for Airflow 7:25 Amazon QuickSight 7:27 Amazon RDS 7:48 Amazon VPC 7:50 Amazon Ec2 7:56 Amazon SNS 7:58 Amazon CloudWatch 8:00 Amazon IAM

[@shafi123]

whats the difference between EMR and Lambda as both are for compute purpose?

[@haarishkhan61]

😊

[@anildangol]

Great Video man!

[@sushantadnaik7548]

Great video brother ur all videos are very helpful for me..

[@hiteshramrakhyani6828]

Well explained Great stuff buddy ✌️

[@gavinkalaher7314]

Great video Darshil, comprehensive explaination and good scope of DE processes. After I've completed the 2 main courses I'm currently studying, I'll switch focus to some AWS projects.

[@DarshilParmar]

Thanks Gavin, all the best 👍🏼

[@VaibhavShewale]

wel that was very useull now i know everything!

[@amitjaiswal781]

Please also make video on GCP Data Engineer Roadmap.

[@rayyanabdulwajid7681]

Book My Show has a crazy ETL pipeline. Thanks for showing us.

[@electricalsir]

Thanks man

[@adaparachana7182]

I really loved your classes but I have one concern if you make any projects related to streaming data using technologies like Hadoop, Kafka, spark, hive, pig, and Flik related to hospital data it would be very helpful for me as a learner of your videos

[@hritikapal683]

After completing your python course this particular video made lot of sense, still practicing and learning these cloud services thanks man! Btw setup seems nice🔥👏🏻

[@DarshilParmar]

Awesome Hritika, keep practising and you’ll understand it better

[@abhishekravi6975]

Great video bro❤

[@muskanjaiswar2590]

Keep it up! It helps a lot for students like us who want to get into data engineering

[@DarshilParmar]

Thank you Muskan

[@bilalgaming123]

very informative video and new setup is looking great

[@DarshilParmar]

Thank you Bilal

[@rakeshverma6867]

helpful and providing your insights.

[@DarshilParmar]

Glad it was helpful!

[@sidsaliyam2725]

Man you are a great deal !

[@DarshilParmar]

Hey thanks man

[@istwishmeister9368]

Hi Darshil your videos on Data Engineer are very interesting and to the point. I would like to make a request can you please make videos on Azure. For example showing, completea pipeline for any OLTP or OLAP project? Thanking you in advance

[@DarshilParmar]

I will put this in my list

[@mmp9371]

please make video and courses on azure Data engineering road map, thanks.

[@krightjrar1356]

Bro, What do you think what's the future of data engineering after this recent AI boom ? I joined an MNC as a fresher in for data integration role.

[@anuragsoni1509]

Hey Darshil, I was waiting for this video to understand the integration of cloud services with ETL process and fundamentals of these services. Thanks for the explanation !!

[@DarshilParmar]

Thank you Anurag

[@varunmahanot5766]

We can even modify data through Athena by using ICEBERG type table.

[@DarshilParmar]

Yes

[@yuvraj2913]

Hey Pls tell whether it is mandatory to give AWS certification exam to become cloud data engineer ?

[@Nick-du9ss]

Darshil what do you think in future slowly slowly in data engineering cloud services will be used more and decrease open source tools like hive hbase etc.. means data engineering tech stack will go as much as possible on the cloud ?

[@DarshilParmar]

Open Source will never go away, most of the cloud services use Open Source projects and provide their own feature on top of it, like Managed Airflow, EMR (Hadoop MapReduce) Cloud is future but open is a base for everything

[@meetpatel2720]

Is aws glue and airflow is same?

[@mohdhammadkhan5570]

I used aws glue crawler. I am taking my mssql table to s3 and then through glue to athena so that i can query. In s3 i just checked the columns using objects action button available on bucket and then run a query. So there data is same as mssql. Data in columns where it should be. In athena some columns got mismatched and i don't know how to fix glue crawler now. Kindly help me out Basically crawler got confused because some cols has json type data in rows

[@jacksmith7160]

Darshil in data engineering We should know a lot other than practice Or practice will make us stronger in data engineering

[@DarshilParmar]

Theory + Practice = Win

[@bhaveshnayak6836]

If I start learning cloud, which cloud is good and which one can get a job quickly

[@call_of_duty1327]

i will add one more service, aws cloudformation for automating all these

[@chandann7344]

Data engineer with AWS Book is best to study and understand AWS

[@khalidsk3457]

Hi, if u have this book please send me

[@chandann7344]

I don't have the pdf but i can tell u that orielly's website's providing 10 days free subscription u can read there

[@bjoe2565]

Hello Darshil, Can you please bring some more real time projects? On Azure as well

[@DarshilParmar]

I am working on it, the thing is creating end-to-end projects takes around 3-4 weeks of efforts and requires lot of planning

[@udaybhaskarsavvana8049]

Hi Darshil, Need similar one on GCP

[@diwakarnrp2092]

Hii darshil pls do the project on azure

[@DarshilParmar]

I will put this on my list

[@Machaivelli]

Need your advice brother Between data analyst vs data engineering vs data science 1)Which role requires less coding ? I am a civil engineer who can do decent coding but don’t really enjoyed it like coding geeks do. So need your advice. People suggested me for data analysis profile as it has less doing and needs knowledge of data visualisation but would like to know your inputs 2)which role has more jobs in India ? Data scientist has jobs abroad but in India it’s still in less no and limited to folks from top colleges or brilliant coders, data engineering has decent jobs, no clue about data analyst job scenario Please help

[@aka_rony]

start with data analysis>after gaining some experience move to DE> then you will get a clear understanding of how an end2end business problem is solved. By that time you will know should I go with data science or stay in Data Engineering (which includes skills of data analyst too). hope this helps, happy learning : )

[@vaibhavverma1340]

Can I put on my resume the real time projects after completing your course ?

[@DarshilParmar]

Yes you can

[@pratikmahajan6082]

What about gcp project darshil when he coming

[@DarshilParmar]

Working on it, will update once it is done

[@venkatexplorermetturdam]

bro is this sufficient any free resources and shall i do cloud practisner certification im an aspiring data engineer , can u guide please

[@DarshilParmar]

This is enough but you can do AWS Data Analytics Certification

[@venkatexplorermetturdam]

@@DarshilParmar thanks bro

[@surajitpradhan7848]

​@@DarshilParmarfrom where we can do it ???

[@akashshaw9862]

Can't we do all these in Snowflake ? Why would anyone use these services?

[@DarshilParmar]

Because you can't do everything on snowflake, theoretically you can say (for marketing purpose) but real-world problems are not that straight forward you need custom solution based on requirements

[@s.211]

Bro bura mat maano but , I am following you since 1 year.. And aap ne last 1 saal main 10 baar roadmap pe video banayi aur wahi baatein repeat ki har video main.. wahi baatein pipeline creation main batate ho... Aap isse accha AWS ki pipelines wali ek ek service in depth padhao

[@DarshilParmar]

Last 1 year me 5 end-to-end projects bhi banai hai jisme sab service ka indepth explanation and hands-on bataya hai vo bhi dekhlo ;)

[@s.211]

@@DarshilParmar ok bro , thanks

[@endangeredcreator5050]

You got 6 more hours to make a aprils fools community post 😂

[@DarshilParmar]

😂😂

[@revanthbhuvanagiri9177]

helpful and informative

[@DarshilParmar]

Thank you Revanth