Data Engineering Was Hard Until I Learned These 5 Secrets!

[@DarshilParmar]

1500 Likes and I will upload Fundamentals of Data Engineering Masterclass here

[@sumithraj7954]

For free?? Darshil

[@tech-n-data]

1500 on the video? Because you are about there in one day or that comment? Y'all smash the like on both, we need our masterclass!

[@deadkachu7678]

Can u make videos on these topics u have said so that we can understand properly.

[@RamakshaBirajdar]

@DarshilParmar done, we want the masterclass 🤝

[@Userabj001]

I am among 1.5k .❤

[@ansonnn_]

0:23 1st mistake: understand the purpose of building the data pipeline, ask the why instead of straight dive into the technical details 2:02 2nd mastering sql is non negotiable 3:08 3rd automate everything 4:36 4th mistake: did not learn about debugging, maybe try Grafana / Prometheus, with an alerting mechanism 5:54 5th Learning is a continuous process

[@gokul2555]

Am a ETL/Data guy for 14 years... trust me he is giving the best advice which no one will say... DEBUGGING man I wish some one would have told me when I was fresher

[@prasadjoshi007]

DEBUGGING is mostly skipped all are focusing on building while most of the time you to debug.

[@shivangishrivastava9096]

Could you please make a video on Debugging concepts, especially w.r.t. data engineering?

[@DarshilParmar]

Will do

[@manojthalari397]

I just Googled that how to like a video 1500 times with single account(Unfortunately No option), Please upload Master class

[@DarshilParmar]

Will do it soon

[@srimathiganesan1516]

YOU VIDEOS ARE REALLY AMAZING!! I never comment usually but your work is really inspiring. Data always excites me and being a data research analyst now I wanted to shift to data engineering as I'm in love the work flow of it. Since I knew fundamentals in MYSQL and python for data analysis which includes fundamentals like loops, functions and conditional statements then forwarding to hands on numpy and pandas. Is it enough or is there something I should learn apart from this for data engineering ?

[@xx-pn7it]

How to understand the flow of data for the specific project or work

[@DarshilParmar]

It starts from asking questions to business people, what are they really expecting and do with the data, then from there you can start thinking about how to execute it

[@vikrantshelke2418]

Eagerly waiting for the Fundamentals of Data Engineering Masterclass! PS: Could you please make a video on Microsoft Fabric? Is it helpful as a data engineers? Should we learn this tool?

[@MeanLifeStudies]

Thank you for great contributions to the community and make me realise to contribute something to the community. So I started recently contributing to the community with SQL interview preparation, problem and solution series. Soon I will contribute many Thank you.

[@tejasvijb4228]

Waiting for your Fundamentals of Data Engineering Masterclass 💙

[@DarshilParmar]

Coming soon

[@AtiaNasr]

@DarshilParmar ... Is data Engineering is a suitable for someone who is love coding very much and don't like work with visualization stuff like dashboard, who like to focus on distributed systems, scalablility performanece and need to heavy logic thinking to do

[@uppindra]

Hi Darshil Make Videos on - Palantir Tool - very trending now in DE world ..

[@DarshilParmar]

Can you share link?

[@l_drago2875]

Timelines for next set of cource launch kafka airflow and projects

[@saachisrivastav605]

Can you please upload some thorough projects along with the business mindset behind them and also how we should build further projects

[@DarshilParmar]

I have lot of them on my KZbin - check KZbin analytics

[@jayanthgajula]

Best video editing

[@zeeshansaud9535]

Data Engineer Job Role and AI Contribution: kzbin.info/www/bejne/hIOafWOKiKuniacsi=kj4d_8rCdbQSCpPc

[@Vitali-pe3wg]

I am struggling to become a Data Engineer. In few months I will write my Bachelor thesis and it should be about data engineering. I hope with this thesis i will be able to enter the job market. Any recommendations for a thesis to enter the market?

[@DarshilParmar]

I'd say learn high-demand tools, try to get any job in data, doesn't have to be DE only and make your way from there

[@kakahalwaipunawale7991]

Darshil now that you have more than 1.5k likes Could you please upload the data engineering fundamentals masterclass

[@DarshilParmar]

I need to build a PPT, record, and edit video, it takes time! will upload once it is done

[@kakahalwaipunawale7991]

@@DarshilParmar Hey Thank You for your reply. And Thanks for all the efforts you are taking. I look forward to the masterclass 👍

[@aalamaram7680]

Great video Darshil...If i carrier change to data engineer to learn is difficult to get job ?

[@tanny_tales]

Bro in second part of youtube analysis video s3 trigger is not working properly and the data(parquet files) is not getting uploaded and no graph is visible

[@DarshilParmar]

Check IAM permission

[@tanny_tales]

@@DarshilParmar i checked can it be related to the thing that data types have to be same when creating etl jobs, interface has changed completly,kindly help

[@_DEBANJANDHAR]

I had created an account on 24 June, From today total 8 days are remaining I.e till 25 july for my free trial period. But it gets deactivated and asking my credit card details, WILL THEY CHARGE ME ??

[@lectures3727]

Hi bhaiya Needed a laptop in college and programming i started learning data science and ml but dont do gaming so i didnt take any GPU laptop I have 2 options First lenovo ideapad pro 5 with 14" ultra 9 H processor 32 gb ram 1 tb ssd and 84 wh battery Second is lenovo yoga slim 7i with ultra 7 processor 32 gb ram 1 tb ssd and 65 wh battery Both price same Which one should i go for?

[@sugirthachandrasekaran3523]

Thank you Darshil. How to learn about debugging?

[@mazharsayed7161]

Is ml ,dl is important in data engineering???

[@RobinHood-yn3bb]

no DE is Another sector

[@SurprisedGlobe-yq5uh]

I completed my b. Com computers and later i have competed my masters in marketing domain. Is am a eligible for a data engineer

[@sudarshanmhaisdhune1039]

Thank you Brother!

[@DarshilParmar]

You are welcome

[@krrishshylock6984]

Hi darshil, want to know can data engineerin work on model development as well . can they switch to mlops roles .

[@prasadjoshi007]

Really Helpful!!!

[@DarshilParmar]

Thank you

[@AbhinavPeddemoni]

Bro how can we create a code snippets like you showed in thumbnail? Which tool it is I saw many people use this to post their code on LinkedIn also, That one with 3 color dots on top left?

[@DarshilParmar]

Just write code to image on google

[@AbhinavPeddemoni]

@@DarshilParmar thank you so much bro!

[@user-hb2kh3br9u]

Sir can u pls tell me that whether Information and Data Security subject or Essentials of Data Collection and Ethics subject will be the better option for data engineering career. This is an elective subject in my 3rd sem

[@kanishcktewatia597]

can you please share how you create these animations in your videos?

[@sandhya_608]

Plz ,anyone can tell me which CSE specialization should i choose for data engineering ??

[@gauravg2359]

Helpful

[@ezeefied]

Hey mate , great video. Thank you. Would you let me know what software do you use for video editing

[@DarshilParmar]

I used to use FinalCutPro but now I have an editor who edits my videos

[@lectures3727]

Hi bhaiya Needed a laptop in college I chose 2 options hp pavilion plus 16gb ram fixed 1 tb ram fixed i7 13700h rtx3050 6gb 1.01 lakh Ideapad pro 5 16gb ram fixed 1 tb ssd ultra 7 155H rtx 3050 6gb 1.13 lakh Which one should i prefer and if i want to go more low which can i take

[@DarshilParmar]

Don't go HP

[@lectures3727]

@@DarshilParmar bhaiya actually i am very confused on which laptop to take and dont want to overspend I dont do machine learning now but may do so i dont know if i should go for a dedicated graphics card and which processor to go i7 or ultra 7 and ram Can you please suggest me which configuration should i go for and what price range I am getting lenovo ideapad same laptop with 13700h online for 94k but parents are saying not to buy online so i cant decide what to do but i need a laptop now

[@DarshilParmar]

@@lectures3727 If you play games then graphic card is good else just go for other option Basic config - 16GB ram, 500-1TB SSD is good, any latest processor. If you can afford maybe try Microsoft surface or macbook

[@lectures3727]

@@DarshilParmar no i dont play games Can i go for any latest processor I am learning data science machine learning now Will that laptop be enough if i take ultra 7 or i7 I am mostly getting p type processor

[@izhanabdullah]

Which tool are you using to create architecture diagrams?

[@DarshilParmar]

Google Slides

[@kirankumar3536]

Hi Darshil, I have some experience on Azure adf. This combo will help me to go as a azure data engineer or only for AWS engineers. Have you used Azure cloud in this combo ?

[@DarshilParmar]

Yes, we have project in Spark course on Azure

[@kirankumar3536]

@@DarshilParmar what about pipeline implementation by using Azure ?

[@DarshilParmar]

Yes, it's building data pipeline only!

[@mansinayak3360]

Hello guys, Can anyone send me link to the lecture where Darshil sets the AWS pricing alarm. It would be really helpful. I found the where he created the account then IAM user but not the one where he sets the alarm. It is there but I've lost it.

[@mansinayak3360]

@DarshilParmar Hello, tagging you if you can help with the same.

[@DarshilParmar]

Check cloudwatch - docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html

[@mansinayak3360]

​@@DarshilParmar Sure, Thanks!

[@shreyaupadhyay1447]

Sir, I'm a third year cse student. I want to be a data engineer and could you please suggest resources for that. And, do freshers get job in this field?

[@DarshilParmar]

Check out my roadmap

[@mian7104]

❤

[@OppoTeam-u7s]

How much you making per month

[@uzaifquadri]

Is dsa compulsary to crack an product based company , if yes then how much , any suggestions from where to get this skill set

[@DarshilParmar]

I have a video on this kzbin.info/www/bejne/nXLOaqCVmZdqr80si=fJOrV6Gzxa8Ecyov

[@Tanishq_1803]

Bhaiya how it is data engineering professional certificate by ibm on coursera

[@DarshilParmar]

I suggest you to do GCP one

[@Tanishq_1803]

@@DarshilParmar and for data analytics and data science how is ibm data analytics and data science course?

[@hemanthvarupula9873]

Can a person from outside india . Can take data vidya combo course

[@DarshilParmar]

Yes you can - datavidhya.com/combo-course-info

[@hemanthvarupula9873]

@@DarshilParmar and the subscription is for life time

[@DarshilParmar]

@@hemanthvarupula9873 Yes

[@vedantterse8859]

can we get more discount on your combo course for limited period like it is still expensive for students who are just starting off in the field , hope you understand and you'll do consider it

[@DarshilParmar]

I have 50% offer going on, if you want you can start with one course first and then go one by one

[@avinashvyas1597]

@@DarshilParmarI was trying to buy your sql course but there is no discount even the coupon code is also not valid Is there any other way around ?

[@DarshilParmar]

@@avinashvyas1597 Combo Package Python + SQL + Data warehouse (Snowflake) + Apache Spark: com.rpy.club/pdp/yYnEMzLOX?plan=6607b619c69cf00b7b934479 USE CODE: COMBO50 for 50% off Purchase Individual Courses with 50% off: 1. Python for Data Engineering - pay.datavidhya.com/cop/UHfHDhbKVS?code=PYT50 2. SQL for Data Engineering - pay.datavidhya.com/cop/x4UpBcaKsi?code=SQLT50 3. Data Warehouse for Data Engineering with Snowflake - pay.datavidhya.com/cop/BpfGFSahpQ?code=DWT50 4. Enroll in the Apache Spark Course Here - datavidhya.com/courses/apache USE CODE: EARLYSPARK for 50% off

[@avinashvyas1597]

@@DarshilParmar Thank you Darshil Also you’ll be mentoring me now is that correct ?

[@DarshilParmar]

@@avinashvyas1597 There are no 1 to 1 session but we have dedicated discord for this so if you have questions I will help you out

[@sasirugunathilaka3745]

did you had a knowledge on data engineering when you started your internship

[@DarshilParmar]

I had knowledge about ML and Data Science but then I learned DE on internship

[@gadget_comparision_360]

Kya thi cho bhaiband?

[@mayankkrsingh-zq1vj]

Please provide discount in your course . please provide an email id or whatsapp to connect with you

[@DarshilParmar]

50% offer is going on, please check description