Azure Data Factory [Full Course] 💥

[@pratikparbhane8677]

00:01 Learn about Azure Data Factory architecture and components. 02:14 ETL process involves extract, load, and transform from different data sources 06:42 Azure Data Factory activities and components 08:58 Creating self-hosted integration to move data from SQL Server to Azure space. 13:22 Azure Data Factory components overview 15:24 Select storage account name and configure performance and redundancy 19:33 Creating Azure Data Factory in Azure portal 21:44 Launching Azure Data Factory Studio 25:36 Moving employee data with Azure Data Factory 27:33 Creating a data pipeline in Azure Data Factory Studio 31:26 Validation and publishing steps in Azure Data Factory 33:26 Performing a delete activity in Azure Data Factory 37:18 Setting up data source and connection in Azure Data Factory 39:22 Performing data transformation using Azure Data Factory 43:31 Creating a data flow in Data Factory Studio 45:24 Exploring data preview and filter transformation in Azure Data Factory 49:45 Creating and running pipelines in Azure Data Factory 51:54 Understanding joint and union transformations 55:46 Configuring source and data preview 57:41 Performing inner join to select matching records from both tables. 1:01:55 Using Union and Join transformation in Azure Data Factory 1:04:35 Stored procedures in SQL for efficient query execution 1:08:36 Creating link service and configuring store procedure parameters 1:10:47 Demonstrating the use case of deleting an ID during runtime 1:14:51 Creating a pipeline to copy data into output container 1:17:03 Using Get Metadata and For Each Activity in Azure Data Factory 1:21:24 Setting up sink for data storage in Azure Data Factory 1:23:27 Using lookup activity to retrieve data for pipeline transformation. 1:27:40 Creating a store procedure to delete employee IDs 1:29:49 Configuring activities in Azure Data Factory 1:33:43 Using Get Metadata activity in Azure Data Factory 1:35:57 Understanding the Get Metadata activity in Azure Data Factory 1:39:56 Check column count to determine file data presence 1:42:10 Setting dynamic file name and checking column count in Azure Data Factory

[@learnbydoingit]

Thanks alot

[@pratikkadam4223]

Thank you brother. This really helps. I request you to create many such "full course" videos. They're a better one stop learning solution. Wishing you luck in your KZbin journey!!

[@OmolaraFash]

This is indeed a full course, thanks you for your effort🙏, God bless you

[@vijayandela]

The best tutorial i saw on ADF. really great stuff, thank you

[@tesfayemashka5167]

This is good to understand than many other videios in this area. i like it Thank you .I wish best for you and Keep it up.

[@sivmoorthy]

Your training is awesome. Appreciate your efforts

[@olunugaoladipupo7058]

Found this really helpful, covers a lot yet precise and straight to the point. Thank you.

[@keerthivasan_Dataguy]

Best refresh before interview ... on point in every topic

[@shreyapandey9735]

It was really a very understanding video..understood all concepts. Thanks buddy.really helpfull🎉

[@ANILKUMAR-in3dp]

Simple, Concise and easy to understand. thanks, you so much.

[@learnbydoingit]

Thanks Pls like

[@nandeeswarareddy4680]

it's really helpful full, short, and clear. Thanks a lot I am learning a lot from your channel.

[@yashwantdhole7645]

Hello brother. I completed a 15 hours course from Udemy(from which I barely understood anything). But, seeing your video, everything looks very easy and clean. You actually are giving a lot of confidence to newbies like me. Lots of gratitude and love! ❤❤

[@learnbydoingit]

Do follow adf playlist u will get more ideas

[@Lakshvedhi]

Great channel i have found to learn azure. Thank you bro

[@bekalutadele2048]

Really I like your Tutorials way of delivering ,contents and examples Keep in touch dear. i am waiting other tutorials on Snowflake in easy way like ADF.

[@RajendraK-b8s]

Really Helpful sir simply understand Thank you .

[@RajithaReddy-sk5he]

One of the best explanations. Thank you so much and I subscribed to your channel. please make a video of ADF interview questions also

[@alareqi2023]

it is the best video I have seen so far, thanks a lot. I have a question please, when you created the stored procedure linked service you used the sql server name, could you please mention from where can i get it?

[@shirazhussain7965]

Best video available on ADF

[@nitindba]

great introduction to ADF .

[@gaurie1030]

fantastic, but in if condition, when i click on expression, i am not getting Column Count to select as dynamic how to work on it?

[@tulsik4811]

Hi, I have an Excel file used as a data source in Data Factory, where I encountered a challenge with multiple rows for column headers (Nested headers). Specifically, I aim to consolidate two rows into a single row to streamline the header columns. how can i do that

[@josemanuelgutierrez4095]

I have a question why in the foreach part min (1.24.05) the output result of all the files is TXT when everything should be csv because you select CSV in the output or I'm wrong ? . thank you

[@swarooppabbineedi1820]

Wish you become more popular bro.. Keep posting more contents like this..

[@kishoreevani5813]

Thank you so much for sharing this video. Keep doing wonderful videos like this. Good luck 🎉

[@NivasReddyMunnangi]

Could you please tell me where we will find the datasets for this ADF course.

[@learnbydoingit]

Pls do follow latest video playlist ..you will get complete

[@anonymous-254]

Thanks for this video ! Can you please make one video on industry level ADF project

[@piyusworld2550]

Its really helpful.

[@oluwasegunosinaike]

please always attached the files to use like excel or cvs document. thanks for the lesson

[@dailylearn4326]

bro thank you so much for beautiful video on ADF.. While using dataflows for filter and sort i am getting this error is there any video regarding this error.... "Dataset is using 'AzureSqlDatabase'linked service with SQLversion V2 type, which is not supported in data flow."

[@learnbydoingit]

Please do follow kzbin.info/www/bejne/mX-mapiIi7efiMksi=DZyUEmQyPTy_I3B6

[@neuera9556]

hi ru a working data engineer as many institutes online youtubers just mug up from other online and post videos half knowledge videos and mis guide im starting from today to follow your series please reply all this one video playlist is enough to get job in data enginerr and get certified thanks i like your teaching thats y

[@learnbydoingit]

Hi just follow u will get complete idea even we are covering interview also... Please watch azure data engineer playlist... And also sql playlist

[@neuera9556]

@@learnbydoingit ok thanks

[@ParixitC-z3x]

Very Helpful

[@gaurie1030]

Amazing, fantastic as always, can you please provide the datasets used for us to practice as well ?

[@Mohitkumar-hr4hy]

Thanks bro ..very nice work 🙂

[@tuneinsight8497]

Hi sir , need your help regarding creating expression, Eg : i want to create a new column with value being set based on multiple condition like using multiple if else in python

[@learnbydoingit]

Use derive column

[@BewithuNnk]

Tq Can you please advise me, is it required to know azure cloud concepts to start to learn ADF. ? I've exp in SQL pow BI

[@learnbydoingit]

No with this practical you will get idea about adf

[@manindersingh6371]

Nice video really helpful

[@mosesademola4963]

I'm new to this Azure Data factory and setting up a free account. Could you please explain how I can create a free account. Also, it mentioned a 200 dollars credit. I don't seem to understand the impact of that 200$ credit on the free account. many thanks

[@learnbydoingit]

You can use freely for 1 month and for that amount they have mentioned

[@mosesademola4963]

@@learnbydoingit Thank you very much.

[@ashutoshnayak6242]

Keeping public end point is good or we should keep it private while creating the azure data factory? Please tell.

[@learnbydoingit]

Public is only for practicing but real u have to make private for security

[@ashutoshnayak6242]

​@@learnbydoingit yes but I am the lone person of my organisation is using azure data factory so public will be ok , whats your opinion. Thank you

[@learnbydoingit]

@@ashutoshnayak6242 for org it's private

[@ashutoshnayak6242]

@@learnbydoingit ok

[@teluguallrounder9465]

lot of thanks

[@umaadvitha24]

sir pls do full databricks and pyspark course

[@yuvarajgopi98]

can I get dataset csv files which you are using in the video.?

[@learnbydoingit]

Bro follow latest playlist adf complete u will get better idea and approach ...dataset also mentioned there in description

[@yuvarajgopi98]

@learnbydoingit what's the playlist name?

[@yuvarajgopi98]

Thanks I will check.

[@omkarm7865]

Thanks buddy for this

[@kirtichavan4754]

Good one

[@prititiwari8156]

Thanks

[@gaddipatiprabhakar2889]

Good Videos... Thank you

[@Lakshvedhi]

Bro please make video on what required to get first job as data engineer in Azure topic.

[@learnbydoingit]

Sure

[@sailatachalla9032]

@@learnbydoingit also please do videos on Azure synapse and databricks

[@learnbydoingit]

@@sailatachalla9032 ok

[@roshangangurde7461]

Thankyou for this wonderful video.i wnat try these things on my own. Are these services are available in free tier subscription?

[@learnbydoingit]

Free for 30 days

[@rakeshkoli4384]

thank you so much bro it means a lot to us and save the 30k for paid class

[@learnbydoingit]

Do follow latest playlist...

[@suprajachinnu8096]

Is this video enough for learn azure data factory for search of job??? Can anyone please let me know

[@learnbydoingit]

Yes you will get idea about in adf how we create data pipeline different transformation..after this pls do project available in channel

[@gauravmane9287]

is this is complete process for doing etl testing on cloud ? please guide me ,i want learn etl testing on cloud

[@learnbydoingit]

Yes u will learn about etl

[@neeraj_dama]

thanks for this

[@munimudim6487]

Nice

[@premsoni619]

is it a full course?

[@dineshbaskaran3135]

❤

[@balajikomma541]

kindly add timestamp for this video

[@learnbydoingit]

Will add