Part 4- Data Ingestion (2) | End to End Azure Data Engineering Project

#azuredataengineer #endtoendproject #azuredataengineeringproject #azureintamil #azuredatafactory #azuredatabricks #azuresynapseanalytics #azuredatalake #datalake #powerbi #keyvault
This is a long awaited video of mine- Lets build a complete End to End Azure Data Engineering Project. In this project we are going to create an end to end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.
The tools that are covered in this project are,
1. Azure Data Factory
2. Azure Data Lake Storage Gen2
3. Azure Databricks
4. Azure Synapse Analytics
5. Azure Key vault
6. Azure Active Directory (AAD) and
7. Microsoft Power BI
The use case for this project is building an end to end solution by ingesting the tables from on-premise SQL Server database using Azure Data Factory and then store the data in Azure Data Lake. Then Azure databricks is used to transform the RAW data to the most cleanest form of data and then we are using Azure Synapse Analytics to load the clean data and finally using Microsoft Power BI to integrate with Azure synapse analytics to build an interactive dashboard. Also, we are using Azure Active Directory (AAD) and Azure Key Vault for the monitoring and governance purpose.
Part 5 will be uploaded soon. Stay tuned.
- - - Book a Private One on One Meeting with me (1 Hour) - - -
www.buymeacoffee.com/mrktalks...
- - - Express your encouragement by brewing up a cup of support for me - - -
www.buymeacoffee.com/mrktalks...
- - - Other useful playlist: - - -
Azure Data Factory Playlist: • Azure Data Factory Tut...
Azure General Topics Playlist: • Azure Beginner Tutorials
Microsoft Fabric Playlist: • Microsoft Fabric Tutor...
Azure Databricks Playlist: • Azure Databricks Tutor...
Azure End to End Project Playlist: • End to End Azure Data ...
Databricks CICD Playlist: • CI/CD (Continuous Inte...
End to End Azure Data Engineering Project: • An End to End Azure Da...
- - - Let’s Connect: - - -
Email: [e-mail]
Instagram: mrk_talkstech
- - - Tools & Equipment (Gears I use): - - -
Disclaimer: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide, I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!
DJI Mic: amzn.to/3sNpDv8
Dell XPS 13 Plus 13.4" 3.5K : amzn.to/45KqH1c
Rode VideoMicro Vlogger Kit: amzn.to/3sVFW8Y
DJI Osmos Action 3: amzn.to/44KYV3x
DJI Mini 3 PRO: amzn.to/3PwRwAr
- - - About me: - - -
Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN ABOUT THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"
I will be creating playlist which covers the below topics (with DEMO)
1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)
After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,
1. Data Engineer
2. Data Analyst
3. Data Scientist
Can't wait to help people with my videos.
- - - Support me: - - -
Please Subscribe: / @mr.ktalkstech

[@parullnpriyaadarshani8512]

I must say. This is the best data engineering project. Thank you so much for all your efforts and great explanation.

[@mr.ktalkstech]

Thank you so much :)

[@anithaveeramani8909]

Great Video Mr.K. I am completely new to Azure, But I am able to follow since it is well explained in short. Thank you!!!

[@prasanthbugga8381]

Honestly, You are a legend...its just awesome, the way you are teaching is superb...i was into GCP and came across this project, with this single project, you are making me to choose between azure and GCP, thank you so much Mr.K

[@mr.ktalkstech]

Thank you soo much :)

[@prabhatgupta6415]

Good Going plz bring more projects

[@girijasahu7409]

great example end to end project. This is very useful for any Azure ADF projects.

[@selairendri9997]

Excellente Course

[@Mark_Trail]

Great stuff! I'll keep watching!

[@anjibabumakkena]

Great explanantion sir, Thank you very much for providing these videos

[@user-rm7yn7us8q]

Thank you so much for all your efforts and great explanation. Plz bring more projects.

[@pritidongarwal739]

Very good video . I like the short video and it gives an idea of the key steps to be followed.

[@chubsmash7602]

Great video, simple and short, and yet covering pretty most of the important things you need to migrate to Azure.

[@ChanceMinus]

Outstanding. Thank you.

[@helovesdata8483]

The best! I'm coming from AWS and my new job needed help with an Azure initiative. Your videos are helping me to get up to speed quickly.

[@mr.ktalkstech]

Glad to hear that, All the best :)

[@lifeitnow9600]

So much of confidence while you deliver the content!! truly a note to make..... thanks for putting up the efforts to build the azure playlists... keep going!!! Need more such videos

[@mr.ktalkstech]

Thank you so much :)

[@user-qt8bg9ov6e]

This is awesome. I’m glad I chose the Azure cloud.

[@mr.ktalkstech]

Thank you so much :)

[@SuperAnurag121]

Awesome

[@4nxh___]

I appreciate your great effort and time given to spread Data Engineering-Project knowledge. Well Delivered!

[@mr.ktalkstech]

Thank you so much :)

[@gaganaulakh41]

The best content I have ever seen so far on youtube!!!!!

[@mr.ktalkstech]

Thank you so much :)

[@90harvinder]

The best video on ADF i have seen. Bravo.

[@mr.ktalkstech]

Thank you so much :)

[@kirtiagg5277]

wonderful work..Thank you so much for good explaination !!

[@mr.ktalkstech]

Thank you so much :)

[@shashibhushansingh1628]

great man now iam feeling higly confident in azure thanks man

[@mr.ktalkstech]

Glad it helped you :)

[@ericbernal8714]

Great job! bring more projects.

[@mr.ktalkstech]

Thank you :)- sure, will do.

[@user-wx7os2zo8z]

hi sir please upload next videos asap, and do more these kind of videos, thank you for great teaching.

[@manjunathbn9513]

Great video. Shared in whatsapp group. Pls add all videos asap. helpful to many people. thanks.

[@mr.ktalkstech]

Thank you so much :) Sure, will upload the rest asap

[@sraoarjun]

This is such a good presentation and to the point crisp explanation !!

[@mr.ktalkstech]

Thank you so much :)

[@easypeasy5523]

You are simply amazing bro thank you for these content this such a premium level

[@mr.ktalkstech]

Thank you so much :)

[@GloryBharat]

Very well delivered. Super impressive and helpful. Love the passion with which you deliver. Very easy and informative to understand. Keep it up

[@mr.ktalkstech]

Thank you so much :)

[@salmashaik6294]

One of the best explanations ever. Really liked this series

[@mr.ktalkstech]

Thank you so much :)

[@salmashaik6294]

@@mr.ktalkstech Can I know if the same project applicable for incremental data?

[@user-of1br4hd8m]

Very Good Explanation, please come up with more Real time projects which will be helpful. May the coming year i'll reach my short term goal with your teaching and support.

[@mr.ktalkstech]

Thank you so much :) Sure :)

[@RohanKumar-mh3pt]

very amazing can you make one video for Full and Incremental Data load for Ingestion part?

[@noufalrijal9811]

To be honest, the presentation is quite straightforward to follow, even for someone like me who comes from an AWS background. I would like to know if there is a method to create this without using drag and drop and instead use code.

[@pridename2858]

Wow!! excellent sir,this is called teaching...where teacher is able to inject the the real materials in the students mind. hats off for you sir. just curious if you have covered, how we will get SSMS on our laptop, so we can use the sample DB to injest in bronze. Thanks

[@mr.ktalkstech]

Thank you so much :) Please check the description of the video to import the sample DB :)

[@ranjansrivastava9256]

Dear Kishor, One suggestion here is :- Here you have written the SQL Script to get all the lists of tables using the sys tables and sys schema, it's really works well but suppose QA/Developers create similar kind of tables for their testing we can not control by this approach. So My opinions are [a]. We can create a table which has table name, table schema, active flag and will create SP that will fetch all the active tables by using the Lookup activity and will do the further activities. [b]. We can create json file with table names and schemas and upload into the ADLS and we can read from there using the lookup activity. Kindly correct me

[@mr.ktalkstech]

The synapse views will be automatically reflected as long as there is no schema change happening in the source.

[@centebatech]

This looks very similar to ETL looks like Informatica? Why would you use this instead of a standard ETL tool that is portable if you decide to not use Azure and want to go to, say, AWS or GCloud? Thanks

[@sidsan000]

Thanks a lot its really a good hands on experience. Right now I have two issues, can you pls help on this 1. in Azure blob storage a o byte file is getting created every time when a folder is generated. 2. My Cluster is not getting up. can you pls help on this

[@nilgiripaiya]

very w well explained, Could you please complete the complete project flow video

[@mr.ktalkstech]

Thank you :) will do it asap

[@centebatech]

What is the language you are using for all the macros and scripts you are writing as you build out the pipeline? Do you have a tutorial for that? Nice job, by the way!

[@mr.ktalkstech]

It's called as ADF expressions. I haven't posted any videos specifically for that, but you can see the below Microsoft documentation to get familiar with that, cheers :) learn.microsoft.com/en-us/azure/data-factory/control-flow-expression-language-functions#functions

[@hritiksharma7154]

Hii , your videos are great. Can you please create a video on unity catalog in databricks

[@mr.ktalkstech]

Thank you :) Sure, will do in the near future.

[@user-fe4gm3qv1o]

Hello.. its a great learning video... I have 4 yrs gap & i was stuck to choose Salesforce vs data engineer field. as many say data engineering is hard & complex tasks involved. does all the azure data engineer projects follow the same/similar kind of steps as you have explained in this whole project

[@pridename2858]

Hi User, I would say, keep all the things done in this project in mind a rough Idea and watch another e-to e project. They are similer, sometime tools can be different...so I would say its a great start. and solid start.

[@ranjansrivastava9256]

Dear Kishor. Keeping performance in mind , up to what maximum number of tables can be ingested from source to destination with more data coming in the future. How can we handle this situation. I am asking because I am facing the same issue, might others benefited as well from this.

[@mr.ktalkstech]

It depends upon the volume of each tables, mostly determined based on the trial and error method :)

[@anjibabumakkena]

Kindly provide the Realtime scenario series about pyspark

[@mr.ktalkstech]

Sure :)

[@zuhailiaminuddin520]

I got this error message when trigger the pipeline 'Operation on target ForEach1 failed: Activity failed because an inner activity failed' do i need to provide actual value of the parameters to preview data of @item().SchemaName and @item().TableName in the source

[@zuhailiaminuddin520]

ok i know where i do wrong @{concat('SELECT * FROM^', item().SchemaName,'.', item().TableName)} in this syntax i should put 'space' indicate by '^' .. i discover this when i see the input of the fail trigger activities

[@naturetime5266]

can we upload data manually? instead of connecting with SQL server!

[@mr.ktalkstech]

Yup, you can do that- if you have CSV files then you can just upload it to the RAW zone and then process it using Databricks

[@shiinobeats1329]

I'm getting pipeline fails due to this errror ErrorCode=InvalidTemplate, ErrorMessage=Unable to parse expression '(concat('SELECT * FROM ', item().SchemaName, '.', item ().TableName)}'

[@mr.ktalkstech]

should be something to do with the syntax based on your comment, I am seeing a space between item and () , it should be item().TableName. Please have a look if there is a space between item and ()

[@shiinobeats1329]

@@mr.ktalkstech thank you for responding. I’ve tried everything, is there somewhere I can get the original syntax

[@mr.ktalkstech]

This is the original syntax that I used in the demo, @{concat('SELECT * FROM ', item().SchemaName, '.', item().TableName)}

[@ranjeetkumarkushwah6750]

Hi Sir, I am geeting the error like " incorrect syntax near 'fromEmp,},],' Emp is my schema name

[@mr.ktalkstech]

Should be a typo in your script, did you fix this?

[@ranjeetkumarkushwah6750]

@@mr.ktalkstech Hi Sir, I had tried many times but each and every time i got the same error "Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Incorrect syntax near 'FROMEMP'.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Incorrect syntax near 'FROMEMP'.,Source=.Net SqlClient Data Provider,SqlErrorNumber=102,Class=15,ErrorCode=-2146232060,State=1,Errors=[{Class=15,Number=102,State=1,Message=Incorrect syntax near 'FROMEMP'.,},],' "

[@ajinkyaadhotre5336]

next video when ??

[@mr.ktalkstech]

Will upload it tomorrow :)

[@charles-sambo]

@@mr.ktalkstech Thanks and looking forward

[@ajinkyaadhotre5336]

@@mr.ktalkstech ok , Thanks :D