Azure : Data Factory and DataBricks End to End Project

In this Project we will cover end to end Movie recommendation system using Spark ML, which will be implemented in Azure DataBricks and Azure Data Factory. At the end of the this processing we will send recommended movie to our end user in Gmail using Azure logic apps.
0:00 Intro
2:50 Azure Resource Group
7:55 Azure Storage
22:15 Azure Databricks
33:20 Creating Keys for Mounting
Dataset grouplens.org/datasets/moviel...

[@Haribabu-zj4hd]

Very nice video by covering all the important aspects as part of ADB and ADF integration.

[@lbb2rfarangkiinok]

0:00 Intro 2:50 Azure Resource Group 7:55 Azure Storage 22:15 Azure Databricks 33:20 Creating Keys for Mounting will update soon when I finish. Thanks for this.

[@azimsayed1971]

great knowledge Mate what i understood so basically ETL Extraction of data from different sources,. transfrom them using databricks or dataflow etc and load it to a sql pool or DWH for further analysis using power bi right ? extenting further content after creating mount function we create schema and put it in the data frame, transfomation then will load it into SQL DWH right then Power bi mate can you also tell me what the delta lake please

[@maq6246]

pin this comment @author

[@lopokizito]

@@azimsayed1971 Quite some time later, but if you have any responses by now I'd be glad if you could share them :)

[@sachinmore8938]

Greate Video, it's difficult to explain so many topics in a short time, but you did your best. Thank You!

[@PhongTran-re9kp]

Greate video, You can work with serverless seri?

[@dataengineeringforeveryone]

Sure. Which services you want me to cover ?

[@varun8952]

Awesome explanation. Very Helpful

[@sabihahmad2138]

Amazing Nilendra !!! It really helps..Keep up the good work

[@dataengineeringforeveryone]

Thanks :)

[@cloudandsqlwithpython]

Great work sir

[@rameshakustagi5189]

Really very good video, thanks a trillion

[@junaidmalik9593]

Thanks so much fo the efforts ,really helps a lot. Can u also provide the dummy data so we can also work while watching the tutorial

[@ramanjaneyan5831]

Thanks bro. Very helpful for beginners.

[@dataengineeringforeveryone]

Thanks :)

[@turanfair9364]

Thank you Sir 🙏🏻

[@maiga4244]

Hi, thank you very much for this project. it was really helpful. Please where can I get the datasets used in this tutorial for self-practice?

[@kirtichavan4754]

Good and very helpful

[@3rdEyeSpiritual]

Very nice. .Learning end to end project like this is very useful, thanks for making this kind of video, very useful...greatly appreciated. If possible please share all the code and tables etc ..used in this video , so that i can practice at my local laptop...and let me know if you are giving any training on same ...sen dme website details if you are giving any training . thanks .

[@sanjivranjan5782]

Nice explanation.

[@dataengineeringforeveryone]

Thanks:)

[@ashokbanala7149]

Thanks so much for the efforts ,really helps a lot. Can u also provide the spark code so we can also work while watching the tutorial

[@Marchelo005]

Is there any way to share results between different databricks notebook activities in the same pipeline of azure data factory?

[@vishal259]

Great video, I would like to replicate your project for deeper understanding. Could you provide any git path with your notebooks if possible.

[@ranjansrivastava9256]

It's a really very good demo, Kindly share the code and one more thing, please share the ADF demo separately with step by step progress ! That will help us in better way. Thanks a lot...

[@jayantbishnoi9617]

did you find the data bricks code?

[@ranjansrivastava9256]

@@jayantbishnoi9617 No sir, Kindly share the path of possible.

[@SachalChandio]

you are awesome

[@dataengineeringforeveryone]

Thanks:)

[@azimsayed1971]

great knowledge Mate what i understood so basically ETL Extraction of data from different sources,. transfrom them using databricks or dataflow etc and load it to a sql pool or DWH for further analysis using power bi right ? extenting further content after creating mount function we create schema and put it in the data frame, transfomation then will load it into SQL DWH right then Power bi mate can you also tell me what the delta lake please basically iam very new in this field data engineer using adf it would be nice if you tell me how can i get more information in-order to crack an interview like delloite etc please much appreciate

[@user-wg4bh3rv5i]

Did you get the job in this field

[@devcamp3582]

Please provide the spark notebook. Everything else is crystal clear. Thanks

[@surenderraja1304]

Is Azure Data factory service also enabled in free azure account for 1 year?

[@TradeTacticsAcademy92]

Hello ,it is possible to obtain the code to this projet? and the DataBase used?

[@All_In_One_By_Vinay]

can you share those files which are used in this project, convert it into a zipfile and share with us sir.

[@pranavchakne6377]

in this when i tries tolist my resource it says i dont have access to the storage

[@anonymous-254]

What we will do after this ... ?? Can you explain the whole process till PowerBI... please

[@yashnegi9473]

It is not showing mounting option in my case.

[@user-de6qh6uj3r]

how ratings.csv looks like? Its not shown in the video

[@uwefuchs1414]

41:34 I recieved a 403 error, make sure your SP has 'Storage Blob Data Contributor' access on the storage account. I orginally had Owner, but had to add this for it to work

[@satishgummadi6022]

what is SP?

[@satishgummadi6022]

I have tried going to IAM under my ADLS account and provided 'Storage Blob Data Contributor' access but still i am getting 403 error. What should I do?

[@valentinloghin4004]

Nice , can you please provide the resources for this tutorial ? Thanx

[@HemantKumar-su1qt]

Hi sir Hope you are doing well I am an enthusiastic fresher data engineer. I want to create a data engineering project by taking a one month free subscription on Azure Cloud and show that project on my resume. If my one month free subscription on Azure Cloud expires and the resources get exhausted, will my data engineering project disappear or I will not be able to see it? Can I still show my data engineering project on my resume and the company can see it even after my one month free subscription on Azure Cloud expires? Thank you so much

[@sreenair2168]

NICE DEMO can you pls..share the notes

[@matlowe99]

has anyone got a copy of the notebook?

[@SanjayKumar-rw2gj]

The explanation is very good and clear but as you did not provide the databricks notebook it is not going to help viewers because we learn and understand better through practical.

[@Nikhilsharma-dj2jw]

hey everyone, can anyone tell will this project cost me something?

[@jayantbishnoi9617]

does anyone have the spark code? please share

[@mysteriovvn]

Hello, i have a payasyou go azure account, to complete this project approx how much will it cost?

[@dataengineeringforeveryone]

Hey shouldn’t be much. Less that 20 bucks. If used wisely

[@mysteriovvn]

@@dataengineeringforeveryone 1600/- INR approx right?.. Thanks for the reply

[@sravyamorisetty4326]

Where can I get the mounting script?

[@mathiaslongl]

did you find it?

[@mithunnambiar1433]

concepts are not clear, you are not explaining the details...just here and there....instead of doing from scractch..he is explaining without the base....

[@ADAMSIVES]

Great BUT we need the files please

[@maq6246]

can I get that one note pls ❤️

[@mehul4mak]

All these steps can be done within Databrick only the. Why you have to use these many services and what benefits do they add? Secondly, where is the trained model and how one can consume notebook as proxy of ML model?

[@dataengineeringforeveryone]

This purpose of this project is show how Azure Data Factory works with other Services. Yes it can be easily done in scala-spark etc.

[@the_engineer17]

Kindly post the data set link in the description

[@dataengineeringforeveryone]

grouplens.org/datasets/movielens/

[@jayantbishnoi9617]

PLEASE SHARE THE MOUNTING CODE! without it im unable to complete the project and there is no point of the rest of the video. please help

[@dataengineeringforeveryone]

Hey. Sure let me find that for you. I am travelling right now. But will get the code soon.

[@alwaysbehappy1337]

Please share the data bricks code

[@snehitvaddi]

Hey brother! Why not share the dataset just so we can get some hands-on experience.

[@dataengineeringforeveryone]

grouplens.org/datasets/movielens/

[@snehitvaddi]

@@dataengineeringforeveryone Unfortunately, the dataset is not accessible. Please check and resend. In addition please also share the Jupyter file (Databreicks file) which has code.

[@jayantbishnoi9617]

@@dataengineeringforeveryone hey bro I have tried replicating the spark code it's getting very frustrating can you please share the spark code

[@mathiaslongl]

can anyone share the code?

[@praneethjeevan2173]

Please share the documents

[@bandarurohithkumar439]

How to contact you?

[@ramswaroop1560]

Worst Explaination ever... If you want to explain hurry burry Why to make videos..... Datafactory side not even explain anything properly ( get metadata schema parts ) ..... Prepare how to explain..before making video

[@devthakkar4613]

while running the code at 41:18 I received this error java.lang.NullPointerException: authEndpoint Can you please help me

[@maryemnjm4522]

where did you find the code ,?

[@SeattleDataGuy]

Thanks for putting this together. Keep it up!!!

[@muditasharma4074]

Your teaching way is just amazing. You really ease out the topics to understand. Thanks for this outstanding contribution!!

[@dataengineeringforeveryone]

Thanks Mudita for liking the content ! Please share it on LinkedIn too!!

[@anoopsidhu3437]

Great video on introduction on data bricks and data factory. Very clear and concise amazing job. Keep putting these kind of videos for the community thanks so much

[@dataengineeringforeveryone]

Thanks Anoop for your kind words!

[@gauraangsharma2665]

Can you please share the code ?

[@nileshyadav7543]

can i create this project in azure 1 month free trail

[@dataengineeringforeveryone]

I think most of the project can be completed except Databricks tier portion.

[@nagaleelakrishna1561]

Sir please help me with mountpoint. Where can I find mountpoint path. I'm getting the error

[@bharathkumar-hu9cb]

It is a great effort Thank you So much...please make videos end to end projects and deployment process.

[@dheerajkrishna6459]

i may lost my job without this video

[@HukoMoeller]

The copy activity does not preserve the ACL permissions for the files, causing the notebook to fail when it tries to read with the registered app. Has anyone else had this problem?

[@shashankshukla3361]

Hey! Great project. Can you please provide the notebooks as well?!

[@ankan54]

hi, I am not able to create scope, error is Premium Tier is disabled in this workspace. Secret scopes can only be created with initial_manage_principal "users".

[@nagaleelakrishna1561]

Sir could you please tell where can I get those files, datasets?

[@dataengineeringforeveryone]

Search movie lena dataset on google. Its publicly available.

[@nagaleelakrishna1561]

Thank you so much

[@jibejay5357]

great KUDOs to this video!!

[@patrickbateman7665]

Hey, Big Thanks for end to end project video❣️ I really appreciate your efforts 😄 But I do have a question! Do Data Engineers has to build these models? I thoght Data Scients and ML Engineers will take care of building models. Sorry If my question sound stupid.I am quite new to this domain.

[@dataengineeringforeveryone]

Very valid question Dileep. And yes you are correct. But in this project I tried to cover every phase of the project .

[@swamik11]

Great effort!! I think App Registration is nothing but a Service Principal.

[@anilreddybandi8593]

How can we decide cluster size , if i have around 500TB data?

[@sooriyathewhistler5933]

Great effort !! This is the need of the hour for beginners and you made it easy in short time. Kudos to you 👏

[@srikantha7290]

Hi Sir I'm planning to learn Azure data factory . Please suggest how to contact

[@FocusOnU10]

Thanks for the Video. It helped me alot to understand the Databricks

[@federicogonzalez7673]

New subscriber, nice to meet you!

[@ganeshvairagar_4555]

What is different technologies azure service were used in this project

[@Pacal_II]

So what is the purpose of the ML script? It looks like you're just recommending the user the movies they've rated the highest?

[@raghavajagannatham3328]

What different azure technologies were used here???

[@himanshukhandelwal4460]

Hi, I get the complete dataset and your notebook code ?

[@dataengineeringforeveryone]

Hey dataset is publicly available. MovieLens dataset.

[@richrass]

@@dataengineeringforeveryone Hi, thanks for the content but how do we get the python notebook default code?

[@akhilrajt618]

Hello Sir, This is Amazing work...Do you provide private classes, I wanted to discuss few points as I want to build something similar from scratch...

[@lbb2rfarangkiinok]

Would also be interested.

[@mustafakamal5945]

Thanks for this, this was super helpful !!!! Everything was crystal clear, except for 1 point, the app registration ? Why it is used. I believe the Azure Key vault is enough for establishing a secret connection between ADLS and the ADB notebook. Please help me understand this..

[@dataengineeringforeveryone]

I was trying to cover as much topics as I can. Yes it can be done without app registration too.

[@michaljanecek1103]

Why don't you publish the source code + files?

[@mathiaslongl]

did you find it?

[@nanireddy6742]

Hi sir can i have your contact i need support

[@dataengineeringforeveryone]

[e-mail]

[@reach2puneeths]

please come up with more Azure data engineering end to end projects

[@dataengineeringforeveryone]

Sure Puneeth.

[@sujithkumar804]

@@dataengineeringforeveryone I m sorry but I got lost after starting ADF ,no clue about datasets used in get metadata activities , type of validation used(How the data is validated) ? Without that couldn't finish project . Can you please share those details ?

[@reach2puneeths]

Great video on end to end project. really appreciate your efforts. could you please upload the databricks notebook to git or provide link to download.

[@ritujain3656]

Great sir...

[@veerasaisundhar7458]

Nice video 👍

[@sampathch1117]

Hi your teaching amazing,please share azure and databricks documentation link