87. Databricks | Pyspark | Real Time Project: ETL Pipeline Integrating ADF, ASQL, ADLS, Key Vault

[@satijena5790]

Nicely explained the simple real life scenario. It gives lot of learnings e.g. how to call a Databricks notebook from a ADf pipeline (parametrised), integration with AKV, Using connections to ADLS as well as SQL DB with complete clarity. Since, i did see the previous videos of this channel, i could understnad it end to end. Very eager to do hands-on once i get an dev enironment. Great job. Many thanks.

[@rajasdataengineering7585]

Glad it was helpful!

[@Elkhamasi]

You are a great teacher.

[@rajasdataengineering7585]

Glad you think so! Thanks

[@kranthikiran8318]

This guy needs more recognition

[@rajasdataengineering7585]

Thank you

[@sravankumar1767]

Superb explanation Raja 👌 👏 👍

[@rajasdataengineering7585]

Thank you Sravan👍🏻

[@rambevara5702]

Brother I have been waiting for this video..I am devops engineer need to explore and prepare adf adb SQL integration..this video helped me lot..my request please put devops engineers in your mind and do simple video with python or SQL rather pyspark as it is not familiar for us..many thanks brother

[@rajasdataengineering7585]

Thanks Ram for your comment. Glad it helps devops engineers too. Sure will make simple videos

[@rambevara5702]

@@rajasdataengineering7585 thanks brother

[@rambevara5702]

@@rajasdataengineering7585 where we can find that example notebook you have used for this video..

[@neerajsharma-zt8jo]

How have you configured SQL server table details in sink for load to log table before mapping the columns.. Can you share more details there.. We are unable to get insert option there..

[@rajasdataengineering7585]

We are just writing the dataframe output into SQL table using jdbc connection

[@sivag98]

Hi Raja.. Your videos are awesome...Thank you so much... could you please make Databricks delta table real time projects videos and incremental load full load pipeline related videos... Thanks advance

[@rajasdataengineering7585]

Sure Anand, will make delta lake real time videos

[@ravilikhar007]

@@rajasdataengineering7585 Hi Raja, Please make video on delta live real time project. Thanks. Keep doing good work.

[@navalsaini3229]

Nice explanation kindly make an video on stream processing via spark using stream processing as well

[@rajasdataengineering7585]

Sure Naval. I have already created one video on streaming kzbin.info/www/bejne/sKmYeWtom9KIhrc Will also make more videos on advanced streaming exercises

[@abhimanyukakde849]

Hi, could you please make a video on aws ETL Pipeline in real time use case

[@rajasdataengineering7585]

Sure, will make a video on this requirement

[@mission_possible]

Some of the videos are showing as unavailable can you please look into it?

[@rajasdataengineering7585]

Hi Noor, sure I can look into it. Could you please share one of the video topic for which you face such issue

[@mission_possible]

@@rajasdataengineering7585 when I open data bricks playlist there it is showing as 4 unavailable videos

[@rajasdataengineering7585]

Ok, I will check

[@mission_possible]

@@rajasdataengineering7585 Thank you

[@shankardonthireddy9422]

Hi, could you please make a videos on Delta live tables if possible?

[@rajasdataengineering7585]

Sure Shankar, will make videos on DLT soon

[@ManukaDeepika]

databricks using pyspark do more vedios

[@rajasdataengineering7585]

Sure, will create more videos

[@Mehtre108]

Could you pls upload one real time project with experienced people sir

[@suresh.suthar.24]

can we run the community edition notebook using ADF?

[@rajasdataengineering7585]

No, community edition is not supported

[@ManukaDeepika]

Can u show how to install databricks and datalakes demo everytime we can learn easily one more real project show me