83. Databricks | Pyspark | Databricks Workflows: Job Scheduling

[@SqlMastery-fq8rq]

Very well explained Sir, Thank you for sharing knowledge.

[@rajasdataengineering7585]

Thanks and welcome

[@PRUTHVIRAJ-wp9vu]

Sir, Your explanations are very clear & concise. Thank you

[@rajasdataengineering7585]

Thanks and welcome

[@prathapganesh7021]

Simple and awesome. Thank you!

[@rajasdataengineering7585]

Glad you liked it! Thanks

[@FlyingRc_]

Awesome example buddy, Thanks a ton.

[@rajasdataengineering7585]

My pleasure! Glad it was helpful

[@prathapganesh7021]

Awesome video thank you so much

[@rajasdataengineering7585]

Glad you liked it! Thank you

[@rohitwarchali3365]

Hello sir, if our one notebook is performing ingestion tasks from source to sink sequentially. How we can achieve it by doing parallel loading of those tables from source to sink using workflows and jobs

[@SachinGupta-dn7wt]

Great video

[@rajasdataengineering7585]

Glad you enjoyed it! Thanks for your comment

[@avirupmukherjee2080]

Hi Raja, thanks for explaining. Just wanted to check that first cluster you created JOB cluster however later on you created two all purpose cluster. Could you please explain why you have taken all purpose instead of job cluster

[@muruganc2350]

good to learn. thanks!

[@rajasdataengineering7585]

Glad it was helpful! Thanks for your comment

[@oiwelder]

Hello, I really like your series of videos. I would like to recommend doing one on "Integration Runtimes". Database "on-premises" for database in Cloud(Azure)

[@rajasdataengineering7585]

Hi Welder, thank you for your recommendation. Sure, I will create a video on integration run time in ADF

[@maheshchandrabathina1923]

Nice explanation!

[@rajasdataengineering7585]

Glad it was helpful!

[@sourabroy7787]

great explanation . thanks :)

[@rajasdataengineering7585]

Welcome! Keep watching

[@saikoundinya9997]

Hi Sir,Is there any way to skip the task of a job in databricks.

[@baigrais6451]

Thank you for this video. Can I use ADF rather then workflow in databricks? as we can use databricks activity in ADF if i am not wrong.

[@rajasdataengineering7585]

Yes ADF is good choice for orchestration and scheduling

[@vchandm23]

Is it possible to publish these workflow jobs/schedule artifacts across dev to prod as a CICD process ?

[@sravankumar1767]

Nice explanation Raja 👌 👍 👏

[@rajasdataengineering7585]

Thank you Sravan👍🏻

[@pratikshasamindre7004]

Every time we have to change parameters values while running the job?

[@rajasdataengineering7585]

Either we can hard code a value or we can give a logic which generates dynamic values

[@chakradharreddy4481]

Data bricks community edition is supported to workflows or not

[@SATISHKUMAR-qk2wq]

😂 1:19 no it's not supported

[@cantcatchme8368]

How to trigger this workflow from adf?

[@rajasdataengineering7585]

You can trigger only the notebook from ADF. Databricks workflows can be scheduled within databricks itself. Still if you need to trigger from ADF, rest API's are provided by databricks which can be used using ADF web activity

[@cantcatchme8368]

@@rajasdataengineering7585 I need to trigger a notebook which had the program to run the workflows using jobid and other parameters.. I can trigger the base notebook explained above from adf by passing jobid params. Can u pls confirm is this possible? If so how

[@TarakReddy-b7k]

how to pass dynamic parameters in workflow. let consider a scenario. first job is completed and results given some parameter values. how will , i have to use those parameter values to the second job

[@vchandm23]

One hack way is pass the parameters from your first job to your first notebook. Them im your first notebook use RUN command to call your second notebook pass your paramters as args. Hence in that way it is dynamic. Keep injecting the values from jobs. Hope it helps.

[@fortheknowledge145]

Can we create workflow in dev databricks workspace and push it to qa or higher env thru ci cd ? Lets say, azure release pipelines

[@rajasdataengineering7585]

Yes we can create

[@fortheknowledge145]

@@rajasdataengineering7585 could you pls share link or any demo videos if available? I don't see anywhere Only if you have time... Don't worry if you can't due to work. I can totally understand

[@rajasdataengineering7585]

Thanks for understanding. I don't have any video at the moment. But I can create one in future when I get time

[@fortheknowledge145]

@@rajasdataengineering7585 thank you. Thanks for posting a lot of other videos. Great work!

[@kodelapardhu]

How to deploy these jobs in other environments

[@hritiksharma7154]

Great content 👍 .can u create a video on unity catalog setup and explanation ?

[@rajasdataengineering7585]

Thanks Hritik! Sure, will create a video on unity catalog soon

[@vamsi.reddy1100]

Hey, please also create a video on git integration and azure devops...!

[@rajasdataengineering7585]

Sure Vamsi, will create a video on git integration also

[@vamsi.reddy1100]

@@rajasdataengineering7585 thanks man your videos are very helpful

[@rajasdataengineering7585]

Thank you

[@prabhatgupta6415]

hi sir can u create this@@rajasdataengineering7585

[@narayanREDDY-n6d]

Your videos are good, but adding these many ads it's diffcult to watch for watichng 17 minutes, till now only 8 minutes watching more than 5 minutes ads

[@sravankumar1767]

HI Raja, what is delta live tables and what is the importance in Databricks , Why shd we use delta live tables in real time as well as could you please unity catalog and why should we use unity catalog apart from hive metastore

[@rajasdataengineering7585]

Hi Sravan, yes these are advanced and important concepts in databricks. Delta live table is used to create automated streaming data load with declarative approach Unity catalogue is used for data governance. Auto loader is used for efficient incremental data load

[@sravankumar1767]

@@rajasdataengineering7585 in current project we are using workflows apart from notebook activities. Currently we are using unity catalog ,In future we have to use delta live tables