124. Databricks | Pyspark| Delta Live Table: Datasets - Tables and Views

[@user-uh2cs7uy7r]

I am following your content for some time. Looks like , your channel is one "best kept secret"...it deserves many more views and subscribers

[@rajasdataengineering7585]

Thank you so much!

[@abhisekmondal4076]

Your Databricks series is GEM .

[@rajasdataengineering7585]

Thanks!

[@manikanta-zq1yg]

thank you for all your efforts😊😊

[@rajasdataengineering7585]

Glad you like it! Thanks for your comment

[@amazhobner]

Hi Raja, this was a wonderful series and I learned a lot of things in depth. Thank you for doing this for us..!! :) If possible can you also make videos on below topics as you get time: 1. Implementing SCD2 using pyspark. 2. Handling incremental files using pyspark. 3. Like your video on .explain() please make a video to help us understand all components of spark UI as well as the DAG. 4. Please make 1 video where you go through queries and help us understand the .explain() plans as well as DAGs on spark UI. As you already have taught well and in-depth in the explain plan video, you can go through 5-10 different queries quickly and teach us on a high level. 5. What are the common transformations and cleaning of data done in real world. 6. One real world project architecture, you don't have to make a practical video on this, just some slides should be enough, regarding how after data comes in source, what all things are done in terms of ADF, Pyspark, different environments, etc.

[@mateen161]

Very helpful and well explained. Thank you!

[@rajasdataengineering7585]

Thanks! Glad you liked it

[@SureshBabu-kf5jx]

HI Raja, Thank you so much for your videos. Really helpfull. Can you also make a video on CI/CD Integration of notebook and promoting notebooks to higher environments

[@rajasdataengineering7585]

Hi Suresh, thank you for your comment. Sure will create a video on CI/CD

[@sameersamu2427]

Raja Sir it would be great if you can show us one example like take data from azure data lake.

[@rajasdataengineering7585]

Sure Sameer, I will create end to end pipeline once covered all building blocks through concepts

[@quiet8691]

Pls reply raja data engineering I recently landed on your channel to learn databricks. Can you pls tell me the order to watch and go through your channel

[@rajasdataengineering7585]

Pls follow this playlist kzbin.info/aero/PLgPb8HXOGtsQeiFz1y9dcLuXjRh8teQtw&si=E_HULOI6j9wVebOG

[@prathameshpashte6881]

Hii, Raja could you please provide the slides and notebooks used in the tutorials? Thank you!

[@quiet8691]

Hlo sir, Today is the first time i landed on you channel. Can you pls tell me the order to go through your channel?

[@rajasdataengineering7585]

Pls follow this playlist kzbin.info/aero/PLgPb8HXOGtsQeiFz1y9dcLuXjRh8teQtw&si=E_HULOI6j9wVebOG

[@code_nation]

Totally out of context question, but can you please tell me is it possible to do changes like update or delete a row in ssms using databricks. We can make a connection using jdbc, but not sure if we could do changes other than inserting a new row. If it is possible, please make a video on this one or explaining here would be helpful too.

[@rajasdataengineering7585]

We can't do this action directly on SQL server. Instead, we can push Databricks data into SQL server into one of the staging table and we can perform this operation

[@andrew_ng]

May I check when will 125 be released? DLT series will be have how many more videos to complete? Thank you Raja

[@ankitgoyal6458]

Hello - I do not see LIVE demonstration of DLT (i.e. on Notebook) in video # 122-123-124. Are you planning to add more in this series? by when?

[@rajasdataengineering7585]

Hello Ankit, I will add live demo also soon. Due to workload, I couldn't get time to create one yet. Will create soon

[@ATHARVA89]

in which playlist can i watch autoloader concepts ?

[@rajasdataengineering7585]

Pls watch this video - kzbin.info/www/bejne/fZu5Y6Bul56ZhLs

[@DSALectures21]

Why the playlist numbering is incorrect?...

[@rajasdataengineering7585]

The numbering is video creation sequence but playlist is grouping relevant videos on particular topic

[@user-cl8pn8co6q]

Hi sir create End-To-End Project's

[@rajasdataengineering7585]

Hi, sure I will create end to end projects

[@rajunaik8803]

Hi Raja, have a doubt, in materialized view or live table, you mentioned that it is going to re compute the entire dataset. Is it true? I believe it will capture only the CDC data from defined query.

[@rajasdataengineering7585]

Hi Raju, yes it will be recomputed each time based on the query. If we don't give any query but ingesting from source, then no recomputation

[@rajunaik8803]

@@rajasdataengineering7585 Understood. Thanks Raja!!

[@rajasdataengineering7585]

Welcome

[@Manisood001]

do you have a paid course?

[@saketsrivastava84]

Please create a video showing all these. It is not that helpful

[@rajasdataengineering7585]

Sure, will create realtime project video

[@kalyand3959]

Hi raja, Can i get your email id??

[@rajasdataengineering7585]

[e-mail]