how can i get this ppt?

Hi Sir, Thanks for all your videos. Can you please make video on how to read private API data incrementally with credentials and bearer token using databricks?

Hi Sir, Can you please start the Unity Catalogue series.

Hi Sir, Can you tell in these examples you have shown, why many jobs are created for each of the join query you have executed. I have understood the stages, explain plan and the DAG. But, the number of jobs part is not clear for me, can you shed some light on it.

nice

Your explanation is greatly appreciated.

Hi Raja, thanks for explaining. Just wanted to check that first cluster you created JOB cluster however later on you created two all purpose cluster. Could you please explain why you have taken all purpose instead of job cluster

I Required more sql and pyspark problems for practice is there any paid course you have which i can purchase for practice only

Can you please Add video related to unity catalog feature in Databricks.

Hello @rajasdataengineering7585, could you please share the csv files used in the course, the course provides excellent information and if the files are available it would really be helpful.

Greetings, Is this sufficient for Databricks Certified Associate Developer exam? Appreciate any clarity.

I found the explanation very detailed, grt work ,keep it up sir

hi raja can you cover all streaming related usecases wrt spark and databricks?

great video raja

Hi sir, can we use groupby dept and take average and then filter using Having clause?

Where can we get these notebook..please share link for all the notebook.. so that we can revise

great

Really awesome explanation, no video taught difference btw above 3 with much clarity, thank you...

need separate series on Spark Streaming

simpler way: df_flattened=df.select("*",explode("Employee").alias("new_emp"))\ .drop("Employee")\ .select("Department","new_emp.emp_name","new_emp.salary","new_emp.yrs_of_service","new_emp.Age") df_flattened.show()

In step 2 (Databricks cell.no 7) While Executing filterDF , you are concatenating all columns of both source and target and checking for the equivalent string to filter . But in this, columns having different values in both source and target can still can give equivalent string and that data should be updated in final table but it will be filtered out by the given condition. Example : dim1=100,dim2=201,dim=300,dim=400 Target_dim1=100,Target_dim2=20,Target_dim=1300,Target_dim=400 Concat value will be 100201300400 . So in this case it will be filtered out without updating

I learned that Databricks use datelakehouse and delta lake format, in that case if i create or add files to DBFS then it will convert into delta lake or it'll be as it is, and DBFS is delta lake house?

hi raja, can you also add detailed spark streaming section ? thanks your content is great!!

i have chosen my carrier path by learning and watching these videos Raja..Thanks Much

Hi All sessions are well organized and detailed explanation can we have any notes for these topics to summarize or revise it for learning purposes. Thanks Much and great work

Sir, why did you switch from Databricks Community Edition to Azure in this video?

df_ans=df.withColumn("new_id",when((col("id")%2!=0) & (col("id")!=df.count()),col("id")+1)\ .when(col("id")%2==0,col("id")-1)\ .otherwise(col("id")))\ .drop("id")\ .orderBy(col("new_id"))\ .show() df.createOrReplaceTempView("students") spark.sql( ''' select *, case when id%2==0 then id-1 when id%2!=0 and id!=(select count(*) from students) then id+1 else id end as new_id from students order by ''' ).show()

my sol: window_base=Window.orderBy('date') Dataframe API df_t=df.withColumn("diff", dense_rank().over(window_base)- dense_rank().over(window_base.partitionBy("status")))\ .groupBy("status","diff").agg(min("date").alias("start_date")\ ,(max("date").alias("end_date")))\ .orderBy("start_date")\ .show() Spark SQL 1 df.createOrReplaceTempView("match") spark.sql(''' with cte as( select *, dense_rank() over(order by date)- dense_rank() over(partition by status order by date) as diff from match) select status,diff, min(date) as start, max(date) as end from cte group by status,diff order by start ''' ).show() Spark SQL 2 df.createOrReplaceTempView("match") spark.sql( ''' with cte as( select *, dense_rank() over(order by date) as rn1, dense_rank() over(partition by status order by date) as rn2, dense_rank() over(order by date) -dense_rank() over(partition by status order by date) as diff from match) select a.status,max(a.start_date) as start_date,max(a.end_date) as end_date from (select date,status,diff, min(date) over(partition by status,diff ) as start_date, max(date) over(partition by status,diff ) as end_date from cte order by date) a group by a.status,a.diff order by start_date asc ''' ).show() enjoy

great explanation

data = [ ("2020-06-01","Won"), ("2020-06-02","Won"), ("2020-06-03","Won"), ("2020-06-04","Lost"), ("2020-06-05","Lost"), ("2020-06-06","Lost"), ("2020-06-07","Won") ] df=spark.createDataFrame(data,['event_date','event_status'])

Hello, your videos inspire to choose Data Engineering as a Career, why you stopped making Interview questions videos ?

Thanks for the videos and nice explanation, I have a question. Default partition size if 128 MB, so for 2 GB RDD file, no of partitions created will be 16. Here in your example are you changing the default size of partition to 10 MB instead of default 128 MB? Please correct if I am missing anything?

If after reading and writing all csv file we again upload the same 5 csv file will it process it again or skip it after checking the checkpointing metadata?

Such a great info. Awesome..

Hi Anna, nice to see your videos and you do reply to all our doubts. One thing related to this transformations applied here, we haven't done any kind of partioning here yet right just read data and how come we are confirming these as narrow and wide transformations. Will partioning happen by default once data is read? Do clarify anna.

Would watching these videos be enough to help pass the Databricks Certified Associate Developer for Apache Spark 3.0 - Scala exam?

great

Thank you for such clarity. But I have a query - As Catalyst Optimizer will consider the broadcast join itself if a table is small enough to fit in memory, even if we haven't performed any broadcast join. So, is it really going to help us out in performance optimization? Or the performance will remain same only even after applying broadcast join?

Thanks Raja,,

Great video content.. Thank you so much

How to get the opprtunity to work on databricks with pyspark..what all real time scenarios to get started with?

great playlist

How does spark.read differ from spark.load?

Hi sir if possible can you provide CSV files what you have used, thank you.

i believe, now autoloader can be also applied as a batch processing so, it can have triggering effect for whenever files get arrived.

Hi Raja, do you have these code stored in github?

Hi, In this video you say serializing and deserializing overhead is there for off heap memory, but in previous videos you said, we can avoid serialization and deserialization in off heap memory. Could you please clarify which one is correct?

can you please make one video on DLT.

This interview series is really helpful. I haven't worked much on databricks but these videos are giving great insight of internal working & concepts. I am able to crack interviews. Thanks a lot for such informative videos!!

use chatgpt for sample dataset creation