Thanks for the idea! , Will try to create some project videos

Thx in advance

You should try to batch your query (Task will be to mimize the db call) or you can copy the data from db to databricks , Checkout this one : kzbin.info/www/bejne/pGG1hGp6d8Ssb9U

Thank you , Please share the playlist with your Linkedin network which will help this channel to grow.

Thank you so much

Thank you . So git path is not set properly , Check out this one : kzbin.info/www/bejne/oqWcoHd3q9V6aqs , Please share the playlist with your network which will help this channel to grow.

@@anirvandecodes Thank you so much

thank you , please share the playlist with your LinkedIn network so that it reach to wider audience.

Yes , Checkout this video to setup Kafka on confluent cloud : kzbin.info/www/bejne/o5qxZYqCnq-hiKc Playlist link : kzbin.info/aero/PLGCTB_rNVNUNbuEY4kW6lf9El8B2yiWEo

There are a few configuration changes you can do Batching: Set linger.ms (5-50 ms) and increase batch.size (32-128 KB). Serialization: Opt for efficient formats like Avro or Protobuf. Partitions & Proximity: Add more partitions and deploy near Kafka brokers. In production people generally use more scalable solutions than just a Python producer app, Check out this : docs.confluent.io/platform/current/connect/index.html Do share the playlist with your LinkedIn community.

You are making a great progress , Please share with your friends and colleagues

Can you share the complete error text and project structure?

@@anirvandecodes so in your video you pasted the yml file containing sources in the all the 3 folders, since the source is the same for all 3 files I just pasted the model sql files inside the folder and kept the yml file outside the folder so this resolved the error, I believe with the new dbt version you cannot have 2 yml files having the same source referencing the same table at the same folder level currently my folder structure looks like models -staging - - staging_employee_details.sql -intermideate - - intermideate _employee_details.sql -marts - - marts_employee_details.sql -employee_source.yml in the video you pasting the yml file in each 3 folders (staging, intermideate, marts) which gives naming_conflict_error your videos have been very informative, I went through the whole playlist was struggling to install dbt on my system and understand it thank you so much ! 😄😄

@ i think you might have same spirce name mentioned in two place , take a look into that

@@anirvandecodes dbt found two sources with the name "employee_source_EMPLOYEE". Since these resources have the same name, dbt will be unable to find the correct resource when looking for source("employee_source", "EMPLOYEE"). To fix this, change the name of one of these resources: - source.dbt_complete_project.employee_source.EMPLOYEE (models\marts\marts_employee_source.yml) - source.dbt_complete_project.employee_source.EMPLOYEE (models\staging\stg_employee_source.yml)

should the source name always be unique ?

Absolutely , They call it self managed kafka , Check this out www.confluent.io/get-started/?product=self-managed

Made my day , Thank you , Do share with your network.

@@anirvandecodes you deserve it man!

dbt core does not have any out of box column mapping lineage . You can explore column lineage in dbt cloud or check out this one tobikodata.com/column_level_lineage_for_dbt.html

Glad to hear that , Please share the content with your network.

Thank you for the support , Yes I will be publishing content on spark structured streaming with kafka.

Glad it was helpful! , do share with your network

looks like this is some pycharm path related issue , try to debug if path is coming properly in pycharm or you can also select different terminal as git bash , you can get more info on google

This error generally comes when the path is not added in the system ,try to use stackoverflow or chatgpt and you can try to do with git bash

Thank you so much, it really made my day :)

I will upload one video on snapshot soon , check that out !

You can apply distinct in sql to remove the duplicates or use any other strategy to remove the duplicates

sure will create one video on it

Thanks! Will do!

Glad I could help

hello

Yes I have a complete dbt playlist here : kzbin.info/aero/PLGCTB_rNVNUON4dyWb626R4-zrLtYfVLa

sorry i lost the model file

thank you so much , I have two more videos dbt to complete the playlist , will plan after that

Keep watching

Thank you so much 🙂

yes , i am in the process on making video on how to deploy dbt project on cloud. stay tuned!

Yes, definitely. think it as a sql query with which you are filtering out the data.

@@anirvandecodes Hello Anirvan, any code snippet or any format suggestion from your end??

No problem!

that concept is called as change data feed , you need to first find out the records which have changed , There are different techniques , like in sql you can do SELECT * FROM table1 EXCEPT SELECT * FROM table2 to find what rows have changes then only insert those records.

Hi, in this technique we are not adding salt to both dataframe which is not needed , We are adding salt to one dataframe and we are exploding the other dataframe to do a join at the end.

thank you for watching, I am glad it helped you to understand the concept.

Yes that is correct, keep watching