How to setup Kafka Cluster & Kafka Connect Cluster for Snowflake in different places(clusters)

[@shashanksingh7371]

I wish to read data Realtime from Snowflake, Would you please elaborate what could be the possible options? If possible what are the pros and cons of the approaches. I wish to avoid the use stream, if possible since it will require using tasks which have to be triggered every few minutes rather than triggering as soon the data is available. Also, if we use external share along with it, multiple files will be created.

[@abhi-ys1je]

Can you please provide how to run kafka-snowflake-connector in same ec2 instance

[@KnowledgeAmplifier1]

Hello abhi, you can refer this video where Snowflake Sink Connector is running in same EC2 instance where Kafka Cluster is running -- kzbin.info/www/bejne/oZ-YdHaDadN0bMUsi=UignQTupySExJMBO

[@marianek8092]

Great explanation. Could you please share how to install kafka connect on a separate server?

[@KLNP-ot6rq]

Can you do some videos on KSQLDB ?

[@KLNP-ot6rq]

Great session !! appreciated. Quick question - how can we make our producer and consumer keep running, basically make it as a job. Any help is appreciated.

[@KnowledgeAmplifier1]

Hello KLNP1978, to make your Kafka producer and consumer run continuously as a job, you can consider the following options: *Daemon Process* : Write a custom script or application that runs as a daemon or background process. This script will contain the logic for your Kafka producer and consumer, ensuring they continuously send and receive messages. *Containerization* : Use containerization technologies like Docker to create containers for your Kafka producer and consumer. You can then run these containers as separate services or jobs, ensuring they remain active and handle message processing continuously. *Scheduler* : Utilize a job scheduler or task scheduler, such as cron, to periodically trigger your Kafka producer and consumer processes. You can configure the scheduler to execute the producer and consumer scripts at predefined intervals, allowing them to run as recurring jobs. *Stream Processing Frameworks* : Explore stream processing frameworks like Apache Flink, Apache Storm, or Apache Samza. These frameworks provide built-in capabilities to handle continuous data streams and can be used to process Kafka messages continuously. Remember to handle error scenarios, implement proper logging, and ensure fault tolerance in your producer and consumer applications to ensure reliable and resilient data processing. By implementing one of these approaches, you can transform your Kafka producer and consumer into continuously running jobs, enabling seamless message processing and data flow.

[@canigeteb1847]

would be great if you can show us schema registry with Kafka and how to use this while writing the topic into Snowflake . additionally, it would be really helpful if you can post some vdo of Confluent Kafka.

[@mohammedameer9909]

Can you create a video on kafka connect with nosql database like Mongo db or Cassandra

[@canigeteb1847]

like your ado on Kafka and Snowflake. I am trying to send the data from Snowflake to Kafka and there is no connector as such. do you know how we can achieve this?

[@KnowledgeAmplifier1]

Hello CanIget eb1, please refer this -- medium.com/snowflake/streaming-data-from-snowflake-to-kafka-ed76ce0400c2

[@canigeteb1847]

@@KnowledgeAmplifier1 thank you ., do you have any VDO which will does reverse ETL? additionally, I am looking for how to setup Confluent Kafka and write the data into Snowflake. Please pardon my basic questions on Kafka. TIA.

[@digitaltechconnect1318]

Please provide us the session for Confluent Kafka

[@pratikmakwana8488]

can i contact you somehow?

[@KnowledgeAmplifier1]

Hello Pratik , please post your doubt here , If I have any idea on the topic , I will reply asap or make a separate video on the topic ...