How to install Kafka Connect connector plugins

[@reggiechou392]

Thank you for sharing, and look forward to more related introductions about the deployment and maintenance of the production environment

[@mirshahrukhsahito7904]

i am having an issue, i am unable to locate the file where /var/log/message path is written, bcoz of higher storage usage i want to change the directory of that messages file to another directory , plz tell me how do i do it

[@localboyvijay9628]

Thanks a lot.. Do you have any documentation for standalone vs distributed connect? when it is come to production system which is better to manage manual or confluent to use install connectors?

[@MESSAOUDIHamza]

Hi, I want to thank you for this video, I want to use Kafka connect with debezium and CDC for a real time notification of changes in DB, I tried Debezium Engine for test and it work fine, now want to deploy the server version and need to work inside Kafka connect. my question is kafka connect need a virtual machine to install in or not ? i'm using azure cloud. thanks

[@rmoff]

Kafka Connect runs on its own JVM process, separate from the Kafka broker. You can deploy it directly on bare metal/VM, or in a container.

[@MESSAOUDIHamza]

Thank you so much Robin, I don't know what to use, Debezium Server or Debezium with kafka connect ?

[@rmoff]

@@MESSAOUDIHamza I've only used Debezium as a plugin in Kafka Connect, so I don't know about Server, sorry. There is a good community chat for Debezium here where you might ask: gitter.im/debezium/user

[@hymavathikalva8959]

Thanks for the video, how can we install kafka connect in confluent cloud which is used as service in Google cloud platform

[@rmoff]

Hi, Confluent Cloud offers managed connectors, for which you can see a list here: docs.confluent.io/cloud/current/connectors/index.html If there isn't a connector available that you want to use you can run your own Kafka Connect worker to connect to Confluent Cloud, e.g. rmoff.net/2021/01/11/running-a-self-managed-kafka-connect-worker-for-confluent-cloud/

[@MegaSriku]

Hi @Robin , Don't we need to specify the consumer group details , while creating the connectors ?

[@rmoff]

No, you don't need to - Kafka Connect handles the implementation of consumer groups. If you've got any more questions please head over to forum.confluent.io/. Thanks!

[@ayanmukhuty3204]

This command fails curl -s localhost:8083/connector-plugins|jq '.[].class' bash: jq: command not found (23) Failed writing body

[@rmoff]

You need to have jq installed if you want to use it :) If you don't have access to it you would just run curl -s localhost:8083/connector-plugins and manually read the JSON that's returned.

[@OutboxThinker1]

Hi Robin, nice video. When i install the connector manually, i cannot access it through localhost:8083 :(

[@rmoff]

Did you restart the Kafka Connect worker? That's necessary for it to be picked up. If you have any more questions then forum.confluent.io/ is the best place to ask :)

[@OutboxThinker1]

@@rmoff Thanks for your quick reply. I wasn't aware that the connector was running inside my kafka container, that was the reason i was not able to access it from outside of Kafka container. Note that I only had this problem while using the manual installation. Thanks for sharing your knowledge

[@Gr3mlinz]

Hi, thank you for the video. if I installed Confluent platform from RPM, what should be CONFLUENT_HOME? Looks like there is no value that can satisfy all commands, because in that case Confluent is spread all over the system and not in one subdirectory.

[@rmoff]

I don't know, sorry. Try forum.confluent.io/ :)

[@nicholassmith831]

very new to Kafka. we have it installed on a unix machine and its running fine. my problem is with trying to get a kafka source datagen connector to see an avsc file stored in a mounted folder. path name of the schema file is /mnt/file.avsc and the error message I see in the logs is 'Unable to read the '/mnt/file.avsc' schema file'. Any pointers on what I'm doing wrong?

[@rmoff]

Best place to ask is over on cnfl.io/slack

[@TheHitamnoah]

Hi , can I use Kafka connect on windows os? I'm manage to install Kafka and Zookeeper but not manage install connect.

[@rmoff]

Yes, check out this blog for a guide: www.confluent.io/blog/set-up-and-run-kafka-on-windows-linux-wsl-2

[@danielbelemduarte]

Great video. I'm trying to have 2 sink connector to the same MongoDB but to read from different kafka topics and write on different MongoDB Collections. This works fine for one of the topics, but when I configure the second Sink Connector I get the following error: "ERROR Failed to write mongodb documents (com.mongodb.kafka.connect.sink.MongoSinkTask) java.lang.IllegalArgumentException: Invalid BSON field name connect.decimal.precision" I'm following the same maming convetions for both topics, not sure what is happening. Any help ? Thank you, Daniel

[@rmoff]

Hi Daniel, I don't know the answer to this, sorry. You should try asking over at forum.confluent.io/ :)

[@shuchikumari8031]

Hello.. Can we connect ksqldb to kafka, it is possible that we can connect it.

[@rmoff]

Yes, ksqlDB works directly with Kafka. In fact, you *have* to connect ksqlDB to Kafka for it to work. Head to forum.confluent.io/ if you have any more questions.

[@shuchikumari8031]

@@rmoff sir actually my concern is we can connect kafka to mysql, so thier is any possible way that we can connect to ksql db to mysql, because I am doing one project in denodo platform thier only mysql Or oracle this type of db support.

[@rmoff]

@@shuchikumari8031 Please post your question at forum.confluent.io/ and I will answer it there

[@muhammadhozaifa8070]

HI, thank you for this video. This video is really helpful. I just want to know, is there any way we can use kafka connect without docker in windows 10?

[@rmoff]

Yes, check out www.confluent.io/blog/set-up-and-run-kafka-on-windows-linux-wsl-2

[@ghousemohammad1489]

how to connect Kafka to salesforce

[@rmoff]

www.confluent.io/hub/confluentinc/kafka-connect-salesforce

[@brunomartinho4086]

@@rmoff , is authentication required for salesforce or does it auto detect ?

[@rmoff]

@@brunomartinho4086 I don't know, sorry - try asking at forum.confluent.io/

[@georgelza]

why why oh why have i not found this before.... would have saved me so much grieve, hehehe... now to figure out why my local cp kafka connect does not want to sink to MongoAtlas hosted collection...