Unrecoverable exception from producer send callback in kafka mirrormaker2

[palpandy]

Kafka version: 3.7.0
Strimzi: 0.40.0

We often see below error in kafka mirrormaker2 and the weird thing it is only for few tasks and not for all tasks.So, some of the tasks are running and other tasks are failed with exception.

Error from mm2 CR:
trace: "org.apache.kafka.connect.errors.ConnectException: Unrecoverable exception from producer send callback\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.maybeThrowProducerSendException(WorkerSourceTask.java:340)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.prepareToPollTask(WorkerSourceTask.java:120)\n\tat org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:353)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)\n\tat org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:77)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed: credentials for user could not be verified\n"

- connector:
    state: RUNNING
    worker_id: '<mm2_name>:8083'
  name: source->destination.MirrorSourceConnector
  tasks:
    - id: 0
      state: RUNNING
      worker_id: '<mm2_name>:8083'
    - id: 1
      state: RUNNING
      worker_id: '<mm2_name>:8083'
    - id: 2
      state: **FAILED**
      trace: "org.apache.kafka.connect.errors.ConnectException: Unrecoverable exception from producer send callback\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.maybeThrowProducerSendException(WorkerSourceTask.java:340)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.prepareToPollTask(WorkerSourceTask.java:120)\n\tat org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:353)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)\n\tat org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:77)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed: credentials for user could not be verified\n"
      worker_id: 'bkts67-mm2-mirrormaker2-0.bkts67-mm2-mirrormaker2.orms-kafka.svc:8083'
  type: source

It says "Authentication failed: credentials for user could not be verified\n"" but I am sure that authentication is correct.
After restart , the connector tasks started running and no issues but again after sometime (may be in hours or days), it comes back .

Error from mm2 logs:

2025-02-07 01:09:01,425 INFO [source->destination.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-source->destination.MirrorSourceConnector-0] Failed authentication with destination-kafka-2.destination-kafka-brokers.orms-kafka.svc/10.225.151.100 (channelId=2) (Authentication failed: credentials for user could not be verified) (org.apache.kafka.common.network.Selector) [kafka-producer-network-thread | connector-producer-source->destination.MirrorSourceConnector-0]
2025-02-07 01:09:01,426 INFO [source->destination.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-source->destination.MirrorSourceConnector-0] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | connector-producer-source->destination.MirrorSourceConnector-0]
2025-02-07 01:09:01,426 ERROR [source->destination.MirrorSourceConnector|task-0] [Producer clientId=connector-producer-source->destination.MirrorSourceConnector-0] Connection to node 2 (destination-kafka-2.destination-kafka-brokers.orms-kafka.svc/10.225.151.100:9096) failed authentication due to: Authentication failed: credentials for user could not be verified (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | connector-producer-source->destination.MirrorSourceConnector-0]
2025-02-07 01:09:01,977 INFO [source->destination.MirrorSourceConnector|task-2] [Producer clientId=connector-producer-source->destination.MirrorSourceConnector-2] Failed authentication with destination-kafka-2.destination-kafka-brokers.orms-kafka.svc/10.225.151.100 (channelId=2) (Authentication failed: credentials for user could not be verified) (org.apache.kafka.common.network.Selector) [kafka-producer-network-thread | connector-producer-source->destination.MirrorSourceConnector-2]
2025-02-07 01:09:01,978 INFO [source->destination.MirrorSourceConnector|task-2] [Producer clientId=connector-producer-source->destination.MirrorSourceConnector-2] Node 2 disconnected. (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | connector-producer-source->destination.MirrorSourceConnector-2]
2025-02-07 01:09:01,978 ERROR [source->destination.MirrorSourceConnector|task-2] [Producer clientId=connector-producer-source->destination.MirrorSourceConnector-2] Connection to node 2 (destination-kafka-2.destination-kafka-brokers.orms-kafka.svc/10.225.151.100:9096) failed authentication due to: Authentication failed: credentials for user could not be verified (org.apache.kafka.clients.NetworkClient) [kafka-producer-network-thread | connector-producer-source->destination.MirrorSourceConnector-2]
2025-02-07 01:09:02,640 INFO [source->destination.MirrorSourceConnector|task-2] WorkerSourceTask{id=source->destination.MirrorSourceConnector-2} Committing offsets for 0 acknowledged messages (org.apache.kafka.connect.runtime.WorkerSourceTask) [task-thread-source->destination.MirrorSourceConnector-2]
2025-02-07 01:09:02,640 ERROR [source->destination.MirrorSourceConnector|task-2] WorkerSourceTask{id=source->destination.MirrorSourceConnector-2} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-source->destination.MirrorSourceConnector-2]
org.apache.kafka.connect.errors.ConnectException: Unrecoverable exception from producer send callback
at org.apache.kafka.connect.runtime.WorkerSourceTask.maybeThrowProducerSendException(WorkerSourceTask.java:340)
at org.apache.kafka.connect.runtime.WorkerSourceTask.prepareToPollTask(WorkerSourceTask.java:120)
at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:353)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)
at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:77)
at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:237)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed: credentials for user could not be verified

Could you please help me to find the reason for these exceptions ?

[scholzj]

The first error is probably more something to raise in Apache Kafka.

For the second, if it says that authentication failed, then it probably failed. That usually means some misconfiguration on your side. But it could be of course also some issue or outage on the broker side etc. But nobody will give you any details or hints from what you shared. Not without full logs, full configurations etc.