strimzi entity operator and kafka exporter configuration issues

[bono5150]

Hi community.

Configuration description:
Kafka strimzi 0.45.0 (kraft) with Kafka Exporter v1.9.0 installed from helm charts.

I have installed kafka cluster recently based on examples from strimzi kafka-single-node.
The Kafka cluster is working fine, and clients are able to connect to the cluster on port 9092, but I have encoutered some issues with the installation.

• First issue with Kafka Exporter connection port set to 9091.

KAFKA_EXPORTER_KAFKA_SERVER:     my-cluster-kafka-bootstrap:9091

I don't see any possibility to change it to 9092 in strimzi operator yaml file.
I would like to provide the correct connection port in helm values.yaml file (I mean port 9092). Is it possible or it is hardcoded somewhere else ?

• Second issue with entity operator (topic operator to be more precise).
  It is trying to connect to kafka cluster on port 9091 but without a success:
  [AdminClient clientId=strimzi-topic-operator-55beeb4f-3d8e-45b0-aaf5-b8660b257133] Initiating connection to node my-cluster-kafka-bootstrap:9091 (id: -1 rack: null)

The stack trace is provided below:

2025-02-21 17:06:33 INFO  AdminMetadataManager:273 - [AdminClient clientId=strimzi-topic-operator-55beeb4f-3d8e-45b0-aaf5-b8660b257133] Metadata update failed
org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
2025-02-21 17:06:33 DEBUG KafkaAdminClient:1161 - [AdminClient clientId=strimzi-topic-operator-55beeb4f-3d8e-45b0-aaf5-b8660b257133] Timed out 1 call(s) with assigned nodes.
2025-02-21 17:06:33 DEBUG AdminMetadataManager:198 - [AdminClient clientId=strimzi-topic-operator-55beeb4f-3d8e-45b0-aaf5-b8660b257133] Requesting metadata update.
2025-02-21 17:06:45 DEBUG KafkaAdminClient:965 - [AdminClient clientId=strimzi-topic-operator-55beeb4f-3d8e-45b0-aaf5-b8660b257133] Call(callName=listNodes, deadlineMs=1740157605095, tries=1, nextAllowedTryMs=1740157605183) timed out at 1740157605096 after 1 attempt(s)
java.lang.Exception: TimeoutException: Timed out waiting for a node assignment. Call: listNodes
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.handleTimeoutFailure(KafkaAdminClient.java:966) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$Call.fail(KafkaAdminClient.java:935) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$TimeoutProcessor.handleTimeouts(KafkaAdminClient.java:1065) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.timeoutPendingCalls(KafkaAdminClient.java:1144) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1509) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1473) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2025-02-21 17:06:45 DEBUG KafkaAdminClient:1146 - [AdminClient clientId=strimzi-topic-operator-55beeb4f-3d8e-45b0-aaf5-b8660b257133] Timed out 1 pending calls.
Exception in thread "main" java.lang.RuntimeException: Failed to get cluster configuration: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
	at io.strimzi.operator.topic.KafkaHandler.clusterConfig(KafkaHandler.java:110)
	at io.strimzi.operator.topic.BatchingTopicController.<init>(BatchingTopicController.java:88)
	at io.strimzi.operator.topic.TopicOperatorMain.<init>(TopicOperatorMain.java:74)
	at io.strimzi.operator.topic.TopicOperatorMain.operator(TopicOperatorMain.java:162)
	at io.strimzi.operator.topic.TopicOperatorMain.main(TopicOperatorMain.java:157)

The default strimzi configuration is somehow broken or additional configuration is required to run Entity Operator and Kafka Exporter.
Do you have any tips or documentation with working examples how to configure Entity Operator and change the Kafka Exporter port to be working correctly with kafka cluster ?

[scholzj]

The listener on 9091 port is used for all internal connections managed by Strimzi. The user-defined listeners such as the one on port 9092 are for your applications.

There is no way to change where the components such as Kafka Exporter or Topic Operator connect and them connecting to 9091 is completely correct and intentional. However, if you want, you can disable the Strimzi managed components and install and manage them yourself separately as a user applications of course.

[bono5150]

(There is also no Kafka Exporter 1.9.0 in Strimzi 0.45 as Kafka Exporter 1.9.0 was released just few days ago.)

There is no Kafka Exporter in strimzi at all. The Kafka Exporter image is not pointed directly in strimzi operator yaml file like the other images here and the yaml file is published here - lines from 16296 to 16348.

[scholzj]

(There is also no Kafka Exporter 1.9.0 in Strimzi 0.45 as Kafka Exporter 1.9.0 was released just few days ago.)

There is no Kafka Exporter in strimzi at all. The Kafka Exporter image is not pointed directly in strimzi operator yaml file like the other images here and the yaml file is published here - lines from 16296 to 16348.

What? I have no idea what you are talking about.

[bono5150]

(There is also no Kafka Exporter 1.9.0 in Strimzi 0.45 as Kafka Exporter 1.9.0 was released just few days ago.)

I can see in changelog the support for Kafka Exporter v1.9.0 will be available in strimzi 0.46.0. I will try with v1.8.0 in that case.

[bono5150]

(There is also no Kafka Exporter 1.9.0 in Strimzi 0.45 as Kafka Exporter 1.9.0 was released just few days ago.)

There is no Kafka Exporter in strimzi at all. The Kafka Exporter image is not pointed directly in strimzi operator yaml file like the other images here and the yaml file is published here - lines from 16296 to 16348.

What? I have no idea what you are talking about.

I am talking about images versions in strimzi-cluster-operator-0.45.0.yaml file. The Kafka Exporter image is not menthioned at all.

[scholzj]

I am talking about images versions in strimzi-cluster-operator-0.45.0.yaml file. The Kafka Exporter image is not menthioned at all.

Then you are completely wrong as that does not mean anything. Things exist as binaries before they exist as containers. Strimzi just uses its own container with Kafka Exporter binary.

[bono5150]

I see that continuing this thread of conversation makes absolutely no sense. There is no answer on a topic that is important from the point of view of the operation of the Kafka environment and about which I asked.

Thank you for your commitment and time.

[scholzj]

Well, maybe you need to be more clear about what exactly is your question or problem or whatever it is. As I told you right away, using port 9091 by all internal Strimzi components is completely normal, intentional, and there is nothing wrong with it.

[ppatierno]

@bono5150 hi ... let's try to understand better what your problem is, even because at this point I am curious :-)
As Jakub explained, Strimzi uses the port 9091 for components like topic operator, user operator, kafka exporter and others to connect to the Kafka cluster. This way it doesn't interfere with other ports (i.e. 9092) used by your clients.
Focusing on Kafka Exporter, you can enable it within the Kafka custom resource by specifying the kafkaExporter field (which has the following spec https://strimzi.io/docs/operators/latest/configuring#type-KafkaExporterSpec-reference). For example, this way: https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/kafka-metrics.yaml#L53
What Strimzi does for you is deploying Kafka Exporter (by using the default Strimzi related image) configured to connect to the Kafka cluster on port 9091. This thing works out of the box, unless any other issues that maybe we are not able to understand from your description.