Migrating two existing Kafka clusters in the same namespace to use Kafka node pool

[ofamuyiw]

Hi,
We are currently running two Kafka clusters in the same namespace, data-platform and are now planning to migrate them to use KafkaNodePool before eventually migrating them to KRaft mode.

kubectl get pods -n data-platform
a-cluster-kafka-0
a-cluster-kafka-1
a-cluster-kafka-2
a-cluster-zookeeper-0
a-cluster-zookeeper-1
a-cluster-zookeeper-2
b-cluster-kafka-0
b-cluster-kafka-1
b-cluster-kafka-2
b-cluster-zookeeper-0
b-cluster-zookeeper-1
b-cluster-zookeeper-2

kubectl get kafka -n data-platform
a-cluster
b-cluster

Going by this documentation, how can we setup the KafkaNodePool custom resource for the two clusters as we can only have one strimzi.io/cluster label for one of the clusters?
Thanks in advance

[im-konge]

Hey, you can create two KafkaNodePools with broker role, each pointing to particular strimzi.io/cluster -> so one for strimzi.io/cluster: a-cluster and second one for strimzi.io/cluster: b-cluster. Then you can follow the documentation for each of the cluster, as it describes the complete path.

Or did you mean something else?

[im-konge]

Or did you mean having one KafkaNodePool for both of the Kafka clusters? That is not possible.

[ofamuyiw]

Yes, that's what I meant. Oh ok!
Thanks

[scholzj]

You need to use different KafkaNodeools for each cluster. If you have them in one namespace - which we don't recommend - you have to first migrate one cluster to the kafka node pool, then create additional node pool with another name, model all data to it and delete the kafka node pool. And then you can do the same for the second cluster. For more details, please use search here or on Slack where this was discussed multiple times in the past.

[ofamuyiw]

Thanks for the workaround solution

[ofamuyiw]

@scholzj Could you suggest steps for safely moving the data to the new pool with another name and then deleting the kafka node pool with little or no down time? We are thinking of using cruise-control to move data between the brokers. Thanks

[scholzj]

Yes, you can use Cruise Control for that. Check out the remove brokers feature in the KafkaRebalance resource in our docs and examples.

[ofamuyiw]

Thanks

[ofamuyiw]

@scholzj What about the controller KafkaNodePool for the two clusters? They must be named differently for same reason as for kafka KafkaNodePool eventually, right?

[scholzj]

Yes.

[ofamuyiw]

Thanks