strimzi kafka mirrormaker2 with private network load balancers

[L040639]

I am trying to add a external type of private load balancer for strimzi cluster, so added
externalBootstrapService and perPodService annotations to my cluster it deploys private NLB for boostrap servers and brokers with no targets attached in load balancer target-group also the type of target group deployed as IP but not instance

@scholzj Any idea here .

Am i following the correct annotations here .

**externalBootstrapService:
    metadata:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
        service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
        service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
  perPodService:
    metadata:
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
        service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
        service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
        service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"**

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: platform-cluster
  namespace: streams-services
spec:
  kafka:
    authorization:
      type: opa
      url: https://<xxx>:8181/v1/data/kafka/allow
      allowOnError: false
      tlsTrustedCertificates:
        - secretName: lk-ca-root-bundle
          certificate: ca.crt
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.7'
    storage:
      type: persistent-claim
      size: 100Gi
      class: efs-sc
    listeners:
      - authentication:
          checkAccessTokenType: false
          clientId: <XXX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: 'https://login.microsoftonline.com/<xxxx>/discovery/v2.0/keys'
          maxSecondsWithoutReauthentication: 3600
          type: oauth
          validIssuerUri: 'https://login.microsoftonline.com/<XX>/v2.0'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: platform-cluster-cert-secret
        name: oauth
        port: 9093
        tls: true
        type: internal
      - authentication:  ## newly added route for the mirror maker.
          checkAccessTokenType: false
          clientId: <XXX>
          clientSecret:
            key: secret
            secretName: entra-id-client
          jwksEndpointUri: 'https://login.microsoftonline.com/<XXX>/discovery/v2.0/keys'
          maxSecondsWithoutReauthentication: 3600
          type: oauth
          validIssuerUri: 'https://login.microsoftonline.com/<XXX>/v2.0'
          userNameClaim: azp
        configuration:
          brokerCertChainAndKey:
            certificate: tls.crt
            key: tls.key
            secretName: platform-cluster-cert-secret
        name: external
        port: 9094
        tls: true
        type: loadbalancer
    version: 3.7.0
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: kafka-metrics
    template:
      externalBootstrapService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
      perPodService:
        metadata:
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb-ip
            service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance
            service.beta.kubernetes.io/aws-load-balancer-internal: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled: "true"
            service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout: "60"
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - platform-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - kafka
                topologyKey: kubernetes.io/hostname
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: platform-cluster
                strimzi.io/component-type: kafka
  entityOperator:
    topicOperator: {}
    userOperator: {}
  zookeeper:
    storage:
      type: persistent-claim
      class: efs-sc
      size: 10Gi
    replicas: 3
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: zookeeper-metrics-config.yml
    template:
      pod:
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                    - key: strimzi.io/cluster
                      operator: In
                      values:
                        - platform-cluster
                    - key: strimzi.io/component-type
                      operator: In
                      values:
                        - zookeeper
                topologyKey: kubernetes.io/hostname
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
            labelSelector:
              matchLabels:
                strimzi.io/cluster: platform-cluster
                strimzi.io/component-type: zookeeper
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

[scholzj]

No idea, sorry. Please keep in mind that Strimzi just passes the annotations to the Kubernetes service and that is it. The rest is up to kKubernetes. Maybe you can try to get rid of some of the annotations to see which causes the problem?