Question around openshift_portal_net parameter

[DanyC97]

Hi all,

I'm running Origin 1.3 without any router deployed however with some services defined as type: LoadBalancer. (note this is a lab only for now)

Now my questions are around the openshift_portal_net parameter which by default (as per doc) is set to 172.30.0.0/16.

Now my questions are:

• looking through the code [1] and [2] i am trying to understand how this parameter openshift_portal_net relates to the cluster-ip & external-ip service?
• what is the rule (given by which parameter) for subnetting the base network 172.30.0.0/16 ? I'm having hard time understanding how a different subnet - ie 172.30.126.102 or 172.30.96.49 gets allocated for each service. Same rule applies to the external-ip - different subnet (2nd octet) and then variance on 3rd/4th octet)

@sdodson i've seen you were the last person who improved the doc around this area, by any chance do you know the answers to the above ?

Thanks in advance !

oc get svc
NAME                   CLUSTER-IP       EXTERNAL-IP                     PORT(S)                               AGE
service1             172.30.126.102   172.46.24.79,172.46.24.79       9012/TCP                              13h
service2             172.30.4.81      <nodes>                         80/TCP,443/TCP                        13h
service3             172.30.96.49     172.46.31.57,172.46.31.57       9999/TCP,8443/TCP                     13h
service4            172.30.123.225   172.46.3.116,172.46.3.116       9150/TCP                              13h

[1]

openshift-ansible/playbooks/common/openshift-node/config.yml

Line 44 in 2d52f7c

openshift_docker_hosted_registry_network: "{{ hostvars[groups.oo_first_master.0].openshift.common.portal_net }}"

[2]

openshift-ansible/roles/openshift_facts/tasks/main.yml

Line 53 in 701bd04

portal_net: "{{ openshift_portal_net | default(openshift_master_portal_net) | default(None) }}"

[DanyC97]

And situation gets even more interesting where i've added as per doc openshift_portal_net=10.1.111.192/27 and redeployed anything and what i got is again not as per my expectation:

oc get svc
NAME                   CLUSTER-IP       EXTERNAL-IP                     PORT(S)                               AGE
service1             10.1.111.218   172.46.88.105,172.46.88.105       9012/TCP                              13h
service2             10.1.111.204   <nodes>                         80/TCP,443/TCP                        13h
service3             10.1.111.221   172.46.163.231,172.46.163.231       9999/TCP,8443/TCP                     13h
service4            10.1.111.197   172.46.198.127,172.46.198.127       9150/TCP 

My expectation was for the external-ip to match the cluster-ip subnet - i.e 10.1.111.x

Also i see the clusternetwork set as below

oc get clusternetwork
NAME      NETWORK       HOST SUBNET LENGTH   SERVICE NETWORK   PLUGIN NAME
default   10.3.0.0/16   8                    10.1.111.192/27   redhat/openshift-ovs-subnet

[sdodson]

openshift_portal_net sets the kubernetes service subnet, I don't believe this is actually subnetted instead as services are created and updated the kubelet is responsible for defining iptables nat rules which map to the pod ip addresses. I expect that the external-ip field is the list of pod ip addresses which would all be ip addresses assigned to the pods by the SDN. The SDN is routed and each host is given a hostsubnet from clusternetwork.

@openshift/networking can you help answer some of these questions.

[DanyC97]

thanks for the info @sdodson !!

While i was trying to get to the bottom of it i came across https://github.com/redhat-cop/openshift-playbooks/blob/master/playbooks/operationalizing/ingress.adoc which suggest that

allocate External IP addresses using ingress is enabled by default within OpenShift and configured to use the 172.46.0.0/16 range.

and then found that this is set by default by openshift_master_ingress_ip_network_cidr which explain that:

• no subnetting rule between clusterIP and externalIP => made the wrong assumption
• due to the above parameter openshift_master_ingress_ip_network_cidr the external-ip will always be populated for services of type: LoadBalancer even if there is no cloud provider LB (ie VMware) and that does explain why on a pure K8 cluster for the services of type: LoadBalancer the external-ip field is "null"

Will appreciate if someone can confirm/ deny what i said above.

[DanyC97]

And slowly but surely closing the loop ...

i found the PR which implemented the code and has tons of v useful comments, including the usecase.

openshift/origin#9454

Starting from Origin 1.3.3 + the default private subnet was changed from 172.46 to 172.29 - see below

openshift/origin#10387
openshift/openshift-docs#3441

[danwinship]

openshift_portal_net sets the kubernetes service subnet, I don't believe this is actually subnetted instead as services are created and updated the kubelet is responsible for defining iptables nat rules which map to the pod ip addresses.

Right. Kubernetes just treats openshift_portal_net as a single subnet, essentially, and allocates IPs out of it randomly. (Though, nitpick: it's kube-proxy that defines the iptables rules, not kubelet.)

[DanyC97]

will close this issue as i got all the answers, thanks @sdodson & @danwinship