[k8s.io] SchedulerPredicates [Serial] validates that NodeSelector is respected if not matching [Conformance] [Suite:openshift/conformance/serial] [Suite:k8s] 10m9s

[tnozicka]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17599/test_pull_request_origin_extended_conformance_gce/12622/

/tmp/openshift/build-rpm-release/tito/rpmbuild-origin9qJhqo/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/scheduling/predicates.go:242
Dec  8 13:15:38.571: Timed out after 10m0s waiting for stable cluster.
/tmp/openshift/build-rpm-release/tito/rpmbuild-origin9qJhqo/BUILD/origin-3.7.0/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/util.go:5082

[tnozicka]

again https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17599/test_pull_request_origin_extended_conformance_gce/12627/

[tnozicka]

again https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17599/test_pull_request_origin_extended_conformance_gce/12631/

[tnozicka]

blocking queue for release-3.7

[mfojtik]

This is flaking also on 3.9 branch now. For 3.7 I think we can pick #17724

[mfojtik]

@sjenning @deads2k said this is an important feature and we should know why we are flaking here. I made a PR that disable the test for master to unblock the queue, but PTAL before merging that if there is something obvious that is broken.

[aveshagarwal]

I am looking into this.

[aveshagarwal]

it seems that the test is not even running as its timed out here (WaitForStableCluster): https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/test/e2e/scheduling/predicates.go#L310

Dec  8 15:45:34.445: Timed out after 10m0s waiting for stable cluster.

Still looking what else is going on that is causing this to time out.

[aveshagarwal]

The issue is that router pod is failing to schedule:

Dec  8 15:45:34.493: INFO: router-1-deploy                                              Pending         [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2017-12-08 15:21:32 +0000 UTC Unschedulable 0/4 nodes are available: 4 CheckServiceAffinity, 4 MatchNodeSelector.}]

[aveshagarwal]

This is flaking also on 3.9 branch now

@mfojtik can you share link for 3.9 logs too? want to make sure if its always router pod or some other pod that is causing time out.

[smarterclayton]

MatchNodeSelector can only happen if either ansible or the installer broke how node labels are working.

[aveshagarwal]

@smarterclayton @stevekuznetsov @sdodson where to check what is deploying router-1-deploy in https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_gce/12622/consoleFull . Also why I dont see the same router-1-deploy being run/deployed in recent runs of gce conformance tests: https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_gce/12864/consoleFull

[aveshagarwal]

@mfojtik in recent runs of gce conformance tests, I dont see that any scheduler predicates are being run. For example this one: https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_conformance_gce/12864/consoleFull

[stevekuznetsov]

@aveshagarwal I would assume that the installer is placing the router down.

[stevekuznetsov]

@sdodson may be able to help you figure that out

[sdodson]

MatchNodeSelector can only happen if either ansible or the installer broke how node labels are working.

@michaelgugino can you look into this? I don't see any reason for it, but it looks like it's using the default selector of region=infra rather than the selector that's specified in the group vars using openshift_hosted_infra_selector: "role=infra" I believe this is only happening on release-3.7 based on all of the reported failures above.

Can someone confirm that those jobs would have used the release-3.7 branch of ansible, I don't see from the logs where openshift-ansible gets installed.

[sdodson]

/assign michaelgugino

[sdodson]

/assign sdodson

[0xmichalis]

https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_conformance_k8s/182/consoleFull#9553647145898c58db7602c31c0eab717

[aveshagarwal]

https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_conformance_k8s/182/consoleFull#9553647145898c58db7602c31c0eab717

Again both scheduler predicate tests are failing for the same reason as the test for stable cluster is timing out because router-1-deploy pod is failing to schedule:

Dec 14 12:47:18.978: INFO: router-1-deploy                                            Pending         [{PodScheduled False 0001-01-01 00:00:00 +0000 UTC 2017-12-14 11:39:48 +0000 UTC Unschedulable 0/4 nodes are available: 1 NodeUnschedulable, 4 MatchNodeSelector.}]

[tnozicka]

the same thing is happening in release-3.7 branch

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17599/test_pull_request_origin_extended_conformance_gce/13338/

[tnozicka]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17620/test_pull_request_origin_extended_conformance_gce/13330/
and others...