[Feature:Prometheus][Conformance] Prometheus when installed to the cluster should start and expose a secured proxy and unsecured metrics [Suite:openshift/conformance/parallel] 1m51s

[tnozicka]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17827/test_pull_request_origin_extended_conformance_gce/13358/

/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/prometheus/prometheus.go:43
Did not find tsdb_samples_appended_total, tsdb_head_samples_appended_total, or prometheus_tsdb_head_samples_appended_total in:
map[string]*io_prometheus_client.MetricFamily(nil),
Expected
    <bool>: false
to be true
/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/prometheus/prometheus.go:83

[mfojtik]

Seen several times today (https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17726/test_pull_request_origin_extended_conformance_gce/13541/)

[mfojtik]

@bparees the second failure in the linked test report seems to be build related

[bparees]

@bparees the second failure in the linked test report seems to be build related

1. Separate flake issues should be opened for separate failures. Otherwise we're just polluting the discussion.

2. if you're referring to this: StdErr: "Error from server (Conflict): Operation cannot be fulfilled on clusterrolebindings.authorization.openshift.io \"system:build-strategy-custom\": the object has been modified; please apply your changes to the latest version and try again", there's an issue for it already: [Feature:Builds] forcePull should affect pulling builder images ForcePull test case execution docker #17596

[simonpasquier]

I had a look at the logs and it seems that the Prometheus pod never gets to the ready state (and the curl exit code is 7 which means "Failed to connect to host"). So I'd say that the error is unrelated to Prometheus itself but rather Kubernetes not being able to schedule the pod?

From https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17726/test_pull_request_origin_extended_conformance_gce/13541/

Jan  2 12:00:10.427: INFO: unable to get unsecured metrics: host command failed
[...]
Jan  2 12:01:39.828: INFO: unable to get unsecured metrics: host command failed
[...]
Jan  2 12:01:39.949: INFO: prometheus-0   ci-prtest-5a37c28-13541-ig-n-8l85  Pending           [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2018-01-02 11:59:46 +0000 UTC  } {Ready False 0001-01-01 00:00:00 +0000 UTC 2018-01-02 11:59:46 +0000 UTC ContainersNotReady containers with unready status: [prom-proxy prometheus alerts-proxy alert-buffer alertmanager]} {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2018-01-02 11:59:46 +0000 UTC  }]
[...]
Jan  2 12:01:40.176: INFO: prometheus-0 started at 2018-01-02 11:59:46 +0000 UTC (0+5 container statuses recorded)
Jan  2 12:01:40.176: INFO: 	Container alert-buffer ready: false, restart count 0
Jan  2 12:01:40.176: INFO: 	Container alertmanager ready: false, restart count 0
Jan  2 12:01:40.176: INFO: 	Container alerts-proxy ready: false, restart count 0
Jan  2 12:01:40.176: INFO: 	Container prom-proxy ready: false, restart count 0
Jan  2 12:01:40.176: INFO: 	Container prometheus ready: false, restart count 0

[sjenning]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18147/test_pull_request_origin_extended_conformance_install/5850/

[sjenning]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18216/test_pull_request_origin_extended_conformance_gce/15049/

[theute]

@oourfali it seems to be an installer issue rather than an issue in P8s itself ?

[zgalor]

@tnozicka was prometheus installed using openshift-ansible?
what inventory file was used?

[tnozicka]

It is installed as part of the test:

origin/test/extended/prometheus/prometheus.go

Line 41 in e274a14

ns, host, bearerToken, statsPort = bringUpPrometheusFromTemplate(oc)

origin/test/extended/prometheus/prometheus.go

Line 297 in e274a14

func bringUpPrometheusFromTemplate(oc *exutil.CLI) (ns, host, bearerToken string, statsPort int) {

[jcantrill]

/assign @zgalor
/unassign @jcantrill

[zgalor]

@smarterclayton is this also caused by #18317 as #17529?

[smarterclayton]

It’s likely On Feb 12, 2018, at 9:06 AM, Zohar Galor <[email protected]> wrote: @smarterclayton <https://github.com/smarterclayton> is this also caused by #18317 <#18317> as #17529 <#17529>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17901 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p_6m74jJxoo0VcSulSp9O3-2-dtDks5tUEVggaJpZM4RIqAv> .

[zgalor]

@tnozicka the bug related to this: https://bugzilla.redhat.com/show_bug.cgi?id=1539987 has been fixed and verified. can this issue be resolved?

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale