[Feature:Prometheus][Feature:Builds] Prometheus when installed to the cluster should start and expose a secured proxy and verify build metrics [Suite:openshift/conformance/paralle]

[enj]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17617/test_pull_request_origin_extended_conformance_gce/12491/

/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/prometheus/prometheus_builds.go:108
Expected
    <map[string]error | len:1>: {
        "openshift_build_active_time_seconds": {
            s: "query openshift_build_active_time_seconds for tests []prometheus.metricTest{prometheus.metricTest{labels:map[string]string{\"name\":\"frontend-1\"}, greaterThan:true, value:0, success:false}} had results {\"status\":\"success\",\"data\":{\"resultType\":\"vector\",\"result\":[{\"metric\":{\"__name__\":\"openshift_build_active_time_seconds\",\"instance\":\"10.142.0.2:8444\",\"job\":\"kubernetes-controllers\",\"name\":\"mydockertest-1\",\"namespace\":\"extended-test-build-valuefrom-w6mm6-dfgl8\",\"phase\":\"Running\"},\"value\":[1512582511.727,\"1512582414\"]}]}}",
        },
    }
to be empty
/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/prometheus/prometheus_builds.go:171

[bparees]

@gabemontero is this another race where something else produced metrics in the cluster that we didn't expect to see in our test?

[gabemontero]

no it is something else @bparees

the test logic should have ignored the mydockertest-1 build that is in the result data in the error message in the description

and I see the frontend-1 build running and completing during the test's polling window

and I see a prometheus-0 pod up and running (though I thought prometheus was converted to a stateful set ... though maybe that entails a pod under the covers)

I think we need more debug in our ext test on the failure
a) perhaps dump the intermediate results scan results
b) and similar to our jenkins testing, dump the prometheus pod, and see if there was any issue with it scrapping for data from the build controller

[gabemontero]

confirmed the stateful set still maps to a pod (instantiated the example prometheus template manually)

[bparees]

(though I thought prometheus was converted to a stateful set ... though maybe that entails a pod under the covers)

it does. pods are the fundamental unit of any workload. (well, containers are, but you can't have a container w/o a pod)

[gabemontero]

PR with fix merged ... not sure why bot did not close this.

[bparees]

@gabemontero
#17717 (comment)

you had the wrong issue referenced.

[bparees]

(well maybe not wrong but it looks like we had two open?)

[miminar]

Seen it here: https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17783/test_pull_request_origin_extended_conformance_install/5804/

[gabemontero]

the failure in https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17783/test_pull_request_origin_extended_conformance_install/5804/ was the same test as this issue is tracking, but the failure was different than the symptoms originally reported in this issue, and not related to the prometheus build stats being verified.

The sample app build we launched started and failed quickly (deps download glitch I believe), and the active stat query failed.

Separate from the original, precise symptoms which lead me to leave this issue open,
it does beg I question I've asked myself before ... automated testing for the active stat has been tricky, even when developing it, given the batching intervals for prometheus querying the buidl controller. How many flakes need to occur for the active stats to cry uncle and remove it?

Pondering ....

[gabemontero]

If I do delete the active build query, I'll do it in a separate item. Reclosing this issue per original symptom.

[gabemontero]

separate item - #18193