sdn: make pod operation metrics more useful and collectable

[dcbw]

The pod operation error metrics were in the wrong place to capture the
overall pod setup/teardown operation. Move them to capture everything.

Next, the labels of the Latency metric meant that every observation was
a unique metric and no statistics could be determined from them in
aggregate. Change that (and pod errors) to follow the Kubelet dockershim
DockerOperations[Latency|Errors] metric pattern with a label for the
operation instead of the sandbox.

Fixes: #17494

@danwinship @eparis @openshift/networking @knobunc

[danwinship]

LGTM, but while double-checking the err vs result.Err discrepancy between the two pod.go cases, I noticed that there's a bug there a few lines before your patch:

			result.Response, err = json.Marshal(ipamResult)
			if result.Err == nil {

should be if err == nil on the second line. (result.Err is always nil at this point, but then, the json.Marshal() call shouldn't ever fail anyway, so probably this bug has never actually mattered...)

[imcsk8]

LGTM

[dcbw]

@DirectXMan12 might be interesting to you too; the problem we had with NaN was the breakdown of metrics by labels; see the first comment for more info.

[dcbw]

should be if err == nil on the second line. (result.Err is always nil at this point, but then, the json.Marshal() call shouldn't ever fail anyway, so probably this bug has never actually mattered...)

@danwinship fixed in a separate commit here

[knobunc]

LGTM... holding for 3.9

[dcbw]

/test cmd

[dcbw]

/test cmd issue #16317

[dcbw]

/test unit
/test end_to_end

[knobunc]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dcbw, knobunc

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

• pkg/network/OWNERS [dcbw,knobunc]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

[openshift-merge-robot]

Automatic merge from submit-queue.

[akram]

@smarterclayton It is supposed to be fixed here, can you remove the drop for these metrics in https://github.com/openshift/origin/blob/master/examples/prometheus/prometheus.yaml#L556

[smarterclayton]

We don’t use that anymore for real installs. Changes need to be made to cluster monitoring operator. On Aug 28, 2018, at 4:03 AM, Akram Ben Aissi <[email protected]> wrote: @smarterclayton <https://github.com/smarterclayton> It is supposed to be fixed here, can you remove the drop for these metrics in https://github.com/openshift/origin/blob/master/examples/prometheus/prometheus.yaml#L556 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17250 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p6LV84H7cLRrIUbxngMIJuc6Yppzks5uVPlggaJpZM4QYhgk> .