Introduce custom Grafana for openshift prometheus. #17037

mrsiano · 2017-10-25T21:05:37Z

A template creates a custom Grafana instance preconfigured to gather Prometheus openshift metrics.
It uses OAuth token to login openshift Prometheus.

mrsiano · 2017-10-25T21:07:34Z

@jeremyeder @smarterclayton @openshift-ci-robot @csrwng @openshift-bot

smarterclayton · 2017-10-26T00:31:09Z

/assign

jeremyeder · 2017-10-26T01:13:54Z

examples/grafana/README.md

+2. ```oc new-app -f grafana-ocp.yaml```
+3. ```oc expose svc grafana-ocp```
+4. grab the grafana url ``` oc get route |awk 'NR==2 {print $2}' ```
+5. grab the ocp token, from openshift master run: ```oc sa get-token management-admin -n management-infra```


This command doesn't make sense. What is management-infra?

Did you create a user called management-infra and a namespace called management-infra? What about the grafana namespace?

@jeremyeder

IIUC, the grafana server does not use access authentication.

This command is a way to get the admin token for the prometheus server, using the prom-proxy [1] authentication proxy.

note:
a. running a playbook using openshift-ansible will create this sa and namespace
b. when installing prometheus using the prometheus.yaml example (line 31 in reademe) it will use this token to authenticate.

[1] https://github.com/openshift/origin/blob/master/examples/prometheus/prometheus.yaml#L113

p.s.

alternatively you can use kube-system namespace and the prometheus sa

but this (using kube-system namespace) will not work with promethus install using openshift-ansible [1]

[1] https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_prometheus/defaults/main.yaml#L5

will change that

jeremyeder · 2017-10-26T01:15:29Z

examples/grafana/grafana-ocp.yaml

+          - name: data
+            mountPath: "/root/go/src/github.com/grafana/grafana/data"
+          command:
+          - "./bin/grafana-server"


Could you please add a route to this template? Here is an example:
https://github.com/openshift/origin/blob/master/examples/prometheus/prometheus.yaml#L52

note:

when implementing the route, note that here the grafana server run without tls, so outside port is 80 not 443 and you do not need the tls part

p.s.

It will be nicer to run grafana with tls ...

for tls, you will have to edit:
https://github.com/grafana/grafana/blob/master/conf/defaults.ini#L29

you can get the key/cert using a secret, see example [1]

[1] https://github.com/openshift/origin/blob/master/examples/prometheus/prometheus.yaml#L70

@jeremyeder @yaacov
It dose not work, I tried to resolve that already thats why I use oc expose command, it needs further investigation, since configuring grafana to run with https[1] needs cert file, the key is not enough.

[1]
https://github.com/grafana/grafana/blob/master/conf/defaults.ini#L29

If it's OK, let's handle TLS in a subsequent PR.

If it's OK, let's handle TLS in a subsequent PR.

Sounds right by me 👍

+1
So ill revert the https for now and will handle it later on by different patch.

@jeremyeder route is added 🍰 , PTAL

Works for me. I am able to create dashboards now :-)

jeremyeder · 2017-10-26T13:33:08Z

examples/grafana/README.md

+Note: make sure to have openshift prometheus deployed.
+(https://github.com/openshift/origin/tree/master/examples/prometheus)
+
+1. ```oc create project grafana```


it is oc create namespace

oc create project does not work

should be:
oc new-project grafana

yes 👍 create namespace is nicer :-)
oc create namespace grafana

Will fix that. 10X

yaacov · 2017-10-27T06:56:55Z

examples/grafana/README.md

+
+1. ```oc create project grafana```
+2. ```oc new-app -f grafana-ocp.yaml```
+3. ```oc expose svc grafana-ocp```


once the route will be added to the yml file, we will not need to expose the service any more

@mrsiano @jeremyeder

Lets submit issue

I understood that in this PR we will add route for http ( port 80, without tls )
and in a later PR update the grafana config file and update route for https ( port 443, with tls )

did I understand correctly ?

p.s

sttring grafana for https is a setting in the config file [1] ( we have the key/cert from openshift )

[1] https://github.com/grafana/grafana/blob/master/conf/defaults.ini#L29

mrsiano · 2017-10-27T09:02:31Z

Sure known issue. Lets submit issue to let grafana use tls auth

…

On Fri, Oct 27, 2017, 09:57 Yaacov Zamir ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In examples/grafana/README.md <#17037 (comment)>: > @@ -0,0 +1,31 @@ +# grafana-ocp + +This template creates a custom Grafana instance preconfigured to gather Prometheus openshift metrics. +It is uses OAuth token to login openshift Prometheus. + + +## To deploy grafana +Note: make sure to have openshift prometheus deployed. +(https://github.com/openshift/origin/tree/master/examples/prometheus ) + +1. ```oc create project grafana``` +2. ```oc new-app -f grafana-ocp.yaml``` +3. ```oc expose svc grafana-ocp``` once the route will be added to the yml file, we will not need to expose the service any more — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#17037 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AM8S4kV-jI7GDJVQxm6oX0g6bQtSICZiks5swX7qgaJpZM4QGqPV> .

mrsiano · 2017-10-28T18:59:30Z

/test extended_networking_minimal

mrsiano · 2017-10-28T18:59:55Z

/test extended_conformance_gce

yaacov · 2017-10-29T05:27:32Z

LGTM 👍

openshift-ci-robot · 2017-10-29T10:58:55Z

@mrsiano: you cannot LGTM your own PR.

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mrsiano

LGTM

@jeremyeder please review

jeremyeder · 2017-10-29T15:12:29Z

examples/grafana/README.md

+2. ```oc new-app -f grafana-ocp.yaml```
+3. grab the grafana url ``` oc get route |awk 'NR==2 {print $2}' ```
+4. grab the ocp token, from openshift master run: ```oc sa get-token prometheus -n kube-system```
+5. browse to grafana via browser and add new prometheus DS to grafana.


change DS to datasource (could be confused with DaemonSet).

mention that the data source URL is the prometheus route, and put an example URL in here

https://prometheus-kube-system.apps.example.com

jeremyeder · 2017-10-29T15:13:06Z

examples/grafana/grafana-ocp.yaml

+          - name: data
+            mountPath: "/root/go/src/github.com/grafana/grafana/data"
+          command:
+          - "./bin/grafana-server"


Works for me. I am able to create dashboards now :-)

jeremyeder · 2017-10-29T15:13:46Z

Few small nits about the README but otherwise this LGTM for phase 1.

smarterclayton · 2017-10-30T01:08:18Z

examples/grafana/Dockerfile

@@ -0,0 +1,35 @@
+FROM centos:7


Is there a reason this needs to be in origin? We're unlikely to build it, so it would be better as a standalone repo.

Hmm, ok, I've just move it, I did not see any reason to have it here either.
@jeremyeder what do you think ?

Specifically the Dockerfile? Fine to move. @mrsiano you can put it in https://github.com/openshift/svt/tree/master/dockerfiles

https://github.com/openshift/svt/tree/master/dockerfiles/grafana-ocp

yaacov · 2017-10-30T10:18:51Z

examples/grafana/README.md

+1. ```oc create namespace grafana```
+2. ```oc new-app -f grafana-ocp.yaml```
+3. grab the grafana url ``` oc get route |awk 'NR==2 {print $2}' ```
+4. grab the ocp token, from openshift master run: ```oc sa get-token prometheus -n kube-system```


@smarterclayton hi, do you have preference on how to get a token for the Prometheus auth proxy ?

Here we assume the namespace is kube-system, but if we install Prometheus using openshift-ansible [1] the namespace will be prometheus.

[1] https://github.com/openshift/openshift-ansible/blob/master/roles/openshift_prometheus/defaults/main.yaml#L5

cc @zgalor @cben @joelddiaz

@yaacov
I've started to work on another patch, to automate the deployment process.

I use this command,
oc get route --all-namespaces |grep -i prometheus |awk '{print $3}'

let's merge this patch for initial stage, then lets work on other things like https, automation and so.
@smarterclayton

mrsiano · 2017-10-30T14:20:10Z

ok, could you please setup LGTM ?

…

On Mon, Oct 30, 2017 at 4:13 PM Jeremy Eder ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In examples/grafana/Dockerfile <#17037 (comment)>: > @@ -0,0 +1,35 @@ +FROM centos:7 Specifically the Dockerfile? Fine to move. @mrsiano <https://github.com/mrsiano> you can put it in https://github.com/openshift/svt/tree/master/dockerfiles — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17037 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AM8S4qDGkKg57y7GU5LUuDDxf8ufPhmsks5sxdmMgaJpZM4QGqPV> .

mrsiano · 2017-10-30T21:59:17Z

@jeremyeder @smarterclayton
Iv'e update this patch to include only the ocp template for grafana, please review

jeremyeder

LGTM

smarterclayton · 2017-11-07T17:14:59Z

Allowed as a tech preview example. I would expect in 3.8/3.9 for you to continue to refine this and deal with all questions related. We are not productizing this.

/lgtm

jotak · 2017-11-10T13:11:17Z

examples/grafana/README.md

+4. grab the ocp token, from openshift master run: ```oc sa get-token prometheus -n kube-system```
+5. browse to grafana datasource's and add new prometheus datasource. 
+6. grab the prometheus url via ```oc get route -n kube-system prometheus |awk 'NR==2 {print $2}'``` and paste the prometheus url e.g https://prometheus-kube-system.apps.example.com
+6. paste the token string at the token field.


There're two items "6"

By the way, maybe the prometheus config in Grafana could be automated with a side-car container? Grafana has a REST API that we can use to add datasource. But this is only possible is prom URL and token can be found programmatically.

Just for information I did some POC for automatic grafana setup with prometheus in openshift, I don't know if it does have some interest for you: https://github.com/jotak/discomon
(The OS template: https://github.com/jotak/discomon/blob/master/prometheus-grafana-discovery.yml ). For POC I didn't care about securing access)

@jotak yes we will drive grafana via api. #17114 is the start.

jeremyeder · 2017-11-10T13:52:21Z

Per #17037 (comment) I thought this should have merged. The outstanding requested changes by @smarterclayton were made (we moved the dockerfile out to our own repo here https://github.com/openshift/svt/tree/master/dockerfiles/grafana-ocp)

Is there some incorrect flag/label to explain why it's stuck?

dobbymoodge · 2017-11-10T21:05:42Z

/retest

jeremyeder · 2017-11-10T21:18:07Z

Just looked at the detail links next to each of the failures. Can't see how they're elated to this PR.

openshift-bot · 2017-11-11T02:22:27Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2017-11-11T10:26:27Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2017-11-11T18:30:28Z

/retest

Please review the full test history for this PR and help us cut down flakes.

mrsiano · 2017-11-11T19:02:39Z

/test extended_conformance_install_update

openshift-bot · 2017-11-12T02:34:27Z

/retest

Please review the full test history for this PR and help us cut down flakes.

A template creates a custom Grafana instance preconfigured to gather Prometheus openshift metrics. It is uses OAuth token to login openshift Prometheus.

mrsiano · 2017-11-12T10:15:59Z

i always seeing this error message [ERROR] Can not determine install and upgrade version for the atomic-openshift-utils package by extended_conformance_install_update.

any idea ?!
@smarterclayton could add back again the lgtm flag, I've just update the readme typo.

mrsiano · 2017-11-14T14:25:03Z

/retest

openshift-ci-robot · 2017-11-14T15:03:34Z

@mrsiano: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
ci/openshift-jenkins/extended_conformance_install_update	`ca061f2`	link	`/test extended_conformance_install_update`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

0xmichalis · 2017-12-07T09:52:38Z

/retest

0xmichalis · 2017-12-07T09:55:18Z

/lgtm

openshift-ci-robot · 2017-12-07T09:55:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kargakis, mrsiano, smarterclayton

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~examples/OWNERS~~ [smarterclayton]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

openshift-merge-robot · 2017-12-07T11:30:41Z

Automatic merge from submit-queue.

Automatic merge from submit-queue (batch tested with PRs 17114, 18274). Grafana template automation. eventually this patch should be compatible and jenkins friendly. #17037

openshift-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 25, 2017

openshift-merge-robot assigned csrwng and danmcp Oct 25, 2017

danmcp removed their assignment Oct 25, 2017

openshift-ci-robot assigned smarterclayton Oct 26, 2017

jeremyeder suggested changes Oct 26, 2017

View reviewed changes

openshift-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 26, 2017

jeremyeder reviewed Oct 26, 2017

View reviewed changes

yaacov reviewed Oct 27, 2017

View reviewed changes

mrsiano commented Oct 29, 2017

View reviewed changes

yaacov approved these changes Oct 29, 2017

View reviewed changes

jeremyeder suggested changes Oct 29, 2017

View reviewed changes

smarterclayton suggested changes Oct 30, 2017

View reviewed changes

yaacov reviewed Oct 30, 2017

View reviewed changes

mrsiano mentioned this pull request Oct 30, 2017

Introduce dockerfile for grafana-ocp. openshift/svt#325

Merged

openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Oct 30, 2017

jeremyeder approved these changes Oct 31, 2017

View reviewed changes

mrsiano mentioned this pull request Oct 31, 2017

Grafana template automation. #17114

Merged

mrsiano mentioned this pull request Nov 7, 2017

How to connect grafana to prometheus #16552

Closed

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 7, 2017

openshift-merge-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 7, 2017

jotak reviewed Nov 10, 2017

View reviewed changes

smarterclayton added the kind/bug Categorizes issue or PR as related to a bug. label Nov 10, 2017

Introduce custome Grafana for openshift prometheus.

ca061f2

A template creates a custom Grafana instance preconfigured to gather Prometheus openshift metrics. It is uses OAuth token to login openshift Prometheus.

openshift-merge-robot removed the lgtm Indicates that a PR is ready to be merged. label Nov 12, 2017

openshift-ci-robot assigned 0xmichalis Dec 7, 2017

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 7, 2017

openshift deleted a comment from openshift-merge-robot Dec 7, 2017

0xmichalis added the retest-not-required label Dec 7, 2017

openshift-merge-robot merged commit 08a7f8d into openshift:master Dec 7, 2017

jeremyeder mentioned this pull request Dec 13, 2017

Add new grafana playbook. openshift/openshift-ansible#6181

Merged

Introduce custom Grafana for openshift prometheus. #17037

Introduce custom Grafana for openshift prometheus. #17037

Conversation

mrsiano commented Oct 25, 2017 • edited Loading

mrsiano commented Oct 25, 2017

smarterclayton commented Oct 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrsiano Oct 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrsiano commented Oct 27, 2017 via email

mrsiano commented Oct 28, 2017

mrsiano commented Oct 28, 2017

yaacov commented Oct 29, 2017

openshift-ci-robot commented Oct 29, 2017

mrsiano left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremyeder commented Oct 29, 2017

Choose a reason for hiding this comment

mrsiano Oct 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrsiano Oct 30, 2017 • edited Loading

Choose a reason for hiding this comment

mrsiano commented Oct 30, 2017 via email

mrsiano commented Oct 30, 2017

jeremyeder left a comment

Choose a reason for hiding this comment

smarterclayton commented Nov 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeremyeder commented Nov 10, 2017

dobbymoodge commented Nov 10, 2017

jeremyeder commented Nov 10, 2017

openshift-bot commented Nov 11, 2017

openshift-bot commented Nov 11, 2017

openshift-bot commented Nov 11, 2017

mrsiano commented Nov 11, 2017

openshift-bot commented Nov 12, 2017

mrsiano commented Nov 12, 2017 • edited Loading

mrsiano commented Nov 14, 2017

openshift-ci-robot commented Nov 14, 2017 • edited Loading

0xmichalis commented Dec 7, 2017

0xmichalis commented Dec 7, 2017

openshift-ci-robot commented Dec 7, 2017

openshift-merge-robot commented Dec 7, 2017

mrsiano commented Oct 25, 2017 •

edited

Loading

mrsiano Oct 26, 2017 •

edited

Loading

mrsiano left a comment •

edited

Loading

mrsiano Oct 30, 2017 •

edited

Loading

mrsiano Oct 30, 2017 •

edited

Loading

mrsiano commented Nov 12, 2017 •

edited

Loading

openshift-ci-robot commented Nov 14, 2017 •

edited

Loading