Build image change trigger silently fails when update is forbidden on free-int #14725

smarterclayton · 2017-06-17T19:43:56Z

v3.6.114 (online version 3.5.0.20)

Had a chained build (output of one build tagged into another as image source). Image pushed, no build started.

smarterclayton · 2017-06-18T15:43:29Z

https://console.free-int.openshift.com/console/project/openshift-devops-monitor/browse/builds/oauth-proxy?tab=history Did not fire twice in a row

smarterclayton · 2017-06-19T02:11:12Z

Testing locally works fine. Could be timing related, but i don't see how.

pweil- · 2017-06-19T06:42:23Z

@soltysh @bparees @mfojtik

mfojtik · 2017-06-19T09:11:57Z

@smarterclayton maybe related to cache/informers? do we have any master logs collected?

smarterclayton · 2017-06-19T13:05:52Z

No, but it was recreatable. We'd have to bump logs high.

bparees · 2017-06-19T19:25:30Z

do you have a dump of both buildconfigs and the imagestreams after the trigger didn't fire?

bparees · 2017-06-19T19:26:38Z

as long as the imagestream actually got updated, i'd expect the buildconfig to eventually see the change and fire (due to reconcilliation) even if it didn't catch the initial event. I don't remember what our reconcile/relist interval is now though (we raised it from 2 minutes w/ the switch to informers as i recall)

smarterclayton · 2017-06-19T21:26:03Z

I'll give you access

…

On Mon, Jun 19, 2017 at 3:26 PM, Ben Parees ***@***.***> wrote: as long as the imagestream actually got updated, i'd expect the buildconfig to eventually see the change and fire (due to reconcilliation) even if it didn't catch the initial event. I don't remember what our reconcile/relist interval is now though (we raised it from 2 minutes w/ the switch to informers as i recall) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#14725 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_pyMQUJUSqKvcgRB1gRo0E9ZN2qKgks5sFstygaJpZM4N9UIE> .

smarterclayton · 2017-06-19T21:39:59Z

apiVersion: v1
items:
- apiVersion: v1
  kind: BuildConfig
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: '{"kind":"BuildConfig","apiVersion":"v1","metadata":{"name":"oauth-proxy-binary","namespace":"openshift-devops-monitor","selfLink":"/oapi/v1/namespaces/openshift-devops-monitor/buildconfigs/oauth-proxy-binary","uid":"dc138c88-29e8-11e7-987b-0a46c474dfe0","resourceVersion":"9538738","creationTimestamp":"2017-04-25T18:56:15Z"},"spec":{"triggers":[{"type":"ConfigChange"},{"type":"GitHub","github":{"secret":"pohRLhJWRgBHOvv6wcfi"}}],"runPolicy":"Serial","source":{"type":"Git","dockerfile":"FROM
        openshift/origin-release:golang-1.7\nCOPY . /go/src/github.com/bitly/oauth2_proxy\nRUN
        cd /go/src/github.com/bitly/oauth2_proxy \u0026\u0026 \\\n    curl -s https://raw.githubusercontent.com/pote/gpm/v1.4.0/bin/gpm
        | bash \u0026\u0026 \\\n    CGO_ENABLED=0 go build -a -tags netgo .\n","git":{"uri":"https://github.com/smarterclayton/oauth2_proxy.git","ref":"cert"}},"strategy":{"type":"Docker","dockerStrategy":{"from":{"kind":"DockerImage","name":"openshift/origin-release:golang-1.7"}}},"output":{"to":{"kind":"ImageStreamTag","name":"oauth-proxy:binaries"}},"resources":{},"postCommit":{},"nodeSelector":null},"status":{"lastVersion":3}}'
    creationTimestamp: 2017-04-25T18:56:15Z
    name: oauth-proxy-binary
    namespace: openshift-devops-monitor
    resourceVersion: "35898790"
    selfLink: /oapi/v1/namespaces/openshift-devops-monitor/buildconfigs/oauth-proxy-binary
    uid: dc138c88-29e8-11e7-987b-0a46c474dfe0
  spec:
    nodeSelector: null
    output:
      to:
        kind: ImageStreamTag
        name: oauth-proxy:binaries
    postCommit: {}
    resources: {}
    runPolicy: Serial
    source:
      dockerfile: |
        FROM openshift/origin-release:golang-1.7
        COPY . /go/src/github.com/bitly/oauth2_proxy
        RUN cd /go/src/github.com/bitly/oauth2_proxy && \
            CGO_ENABLED=0 go build -a -tags netgo .
      git:
        uri: https://github.com/openshift/oauth-proxy.git
      type: Git
    strategy:
      dockerStrategy:
        from:
          kind: DockerImage
          name: openshift/origin-release:golang-1.7
      type: Docker
    triggers:
    - github:
        secret: 
      type: GitHub
    - type: ConfigChange
  status:
    lastVersion: 10
- apiVersion: v1
  kind: Build
  metadata:
    annotations:
      openshift.io/build-config.name: oauth-proxy-binary
      openshift.io/build.number: "10"
      openshift.io/build.pod-name: oauth-proxy-binary-10-build
    creationTimestamp: 2017-06-18T15:27:25Z
    labels:
      buildconfig: oauth-proxy-binary
      openshift.io/build-config.name: oauth-proxy-binary
      openshift.io/build.start-policy: Serial
    name: oauth-proxy-binary-10
    namespace: openshift-devops-monitor
    ownerReferences:
    - apiVersion: build.openshift.io/v1
      controller: true
      kind: BuildConfig
      name: oauth-proxy-binary
      uid: dc138c88-29e8-11e7-987b-0a46c474dfe0
    resourceVersion: "35899726"
    selfLink: /oapi/v1/namespaces/openshift-devops-monitor/builds/oauth-proxy-binary-10
    uid: a1d1aa92-543a-11e7-9c99-0ac586c2eb16
  spec:
    nodeSelector: null
    output:
      pushSecret:
        name: builder-dockercfg-0b1ft
      to:
        kind: ImageStreamTag
        name: oauth-proxy:binaries
    postCommit: {}
    resources: {}
    revision:
      git:
        author:
          email: [email protected]
          name: Clayton Coleman
        commit: 090ac205af3c0db0b22c6dd14c38970e7c1ec926
        committer:
          email: [email protected]
          name: Clayton Coleman
        message: Add support for unified authentication with OpenShift
      type: Git
    serviceAccount: builder
    source:
      dockerfile: |
        FROM openshift/origin-release:golang-1.7
        COPY . /go/src/github.com/bitly/oauth2_proxy
        RUN cd /go/src/github.com/bitly/oauth2_proxy && \
            CGO_ENABLED=0 go build -a -tags netgo .
      git:
        ref: fix_bar
        uri: https://github.com/smarterclayton/oauth2_proxy.git
      type: Git
    strategy:
      dockerStrategy:
        from:
          kind: DockerImage
          name: openshift/origin-release:golang-1.7
      type: Docker
    triggeredBy:
    - message: Manually triggered
  status:
    completionTimestamp: 2017-06-18T15:31:08Z
    config:
      kind: BuildConfig
      name: oauth-proxy-binary
      namespace: openshift-devops-monitor
    duration: 223000000000
    output:
      to:
        imageDigest: sha256:3c1e0d9bd96e69d6577602ce492496c2fb8e20fa259ebdd2306d1b805fddf07e
    outputDockerImageReference: 172.30.215.46:5000/openshift-devops-monitor/oauth-proxy:binaries
    phase: Complete
    stages:
    - durationMilliseconds: 720
      name: FetchInputs
      startTime: 2017-06-18T15:27:32Z
      steps:
      - durationMilliseconds: 720
        name: FetchGitSource
        startTime: 2017-06-18T15:27:32Z
    - durationMilliseconds: 22680
      name: PullImages
      startTime: 2017-06-18T15:27:33Z
      steps:
      - durationMilliseconds: 22680
        name: PullBaseImage
        startTime: 2017-06-18T15:27:33Z
    - durationMilliseconds: 106509
      name: Build
      startTime: 2017-06-18T15:27:56Z
      steps:
      - durationMilliseconds: 106509
        name: DockerBuild
        startTime: 2017-06-18T15:27:56Z
    - name: PostCommit
      startTime: 2017-06-18T15:29:42Z
      steps:
      - name: RunPostCommitHook
        startTime: 2017-06-18T15:29:42Z
    - durationMilliseconds: 84678
      name: PushImage
      startTime: 2017-06-18T15:29:42Z
      steps:
      - durationMilliseconds: 84678
        name: PushDockerImage
        startTime: 2017-06-18T15:29:42Z
    startTimestamp: 2017-06-18T15:27:25Z
- apiVersion: v1
  kind: BuildConfig
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: '{"kind":"BuildConfig","apiVersion":"v1","metadata":{"name":"oauth-proxy","namespace":"openshift-devops-monitor","selfLink":"/oapi/v1/namespaces/openshift-devops-monitor/buildconfigs/oauth-proxy","uid":"dc226384-29e8-11e7-987b-0a46c474dfe0","resourceVersion":"9550313","creationTimestamp":"2017-04-25T18:56:16Z"},"spec":{"triggers":[{"type":"ImageChange","imageChange":{"lastTriggeredImageID":"172.30.215.46:5000/openshift-devops-monitor/oauth-proxy@sha256:1607697ce8417e3f922b6d32ad861b0cfc8d660370c243abd8b1e58cc6f007cf","from":{"kind":"ImageStreamTag","name":"oauth-proxy:binaries"}}}],"runPolicy":"Serial","source":{"type":"Dockerfile","dockerfile":"FROM        centos:7\nMAINTAINER  Clayton
        Coleman \[email protected]\u003e\n\nCOPY oauth2_proxy                         /bin/oauth-proxy\n\nENTRYPOINT
        [ \"/bin/oauth-proxy\" ]\n","images":[{"from":{"kind":"ImageStreamTag","name":"oauth-proxy:binaries"},"paths":[{"sourcePath":"/go/src/github.com/bitly/oauth2_proxy/.","destinationDir":"."}]}]},"strategy":{"type":"Docker","dockerStrategy":{"from":{"kind":"DockerImage","name":"centos:7"}}},"output":{"to":{"kind":"ImageStreamTag","name":"oauth-proxy:latest"}},"resources":{},"postCommit":{},"nodeSelector":null},"status":{"lastVersion":6}}'
    creationTimestamp: 2017-04-25T18:56:16Z
    name: oauth-proxy
    namespace: openshift-devops-monitor
    resourceVersion: "35637856"
    selfLink: /oapi/v1/namespaces/openshift-devops-monitor/buildconfigs/oauth-proxy
    uid: dc226384-29e8-11e7-987b-0a46c474dfe0
  spec:
    nodeSelector: null
    output:
      to:
        kind: ImageStreamTag
        name: oauth-proxy:latest
    postCommit: {}
    resources: {}
    runPolicy: Serial
    source:
      dockerfile: |
        FROM        centos:7
        MAINTAINER  Clayton Coleman <[email protected]>

        COPY oauth2_proxy                         /bin/oauth-proxy

        ENTRYPOINT [ "/bin/oauth-proxy" ]
      images:
      - from:
          kind: ImageStreamTag
          name: oauth-proxy:binaries
        paths:
        - destinationDir: .
          sourcePath: /go/src/github.com/bitly/oauth2_proxy/.
      type: Dockerfile
    strategy:
      dockerStrategy:
        from:
          kind: DockerImage
          name: centos:7
      type: Docker
    triggers:
    - imageChange:
        from:
          kind: ImageStreamTag
          name: oauth-proxy:binaries
        lastTriggeredImageID: 172.30.215.46:5000/openshift-devops-monitor/oauth-proxy@sha256:b9fc7fc96a669a5d204d00c94df8354feabc453c92052e34c5d03d90b51c632f
      type: ImageChange
  status:
    lastVersion: 9
- apiVersion: v1
  kind: Build
  metadata:
    annotations:
      openshift.io/build-config.name: oauth-proxy
      openshift.io/build.number: "9"
      openshift.io/build.pod-name: oauth-proxy-9-build
    creationTimestamp: 2017-06-17T19:44:20Z
    labels:
      buildconfig: oauth-proxy
      openshift.io/build-config.name: oauth-proxy
      openshift.io/build.start-policy: Serial
    name: oauth-proxy-9
    namespace: openshift-devops-monitor
    ownerReferences:
    - apiVersion: build.openshift.io/v1
      controller: true
      kind: BuildConfig
      name: oauth-proxy
      uid: dc226384-29e8-11e7-987b-0a46c474dfe0
    resourceVersion: "35638154"
    selfLink: /oapi/v1/namespaces/openshift-devops-monitor/builds/oauth-proxy-9
    uid: 5b8a2664-5395-11e7-b9da-0ac586c2eb16
  spec:
    nodeSelector: null
    output:
      pushSecret:
        name: builder-dockercfg-0b1ft
      to:
        kind: ImageStreamTag
        name: oauth-proxy:latest
    postCommit: {}
    resources: {}
    serviceAccount: builder
    source:
      dockerfile: |
        FROM        centos:7
        MAINTAINER  Clayton Coleman <[email protected]>

        COPY oauth2_proxy                         /bin/oauth-proxy

        ENTRYPOINT [ "/bin/oauth-proxy" ]
      images:
      - from:
          kind: DockerImage
          name: 172.30.215.46:5000/openshift-devops-monitor/oauth-proxy@sha256:b9fc7fc96a669a5d204d00c94df8354feabc453c92052e34c5d03d90b51c632f
        paths:
        - destinationDir: .
          sourcePath: /go/src/github.com/bitly/oauth2_proxy/.
        pullSecret:
          name: builder-dockercfg-0b1ft
      type: Dockerfile
    strategy:
      dockerStrategy:
        from:
          kind: DockerImage
          name: centos:7
      type: Docker
    triggeredBy:
    - message: Manually triggered
  status:
    completionTimestamp: 2017-06-17T19:45:37Z
    config:
      kind: BuildConfig
      name: oauth-proxy
      namespace: openshift-devops-monitor
    duration: 76000000000
    output:
      to:
        imageDigest: sha256:2cc67649d8022de15402c6edec5d5c678b1c05c4b5c5212b19057085ea516730
    outputDockerImageReference: 172.30.215.46:5000/openshift-devops-monitor/oauth-proxy:latest
    phase: Complete
    stages:
    - durationMilliseconds: 5431
      name: PullImages
      startTime: 2017-06-17T19:44:26Z
      steps:
      - durationMilliseconds: 856
        name: PullInputImage
        startTime: 2017-06-17T19:44:26Z
      - durationMilliseconds: 1679
        name: PullBaseImage
        startTime: 2017-06-17T19:44:30Z
    - durationMilliseconds: 30166
      name: Build
      startTime: 2017-06-17T19:44:32Z
      steps:
      - durationMilliseconds: 30166
        name: DockerBuild
        startTime: 2017-06-17T19:44:32Z
    - name: PostCommit
      startTime: 2017-06-17T19:45:02Z
      steps:
      - name: RunPostCommitHook
        startTime: 2017-06-17T19:45:02Z
    - durationMilliseconds: 34455
      name: PushImage
      startTime: 2017-06-17T19:45:02Z
      steps:
      - durationMilliseconds: 34455
        name: PushDockerImage
        startTime: 2017-06-17T19:45:02Z
    startTimestamp: 2017-06-17T19:44:21Z
- apiVersion: v1
  generation: 1
  image:
    dockerImageLayers:
    - mediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
      name: sha256:6470a24727fbc6105f989afa9a5f545cc3d7ae4479e3dbf58537b3a46ca7c114
      size: 8548572
    - mediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
      name: sha256:837c198294983337d1c89f2025e50980dc03d262203fdae16b32823c4ce55331
      size: 2754524
    - mediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
      name: sha256:d0231338ad7c98b00bb6aed015c264ad1fab5b77edc664e14b81ffca6b9d9149
      size: 184
    - mediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
      name: sha256:6533c6dac909da5c3f6af32aee57f3792fa7c2e4d74d0f14427f583be18d9034
      size: 191511153
    - mediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
      name: sha256:a71b52a5ae9dc36fc4ddd91440b0f88ba37cfa1e1643be8e8bf61636ddb18687
      size: 1082
    - mediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
      name: sha256:b06b2c9b346baae1453eaaf5b679426c43c9563fb0568e570d95ba37dc96e243
      size: 63259875
    - mediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
      name: sha256:dd6405a9d6445ac370348c852c1d28dbdbbbefca4a40f5a302d02ea59488023d
      size: 72292482
    dockerImageManifestMediaType: application/vnd.docker.distribution.manifest.v2+json
    dockerImageMetadata:
      Architecture: amd64
      Config:
        Cmd:
        - /bin/sh
        - -c
        - hack/build-cross.sh
        Env:
        - PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/go/bin
        - VERSION=1.7.5
        - GOARM=5
        - GOPATH=/go
        - GOROOT=/usr/local/go
        - OS_VERSION_FILE=/go/src/github.com/openshift/origin/os-version-defs
        - TMPDIR=/openshifttmp
        - OPENSHIFT_BUILD_NAME=oauth-proxy-binary-10
        - OPENSHIFT_BUILD_NAMESPACE=openshift-devops-monitor
        - OPENSHIFT_BUILD_SOURCE=https://github.com/smarterclayton/oauth2_proxy.git
        - OPENSHIFT_BUILD_REFERENCE=fix_bar
        - OPENSHIFT_BUILD_COMMIT=090ac205af3c0db0b22c6dd14c38970e7c1ec926
        Hostname: dfa0e46aa7ac
        Image: sha256:578980d73fa296fbaa12f53f7d0eb9e96a9ceda7c65458216881b4b25c00479d
        Labels:
          build-date: "20170510"
          io.k8s.description: This is the standard release image for OpenShift Origin
            and contains the necessary build tools to build the platform.
          io.k8s.display-name: OpenShift Origin Release Environment (golang-1.7.5)
          io.openshift.build.commit.author: Clayton Coleman \[email protected]\u003e
          io.openshift.build.commit.date: Wed Jun 14 12:33:47 2017 -0400
          io.openshift.build.commit.id: 090ac205af3c0db0b22c6dd14c38970e7c1ec926
          io.openshift.build.commit.message: Add support for unified authentication
            with OpenShift
          io.openshift.build.commit.ref: HEAD
          io.openshift.build.name: oauth-proxy-binary-10
          io.openshift.build.namespace: openshift-devops-monitor
          io.openshift.build.source-location: https://github.com/smarterclayton/oauth2_proxy.git
          license: GPLv2
          name: CentOS Base Image
          vendor: CentOS
        WorkingDir: /go/src/github.com/openshift/origin
      Container: 196857c6a92f806511d35884bcb9bf02fa424ff28a721a59285fdc61a3ca2649
      ContainerConfig:
        Cmd:
        - /bin/sh
        - -c
        - '#(nop) '
        - LABEL io.openshift.build.commit.author=Clayton Coleman \[email protected]\u003e
          io.openshift.build.commit.date=Wed Jun 14 12:33:47 2017 -0400 io.openshift.build.commit.id=090ac205af3c0db0b22c6dd14c38970e7c1ec926
          io.openshift.build.commit.ref=HEAD io.openshift.build.commit.message=Add
          support for unified authentication with OpenShift io.openshift.build.source-location=https://github.com/smarterclayton/oauth2_proxy.git
          io.openshift.build.name=oauth-proxy-binary-10 io.openshift.build.namespace=openshift-devops-monitor
        Env:
        - PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/go/bin
        - VERSION=1.7.5
        - GOARM=5
        - GOPATH=/go
        - GOROOT=/usr/local/go
        - OS_VERSION_FILE=/go/src/github.com/openshift/origin/os-version-defs
        - TMPDIR=/openshifttmp
        - OPENSHIFT_BUILD_NAME=oauth-proxy-binary-10
        - OPENSHIFT_BUILD_NAMESPACE=openshift-devops-monitor
        - OPENSHIFT_BUILD_SOURCE=https://github.com/smarterclayton/oauth2_proxy.git
        - OPENSHIFT_BUILD_REFERENCE=fix_bar
        - OPENSHIFT_BUILD_COMMIT=090ac205af3c0db0b22c6dd14c38970e7c1ec926
        Hostname: dfa0e46aa7ac
        Image: sha256:578980d73fa296fbaa12f53f7d0eb9e96a9ceda7c65458216881b4b25c00479d
        Labels:
          build-date: "20170510"
          io.k8s.description: This is the standard release image for OpenShift Origin
            and contains the necessary build tools to build the platform.
          io.k8s.display-name: OpenShift Origin Release Environment (golang-1.7.5)
          io.openshift.build.commit.author: Clayton Coleman \[email protected]\u003e
          io.openshift.build.commit.date: Wed Jun 14 12:33:47 2017 -0400
          io.openshift.build.commit.id: 090ac205af3c0db0b22c6dd14c38970e7c1ec926
          io.openshift.build.commit.message: Add support for unified authentication
            with OpenShift
          io.openshift.build.commit.ref: HEAD
          io.openshift.build.name: oauth-proxy-binary-10
          io.openshift.build.namespace: openshift-devops-monitor
          io.openshift.build.source-location: https://github.com/smarterclayton/oauth2_proxy.git
          license: GPLv2
          name: CentOS Base Image
          vendor: CentOS
        WorkingDir: /go/src/github.com/openshift/origin
      Created: 2017-06-18T15:29:42Z
      DockerVersion: 1.12.6
      Id: sha256:d6d9f2a5c4cb19593461c2c4891a43244f6e12c014af0e2b96fbf9e3d05839b9
      Size: 338377583
      apiVersion: "1.0"
      kind: DockerImage
    dockerImageMetadataVersion: "1.0"
    dockerImageReference: 172.30.215.46:5000/openshift-devops-monitor/oauth-proxy@sha256:3c1e0d9bd96e69d6577602ce492496c2fb8e20fa259ebdd2306d1b805fddf07e
    metadata:
      annotations:
        openshift.io/image.managed: "true"
      creationTimestamp: 2017-06-18T15:31:07Z
      name: sha256:3c1e0d9bd96e69d6577602ce492496c2fb8e20fa259ebdd2306d1b805fddf07e
      resourceVersion: "35899708"
      uid: 25ef359d-543b-11e7-9c99-0ac586c2eb16
  kind: ImageStreamTag
  lookupPolicy:
    local: false
  metadata:
    creationTimestamp: 2017-06-18T15:31:07Z
    name: oauth-proxy:binaries
    namespace: openshift-devops-monitor
    resourceVersion: "35932924"
    selfLink: /oapi/v1/namespaces/openshift-devops-monitor/imagestreamtags/oauth-proxy%3Abinaries
    uid: dc03d2cd-29e8-11e7-987b-0a46c474dfe0
  tag: null
kind: List
metadata: {}
resourceVersion: ""
selfLink: ""

tnozicka · 2017-06-20T12:15:00Z

xref to similar issue: https://bugzilla.redhat.com/show_bug.cgi?id=1462131

mfojtik · 2017-06-20T12:48:04Z

@smarterclayton @tnozicka found that after he restart the master the dc will trigger, so I suspect a cache miss issue?

bparees · 2017-06-20T12:58:15Z

that would imply we have a relist/reconciliation bug too though (unless the interval is just so large we didn't get to a relist?)

tnozicka · 2017-06-20T13:02:38Z

@mfojtik @smarterclayton I am suspecting LastTriggeredImage we newer set it and image update gets ignored on https://github.com/openshift/origin/blob/master/pkg/image/trigger/deploymentconfigs/deploymentconfigs.go#L170-L170. The reason why it gets corrected after restart is because with every modification of DC we update LastTriggeredImage in https://github.com/tnozicka/origin/blob/a71e6f14824d4b5bb3eb852104163b01cf32f1fb/pkg/deploy/registry/generator/config_generator.go#L83-L83

EDIT: This was the issue with Build to DC

smarterclayton · 2017-06-20T17:21:53Z

Turns out this was permission problems:

Jun 20 17:14:45 ip-172-31-50-177.ec2.internal atomic-openshift-master-controllers[63831]: I0620 17:14:45.569510   63831 image_trigger_controller.go:372] Finished syncing resource "buildconfigs.build.openshift.io/openshift-devops-monitor/oauth-proxy" (15.375363ms)
Jun 20 17:14:45 ip-172-31-50-177.ec2.internal atomic-openshift-master-controllers[63831]: I0620 17:14:45.569549   63831 image_trigger_controller.go:327] Error syncing resource buildconfigs.build.openshift.io/openshift-devops-monitor/oauth-proxy: buildconfigs "oauth-proxy" is forbidden: build strategy Docker is not allowed
Jun 20 17:14:46 ip-172-31-50-177.ec2.internal atomic-openshift-master-controllers[63831]: I0620 17:14:46.569775   63831 image_trigger_controller.go:370] Started syncing resource "buildconfigs.build.openshift.io/openshift-devops-monitor/oauth-proxy"

The permission attached to my account was dropped by the 3.5->3.6 upgrade or something later than that.

@jupierce can you link the issue we're using to track lost permissions (the bugz)

It's likely this should have fired event - "couldn't trigger image because of permissions" or something

smarterclayton · 2017-06-20T17:43:37Z

I guess this should be an event....

jupierce · 2017-06-20T19:26:23Z

@smarterclayton Instances of missing data described here: https://bugzilla.redhat.com/show_bug.cgi?id=1461340

smarterclayton · 2017-06-20T20:42:18Z

So the exact bug is:

user has permission to create docker builds
creates docker build
trigger controller tries to update build config (via instantiate?) and gets a permission error

smarterclayton · 2017-06-20T20:44:28Z

This means controllers can't trigger builds unless they have permissions to create all build types, which I guess I can just give to it, since technically it had it before.

smarterclayton · 2017-06-21T02:33:09Z

Policy fix up in #14792

bparees · 2017-06-21T03:25:20Z

@smarterclayton why didn't this also affect s2i builds? i assume the image change controller didn't have permission to create/update those either...

smarterclayton · 2017-06-21T15:45:41Z

all users have those by default on those clusters, and the controller inherited it from "system:authenitcated"

smarterclayton self-assigned this Jun 17, 2017

smarterclayton added the component/build label Jun 17, 2017

smarterclayton added this to the 3.6.0 milestone Jun 17, 2017

smarterclayton added the priority/P0 label Jun 17, 2017

pweil- added the kind/bug Categorizes issue or PR as related to a bug. label Jun 19, 2017

tnozicka mentioned this issue Jun 20, 2017

Fix chained DC IST trigger #14773

Merged

smarterclayton mentioned this issue Jun 20, 2017

Report multiple build causes for image change triggers #14777

Merged

smarterclayton changed the title ~~Build image change trigger failed on free-int~~ Build image change trigger silently fails when update is forbidden on free-int Jun 20, 2017

smarterclayton mentioned this issue Jun 21, 2017

Image change trigger must be able to create all build types #14792

Merged

smarterclayton added priority/P1 and removed priority/P0 labels Jun 22, 2017

smarterclayton closed this as completed in #14792 Jun 27, 2017

jim-minter mentioned this issue Oct 31, 2017

allow image trigger controller to create custom builds #17108

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build image change trigger silently fails when update is forbidden on free-int #14725

Build image change trigger silently fails when update is forbidden on free-int #14725

smarterclayton commented Jun 17, 2017 •

edited

Loading

smarterclayton commented Jun 18, 2017

smarterclayton commented Jun 19, 2017

pweil- commented Jun 19, 2017

mfojtik commented Jun 19, 2017

smarterclayton commented Jun 19, 2017 via email

bparees commented Jun 19, 2017

bparees commented Jun 19, 2017

smarterclayton commented Jun 19, 2017 via email

smarterclayton commented Jun 19, 2017

tnozicka commented Jun 20, 2017

mfojtik commented Jun 20, 2017

bparees commented Jun 20, 2017

tnozicka commented Jun 20, 2017 •

edited

Loading

smarterclayton commented Jun 20, 2017

smarterclayton commented Jun 20, 2017

jupierce commented Jun 20, 2017

smarterclayton commented Jun 20, 2017

smarterclayton commented Jun 20, 2017

smarterclayton commented Jun 21, 2017

bparees commented Jun 21, 2017

smarterclayton commented Jun 21, 2017 •

edited

Loading

Build image change trigger silently fails when update is forbidden on free-int #14725

Build image change trigger silently fails when update is forbidden on free-int #14725

Comments

smarterclayton commented Jun 17, 2017 • edited Loading

smarterclayton commented Jun 18, 2017

smarterclayton commented Jun 19, 2017

pweil- commented Jun 19, 2017

mfojtik commented Jun 19, 2017

smarterclayton commented Jun 19, 2017 via email

bparees commented Jun 19, 2017

bparees commented Jun 19, 2017

smarterclayton commented Jun 19, 2017 via email

smarterclayton commented Jun 19, 2017

tnozicka commented Jun 20, 2017

mfojtik commented Jun 20, 2017

bparees commented Jun 20, 2017

tnozicka commented Jun 20, 2017 • edited Loading

smarterclayton commented Jun 20, 2017

smarterclayton commented Jun 20, 2017

jupierce commented Jun 20, 2017

smarterclayton commented Jun 20, 2017

smarterclayton commented Jun 20, 2017

smarterclayton commented Jun 21, 2017

bparees commented Jun 21, 2017

smarterclayton commented Jun 21, 2017 • edited Loading

smarterclayton commented Jun 17, 2017 •

edited

Loading

tnozicka commented Jun 20, 2017 •

edited

Loading

smarterclayton commented Jun 21, 2017 •

edited

Loading