Fix BuildConfigInstantiateFailed warning when lastVersion == 0 #17146
Conversation
openshift-ci-robot
added
the
size/S
|
@openshift/sig-developer-experience ptal |
|
I don't think this is the right fix. You've not just eliminated the warning, isn't this now going to run two builds (the configchangetriggered one and the imagechangetriggered one)? |
|
@coreydaley: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
@coreydaley did you forget about this? |
|
@bparees nope, it's next on my list |
openshift-ci-robot
added
retest-not-required-docs-only
approved
|
Did you delete the branch? That will do it.
Ben Parees | OpenShift
…On Feb 16, 2018 7:18 PM, "Corey Daley" ***@***.***> wrote:
Not sure why this got closed?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#17146 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AEvl3nJcq30s-F32JJZMrqW2ePney-frks5tVkT6gaJpZM4QPqzA>
.
|
|
I reset my branch so that I could start over on this PR and pushed it, weird that it closed, reopening with new code. |
openshift-ci-robot
added
size/S
|
@openshift/sig-developer-experience ptal |
| // Don't emit this event if the BuildRequest.LastVersion equals zero | ||
| // Caused by a race condition between the ImageChangeTrigger vs the BuildConfigChangeTrigger | ||
| // Fixes https://github.com/openshift/origin/issues/16557 | ||
| if *request.LastVersion != int64(0) { |
There was a problem hiding this comment.
request.LastVersion will always be zero in this codepath. so you're going to mask any/all errors.
I think you need to actually look at the error you got from Instantiate (and/or make Instantiate throw a more distinctive error you can look for)
|
@bparees ptal |
| // Caused by a race condition between the ImageChangeTrigger vs the BuildConfigChangeTrigger | ||
| if !strings.Contains(instantiateErr.Error(), fmt.Sprintf("the LastVersion(1) on build config %s/%s does not match the build request LastVersion(0)", bc.Namespace, bc.Name)) { | ||
| c.recorder.Event(bc, kapi.EventTypeWarning, "BuildConfigInstantiateFailed", instantiateErr.Error()) | ||
| } |
There was a problem hiding this comment.
What if a single request comes through (i.e. not a race between ICT and CCT) after the initial BC create with a bad last version? Wouldn't we be masking a legitimate error in that case?
Unfortunately, don't know if I have a super great solution ... perhaps we fetch the BC before performing this filter and see if it was recently created ... where determining what is recent is perhaps kludgy ... but maybe at least an improvement.
Dumb non-transactional databases
There was a problem hiding this comment.
What if a single request comes through (i.e. not a race between ICT and CCT) after the initial BC create with a bad last version?
request comes from where? to where? this logic is all in the buildconfig controller. There are no other buildrequests its going to process.
@coreydaley your check on LastVersion(1) is too specific. I could potentially start multiple builds before the buildconfig controller puts in its request. Then the error would be "the LastVersion(5) on build config..." or whatever.
Just check for the back half: does not match the build request LastVersion(0)
|
@jim-minter any thoughts? |
|
On Mon, Feb 19, 2018 at 4:59 PM, Ben Parees ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In pkg/build/controller/buildconfig/buildconfig_controller.go
<#17146 (comment)>:
> @@ -136,7 +137,11 @@ func (c *BuildConfigController) handleBuildConfig(bc *buildapi.BuildConfig) erro
} else if buildgenerator.IsFatal(err) || kerrors.IsNotFound(err) || kerrors.IsBadRequest(err) || kerrors.IsForbidden(err) {
instantiateErr = fmt.Errorf("gave up on Build for BuildConfig %s due to fatal error: %v", bcDesc(bc), err)
utilruntime.HandleError(instantiateErr)
- c.recorder.Event(bc, kapi.EventTypeWarning, "BuildConfigInstantiateFailed", instantiateErr.Error())
+ // Fixes #16557
+ // Caused by a race condition between the ImageChangeTrigger vs the BuildConfigChangeTrigger
+ if !strings.Contains(instantiateErr.Error(), fmt.Sprintf("the LastVersion(1) on build config %s/%s does not match the build request LastVersion(0)", bc.Namespace, bc.Name)) {
+ c.recorder.Event(bc, kapi.EventTypeWarning, "BuildConfigInstantiateFailed", instantiateErr.Error())
+ }
What if a single request comes through (i.e. not a race between ICT and
CCT) after the initial BC create with a bad last version?
request comes from where? to where? this logic is all in the buildconfig
controller. There are no other buildrequests its going to process.
So you are saying somebody couldn't manually hit the build config rest
endpoint with a HTTP req and interfere with this?
… @coreydaley <https://github.com/coreydaley> your check on LastVersion(1)
is too specific. I could potentially start multiple builds before the
buildconfig controller puts in its request. Then the error would be "the
LastVersion(5) on build config..." or whatever.
Just check for the back half: does not match the build request
LastVersion(0)
—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
<#17146 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADbadDDqoJdDiT_-Yj_zekzhpI6791kyks5tWe63gaJpZM4QPqzA>
.
|
interfere with this how? |
|
@bparees update pushed |
|
On Mon, Feb 19, 2018 at 5:14 PM Ben Parees ***@***.***> wrote:
So you are saying somebody couldn't manually hit the build config rest
endpoint with a HTTP req and interfere with this?
interfere with this how?
create a request that is independent of the concurrent requests @coreydaley
noted coming from the triggers
And when the say the human manually builds ths json for the request he uses
the wrong last version
… —
You are receiving this because you are on a team that was mentioned.
Reply to this email directly, view it on GitHub
<#17146 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADbadAM2ioJGQQH4joKYL9HRhO3FROtfks5tWfJGgaJpZM4QPqzA>
.
|
the code that handles that buildrequest is not the codepath that @coreydaley added his error checking/filtering to. (both codepaths call instantiate, but only the buildconfigcontroller codepath is having the filtering added to it) |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bparees, coreydaley The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these OWNERS Files:
You can indicate your approval by writing |
openshift-ci-robot
added
the
approved
|
Automatic merge from submit-queue. |
Build config instatiate should not emit a warning when
bc.Status.LastVersion != lastVersion and lastVersion == 0
Fixes #16557