-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Gather build step timing information and output using oc describe #13307
Gather build step timing information and output using oc describe #13307
Conversation
@openshift/devex ptal This PR is dependent on code from pull request openshift/source-to-image#694 being updated in vendor/github.com/openshift/source-to-image , so if you clone this and test it, you'll need to also clone my new code in source-to-image on the add_timing_info branch and copy it into the vendor dir.
|
@coreydaley - Took a glance ... struggled with a way to suggest a wrappering of the timestamp before and steps update after into a utility method, where we pass in the function being measured as a parameter into that method, but couldn't sort out a clean way to handle varying parameters and return types of the functions being measured. My guess is you already entertained that, but thought I would mention it just the same. Otherwise, no suggestions, etc. occurred to me. For what it is worth, IGTM. |
@gabemontero Thanks for taking a look. I did entertain your suggestions. See the conversation between @jim-minter and @bparees on openshift/source-to-image#694 for more info on why that wasn't the way this went. |
pkg/build/api/types.go
Outdated
type Steps []StepInfo | ||
|
||
// RecordInfo records details about each build step | ||
func (s *Steps) RecordInfo(stepName StepName, startTime unversioned.Time, stopTime unversioned.Time) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@coreydaley We don't usually add methods to API types, and don't put them here in types.go. You could add a helper function in helper.go if needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@csrwng maybe i'm being too javay, but it seems pretty logical to have a method like this on the data structure itself rather than as a helper. For all the usual data encapsulation/abstraction reasons.
as for where such a definition belongs. personally i prefer to see struct methods defined in the same place as the struct (again, perhaps being too javay), though i can see how that might start to pollute types.go.
is there a reason why we have historically not added heper methods to api types? it's not like it's part of the public api or anything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bparees nothing comes to mind except convention (see types.go in kubernetes). @deads2k?
To my knowledge, convention. The structs in types.go are supposed to be pure value objects without any higher order logic on them. Doing it that way makes it possible to separate the type definitions packages from any logic performed on them.
@smarterclayton smacked my hand a few times for wanting to have helper methods attached. Go is nasty no matter what you do, so it didn't bother me much.
pkg/build/api/v1/types.go
Outdated
StepName StepName `json:"stepName,omitempty" protobuf:"bytes,1,opt,name=stepName"` | ||
|
||
// StartTime identifies when this step started. | ||
StartTime unversioned.Time `json:"startTime,omitempty" protobuf:"bytes,2,opt,name=startTime"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are clients supposed to assume that these don't overlap? What happens when they do overlap?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct, they should not overlap the way that we have wrapped the steps that occur. Maybe adding some tests that check for overlapping steps would be useful here.
pkg/build/api/v1/types.go
Outdated
@@ -176,8 +176,46 @@ type BuildStatus struct { | |||
|
|||
// output describes the Docker image the build has produced. | |||
Output BuildStatusOutput `json:"output,omitempty" protobuf:"bytes,10,opt,name=output"` | |||
|
|||
// steps contains details about each build step. | |||
Steps Steps `json:"steps,omitempty" protobuf:"bytes,11,opt,name=steps"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At some point, jenkins needs to record status in a real object. When it does that, I assume it'll be stages, with steps in them. For this PR can you describe this in terms of terminology jenkins uses, and the pipeline plugin will use, so we can end up with a struct in Status that looks as much as possible like the destination will be?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
here's the jenkins metadata we record into the build object. it has "stages" and "stageflownodes" within the stages, which appear to be the steps.
for both stages and stageflownodes/steps it provides:
startTimeMillis
durationMillis
pauseTimeMillis
openshift.io/jenkins-status-json: '{"_links":{"self":{"href":"https://jenkins-c1.router.default.svc.cluster.local/job/c1-sample-pipeline/1/wfapi/describe"}},"id":"1","name":"#1","status":"SUCCESS","startTimeMillis":1489611336514,"endTimeMillis":0,"durationMillis":0,"queueDurationMillis":97492,"pauseDurationMillis":0,"stages":[{"_links":{"self":{"href":"https://jenkins-c1.router.default.svc.cluster.local/job/c1-sample-pipeline/1/execution/node/6/wfapi/describe"}},"id":"6","name":"build","execNode":"","status":"SUCCESS","startTimeMillis":1489611434006,"durationMillis":42214,"pauseDurationMillis":0,"stageFlowNodes":[{"_links":{"self":{"href":"https://jenkins-c1.router.default.svc.cluster.local/job/c1-sample-pipeline/1/execution/node/7/wfapi/describe"}},"id":"7","name":"Trigger
OpenShift Build","execNode":"","status":"SUCCESS","startTimeMillis":1489611434138,"durationMillis":42074,"pauseDurationMillis":0,"parentNodes":["6"]}]},{"_links":{"self":{"href":"https://jenkins-c1.router.default.svc.cluster.local/job/c1-sample-pipeline/1/execution/node/11/wfapi/describe"}},"id":"11","name":"deploy","execNode":"","status":"SUCCESS","startTimeMillis":1489611476231,"durationMillis":20327,"pauseDurationMillis":0,"stageFlowNodes":[{"_links":{"self":{"href":"https://jenkins-c1.router.default.svc.cluster.local/job/c1-sample-pipeline/1/execution/node/12/wfapi/describe"}},"id":"12","name":"Trigger
OpenShift Deployment","execNode":"","status":"SUCCESS","startTimeMillis":1489611476253,"durationMillis":20299,"pauseDurationMillis":0,"parentNodes":["11"]}]}]}'
So if we want to mirror the jenkins api, we should add the following struct to our build objects:
Stage[] {
name
startTime
duration
Steps[] {
name
startTime
duration
}
@smarterclayton @jwforres tentative agreement?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seems reasonable, @spadgett
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can confirm the stage flow nodes are the steps.
The nodes also have an ID and a parent node. Parent node might be important if we want to tell if steps are parallel vs sequential (although I'd need to look at this more).
There is some doc on the JSON here:
https://github.com/jenkinsci/pipeline-stage-view-plugin/tree/master/rest-api
I'm ok with being consistent. Be sure to call the field
durationMilliseconds if you do so
…On Wed, Mar 15, 2017 at 7:31 PM, Sam Padgett ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In pkg/build/api/v1/types.go
<#13307 (comment)>:
> @@ -176,8 +176,46 @@ type BuildStatus struct {
// output describes the Docker image the build has produced.
Output BuildStatusOutput `json:"output,omitempty" protobuf:"bytes,10,opt,name=output"`
+
+ // steps contains details about each build step.
+ Steps Steps `json:"steps,omitempty" protobuf:"bytes,11,opt,name=steps"`
I can confirm the stage flow nodes are the steps.
The nodes also have an ID and a parent node. Parent node might be
important if we want to tell if steps are parallel vs sequential (although
I'd need to look at this more).
There is some doc on the JSON here:
https://github.com/jenkinsci/pipeline-stage-view-plugin/
tree/master/rest-api
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#13307 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABG_p_afbyFzaeefwwQuucZESJnr_IsXks5rmHTogaJpZM4MXAi5>
.
|
@openshift/devex ptal
Suggestions for Stage/Step names & mappings are welcome! |
pkg/build/api/types.go
Outdated
StartTime unversioned.Time | ||
|
||
// EndTime identifies when this stage ended. | ||
EndTime unversioned.Time |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
drop endtime
pkg/build/api/types.go
Outdated
StartTime unversioned.Time | ||
|
||
// EndTime identifies when this step completed. | ||
EndTime unversioned.Time |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
drop endtime
pkg/build/api/types.go
Outdated
StepFetchSource StepName = "FetchSource" | ||
|
||
// StepDockerPullImage pulls the docker image. | ||
StepDockerPullImage StepName = "DockerPullImage" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should have different step names for the different images we pull (builder image, base image, input image, etc)
pkg/build/api/types.go
Outdated
// Valid StageNames | ||
const ( | ||
// StageFetch fetches the source. | ||
StageFetch StageName = "Fetch" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FetchInputs
pkg/build/api/types.go
Outdated
StageBuild StageName = "Build" | ||
|
||
// StagePost executes post steps. | ||
StagePost StageName = "Post" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PostCommit ?
pkg/build/api/types.go
Outdated
StagePost StageName = "Post" | ||
|
||
// StagePushImages pushes the docker images. | ||
StagePushImages StageName = "PushImages" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PushImage (there's only one)
@bparees |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is close enough that you should start working on tests.
@openshift/api-review I think the api is ready for another look. We've roughly mirrored the jenkins stage/step model of having start time + duration for each stage+step. |
@bparees ptal |
[test] |
[testextended][extended:core(builds)] |
pkg/build/api/v1/types.go
Outdated
} | ||
|
||
// Stages is a slice of build stages that have been recorded. | ||
type Stages []StageInfo |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally we don't do this unless there's a reason that has to do with protobuf serialization (null vs empty behavior), and that doesn't apply here. Recommend removing this alias type.
pkg/build/api/v1/types.go
Outdated
StartTime unversioned.Time `json:"startTime,omitempty" protobuf:"bytes,2,opt,name=startTime"` | ||
|
||
// Duration identifies how long this stage ran. | ||
Duration unversioned.Duration `json:"duration,omitempty" protobuf:"bytes,3,opt,name=duration"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Duration is not allowed in the public API. You must use DurationMilliseconds
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And an int64
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we're using time.Duration in the BuildStatus, is not not acceptable?
Duration time.Duration json:"duration,omitempty" protobuf:"varint,7,opt,name=duration,casttype=time.Duration"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and @smarterclayton are you saying the value must be stored as milliseconds?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the field must be:
DurationMilliseconds int64 `json:"durationMilliseconds"`
and when you store it you need to do the conversion to milliseconds.
pkg/build/api/v1/types.go
Outdated
StepPullInputImage StepName = "PullInputImage" | ||
|
||
// StepDockerPushImage pushes the docker image to the registry. | ||
StepPushDockerImage StepName = "PushDockerImage" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PushImage
instead of PushDockerImage
. Only reference Docker
directly if it is relevant to the inputs (i.e. Docker strategy is compatible with docker build
and so it's appropriate) - otherwise use the generic Image
.
pkg/build/api/v1/types.go
Outdated
// Valid StepNames | ||
const ( | ||
// StepExecPostCommitHook executes the buildconfigs post commit hook. | ||
StepExecPostCommitHook StepName = "ExecPostCommitHook" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than Exec
, why not RunPostCommitHook
? Reads better at least to me.
pkg/build/api/v1/types.go
Outdated
// StageInfo contains details about a build stage. | ||
type StageInfo struct { | ||
// StageName is the identifier for each build stage. | ||
StageName StageName `json:"stageName,omitempty" protobuf:"bytes,1,opt,name=stageName"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not just Name
? Will a Stage
ever have a different Name
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we have constants defined like StageFetchInputs StageName = "FetchInputs",
so "Name" would be pretty generic there?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
change the field name, not the type name.
pkg/build/api/v1/types.go
Outdated
// StepInfo contains details about a build step. | ||
type StepInfo struct { | ||
// StepName is the identifier for each build step. | ||
StepName StepName `json:"stepName,omitempty" protobuf:"bytes,1,opt,name=stepName"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Name
instead of StepName
unless we can make an argument that there will ever be a more basic "name" for a step that isn't this field. Basically build.status.steps[0].name
reads better than build.status.steps[0].stepName
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have constants defined for StepName, so using just Name seems to generic there and would overlap the StageName
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
he's talking about the field name, not the type name.
pkg/build/api/helpers.go
Outdated
if stageVal.Name == stageName { | ||
stages[stageKey].DurationMilliseconds = endTime.Rfc3339Copy().Time.Sub(stages[stageKey].StartTime.Rfc3339Copy().Time).Nanoseconds() / int64(time.Millisecond) | ||
if len(stages[stageKey].Steps) == 0 { | ||
stages[stageKey].Steps = make([]StepInfo, 0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And again...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are only displaying using second-level precision, I was using that so we don't get something like 2.4738495747s in the display.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Corrected in next push
pkg/build/api/helpers.go
Outdated
stages = make([]StageInfo, 0) | ||
} | ||
|
||
// If the stage already exists in the slice, update the endTime and DurationMilliseconds, and append the new step. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comment needs an update.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, will push an update
pkg/build/api/helpers.go
Outdated
} | ||
|
||
// MergeStageAndStepInfo merges one Stages slice into another. | ||
func MergeStageAndStepInfo(stages []StageInfo, stagesToMerge []StageInfo) ([]StageInfo, int64) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this function is quite brittle - if for some reason in the future this is called with two []StageInfos which each contain a StageInfo with an identical name, but different StartTimes, I think bad things could happen.
Is it possible to get rid of this function entirely by passing a *StageInfo argument into functions like fetchSource, and not increasing the number of return values? I think that would be cleaner if feasible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure i love carrying a StageInfo object around through all our build calls (as opposed to letting the calls return the stages they created and merging them in a common place) but i agree there's a risk of breakage here.
Perhaps we could compromise on having this function throw an error if the merge includes the same stage name twice w/ different start times? (if we are using this properly, i don't think we have that scenario)
alternatively we could always use the earliest start time for the stage if the stage appears twice, which also seems like reasonable behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The way this works is that Stages are like categories, and Steps are like items in those categories. The first Step that is added sets the start time for the Stage, and each step that is added extends the duration to that Step's stop time. So the question would seem to be, what if a second Step with the same name is added? Should we replace the first one? Go ahead and add this second Step to the Stage? or throw an error?
We shouldn't ever get a duplicate step, but if we did, it would not hurt anything, it would just dump it into the Step slice under the correct Stage and keep on chugging along.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that's @jim-minter's concern. His concern is w/ regard to the behavior when you have duplicate stage names. (but duplicate step names within a stage could be a problem too, certainly, depending how those are being merged)
pkg/build/api/helpers.go
Outdated
for _, stage := range stagesToMerge { | ||
for _, step := range stage.Steps { | ||
stages = RecordStageAndStepInfo(stages, stage.Name, step.Name, step.StartTime, unversioned.NewTime(step.StartTime.Add(time.Duration(step.DurationMilliseconds)*time.Millisecond))) | ||
subStepDuration = subStepDuration + step.DurationMilliseconds |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the fact that you're returning subStepDuration is indicative of an encapsulation problem. I think that any code that uses the returned subStepDuration should be pulled out to a helper.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i think there's a more fundamental issue in thinking we even need this information, i commented on the relevant code below.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Corrected in next push
pkg/build/builder/docker.go
Outdated
@@ -66,7 +68,14 @@ func (d *DockerBuilder) Build() error { | |||
if err != nil { | |||
return err | |||
} | |||
sourceInfo, err := fetchSource(d.dockerClient, buildDir, d.build, initialURLCheckTimeout, os.Stdin, d.gitClient) | |||
startTime := unversioned.Now() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bparees I still think that we should be using something like a helper function and closure, for readability and maintainability, to avoid blow-up of 1 line to 8 at measurement points.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
well i think this block is more complicated than it should be. it should just need to be:
getStartTime
runStepLogic
merge(?)
recordStep/Stage
As it stands, it's 7 lines and i think at least 3 of these lines are unnecessary(variable declaration, storing stopTime temporarily, and the logic related to subStepDuration), and most invocations wouldn't have the merge step (and the closure wouldn't help with the merge step, i don't think)
In any case, I'm still not convinced creating closures is going to result in more readability (or reduced linecounts), since it means creating wrapper functions around the blocks of code we want to measure, blocks that otherwise serve no purpose, in exchange for a slightly more awkward callstack and return path.
pkg/build/builder/docker.go
Outdated
stopTime := unversioned.Now() | ||
var subStepDuration int64 | ||
d.build.Status.Stages, subStepDuration = api.MergeStageAndStepInfo(d.build.Status.Stages, stages) | ||
stepDuration := int64(stopTime.Time.Sub(startTime.Time)) - subStepDuration |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
at a minimum this needs some comments but fundamentally it seems like we're doing more manipulation than i'd expect...what's going on here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it seems like you're treating "fetchSource()" as a step, but fetchSource() itself also has steps. i'd expect one or the other (and i'd expect it to be the latter).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Corrected in next push
pkg/build/builder/sti.go
Outdated
var subStepDuration int64 | ||
s.build.Status.Stages, subStepDuration = api.MergeStageAndStepInfo(s.build.Status.Stages, stages) | ||
stepDuration := int64(stopTime.Time.Sub(startTime.Time)) - subStepDuration | ||
s.build.Status.Stages = api.RecordStageAndStepInfo(s.build.Status.Stages, api.StageFetchInputs, api.StepFetchSource, startTime, unversioned.NewTime(startTime.Add(time.Duration(stepDuration)))) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same issue here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Corrected in next push
pkg/build/builder/docker.go
Outdated
startTime := unversioned.Now() | ||
sourceInfo, stages, err := fetchSource(d.dockerClient, buildDir, d.build, initialURLCheckTimeout, os.Stdin, d.gitClient) | ||
stopTime := unversioned.Now() | ||
var subStepDuration int64 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you don't need to declare this, you can just use := below (assuming this value even sticks around, but per my other comments i hope it doesn't)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Corrected in next push
@bparees ptal |
test/extended/builds/build_timing.go
Outdated
verifyStages(br.Build.Status.Stages, expectedBuildStages) | ||
|
||
}) | ||
g.It("should fail if a duplicate step is added", func() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be a unit test on the timing package.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated in next push
@bparees updated per your suggestions, ptal |
test flake #13739 |
test flake #11114 |
@bparees ptal, looks like it was just a couple of test flakes (as noted above) |
test flake #13694 |
pkg/build/api/helpers.go
Outdated
} | ||
|
||
// AppendStageAndStepInfo appends the step info from one stages slice into another. | ||
func AppendStageAndStepInfo(stages []StageInfo, stagesToMerge []StageInfo) ([]StageInfo, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is stages=append(stages,otherstages)
not sufficient for your use case? (do you actually have a use case where "stages" and "otherstages" have an overlap in stage names and you need to merge things at a step level?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it is possible
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i'm not sure "it is possible" answers my question.
- is the code currently doing so?
- if not, do you have an explicit scenario in mind where it would?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both the origin code and the source-to-image code record steps in the StagePullImages stage, so those would need to be merged together. There is also the potential as we record more fine grained details that there will be more steps recorded for each stage, again needing to be merged instead of just appended.
pkg/build/builder/sti.go
Outdated
return err | ||
} | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why isn't this using your merge function?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The stages/steps returned from the s2i code is the wrong type. I guess I could just cast everything in the merge function.
@@ -283,8 +297,17 @@ func (s *S2IBuilder) Build() error { | |||
handleBuildStatusUpdate(s.build, s.client, nil) | |||
return err |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shouldn't you add the stage info up here, before returning?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
@@ -293,6 +316,11 @@ func (s *S2IBuilder) Build() error { | |||
return err |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same here...add the stage info before returning. I still care how long my postcommithook took to run, even if it failed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
pkg/build/builder/docker.go
Outdated
if err = pullImage(d.dockerClient, imageName, pullAuthConfig); err != nil { | ||
d.build.Status.Phase = api.BuildPhaseFailed | ||
d.build.Status.Reason = api.StatusReasonPullBuilderImageFailed | ||
d.build.Status.Message = api.StatusMessagePullBuilderImageFailed | ||
handleBuildStatusUpdate(d.build, d.client, nil) | ||
return fmt.Errorf("failed to pull image: %v", err) | ||
} | ||
if d.build.Status.Stages, err = api.RecordStageAndStepInfo(d.build.Status.Stages, api.StagePullImages, api.StepPullBaseImage, startTime, unversioned.Now()); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you've got the context object, why not consistently call timing.RecordNewStep everywhere? It's a bit confusing having two mechanisms for recording new steps being used in different places.
test/extended/builds/build_timing.go
Outdated
expectedDurations, ok := expectedStages[string(stage.Name)] | ||
if !ok { | ||
fmt.Fprintf(g.GinkgoWriter, "Unexpected stage %v was encountered", stage.Name) | ||
o.Expect(ok).To(o.BeTrue()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
when calling gomega from within a helper function, use ExpectWithOffset so the error reporting shows you who the actual caller.
test/extended/builds/build_timing.go
Outdated
if expectedDurations[0] != "" { | ||
expectedMinDuration, _ := time.ParseDuration(expectedDurations[0]) | ||
if stage.DurationMilliseconds < expectedMinDuration.Nanoseconds()/int64(time.Millisecond) { | ||
fmt.Fprintf(g.GinkgoWriter, "Stage %v ran for %v, expected greater than %v", stage.Name, stage.DurationMilliseconds, expectedMinDuration) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should this be a failure?
test/extended/builds/build_timing.go
Outdated
} | ||
expectedMaxDuration, _ := time.ParseDuration(expectedDurations[1]) | ||
if stage.DurationMilliseconds > expectedMaxDuration.Nanoseconds()/int64(time.Millisecond) { | ||
fmt.Fprintf(g.GinkgoWriter, "Stage %v ran for %v, expected less than %v", stage.Name, stage.DurationMilliseconds, expectedMaxDuration) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should this be a failure?
@bparees ptal, updated per your suggestions |
pkg/build/builder/sti.go
Outdated
defer func() { | ||
s.build.Status.Stages = api.AppendStageAndStepInfo(s.build.Status.Stages, timing.GetStages(ctx)) | ||
handleBuildStatusUpdate(s.build, s.client, nil) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
delete this newline.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
Evaluated for origin testextended up to 915246a |
continuous-integration/openshift-jenkins/testextended SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended/222/) (Base Commit: 184b859) (Extended Tests: core(builds)) |
Evaluated for origin test up to 915246a |
continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin/912/) (Base Commit: 092bd57) |
@bparees ptal |
[merge] |
Queue is being held for rebase
|
i know..I assume you guys are using some alternate tag for merging and I can just leave this tag here for when you re-enable the queue for the rest of us? |
Evaluated for origin merge up to 915246a |
continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_request_origin/486/) (Base Commit: 33052f6) (Image: devenv-rhel7_6183) |
Gathers build step timing information for significant step of the build process and makes it available to the 'oc describe' command.
Dependent on openshift/source-to-image#694