test_pull_request_origin_unit sometimes fails without an explanation or (obvious) reason

[tnozicka]

It says:

[WARNING] While the jUnit report found no failed tests, the `go test` process failed.
[WARNING] This usually means that the unit test suite failed to compile.
[ERROR] hack/test-go.sh exited with code 1 after 00h 36m 10s
make: *** [test-unit] Error 1
[ERROR] PID 12927: hack/lib/build/environment.sh:172: `return "${exitcode}"` exited with status 2.
[INFO] 		Stack Trace: 
[INFO] 		  1: hack/lib/build/environment.sh:172: `return "${exitcode}"`
[INFO] 		  2: hack/lib/build/environment.sh:281: os::build::environment::withsource
[INFO] 		  3: hack/env:42: os::build::environment::run
[INFO]   Exiting with code 2.
++ export status=FAILURE
++ status=FAILURE
+ set +o xtrace
########## FINISHED STAGE: FAILURE: RUN UNIT TESTS [00h 38m 35s] ##########

Started to see this here https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18495/test_pull_request_origin_unit/9196/
Went through the S3 artifacts but couldn't see what failed :/ (Apparently something in the test image does.)

The I saw the same just now with https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18495/test_pull_request_origin_unit/9196/ from https://github.com/openshift/origin/pull/18495/files which is very unlikely to make unit tests not to compile given how the change is small and where it's located.

[tnozicka]

/cc @stevekuznetsov

[tnozicka]

/cc @mfojtik

[stevekuznetsov]

Yeah, I've noticed this as well but wasn't really sure what to do here... the process had no output to stderr and just exited non-zero. Wasn't really sure how to debug this in any way. @tnozicka any thoughts?

[tnozicka]

Not really sure what that could be, I guess we hide the error somewhere. Can we run with bash -x for a while inside the image?

[stevekuznetsov]

We could but not sure that would help, the stderr redirect is below that level

https://github.com/openshift/origin/blob/master/hack/test-go.sh#L140

[tnozicka]

For some reason that error file is empty in jenkins artifacts. My thinking was that we would see the exact command that failed as well as its arguments; might help to uncover whatever is causing the flake, or not.

[stevekuznetsov]

It is empty because go test had no output to stderr

[tnozicka]

badly phrased; meant the same thing. The question is why there is no error output but exit status 2.

bash -x was really all I had :(

[stevekuznetsov]

See also #17881

[sosiouxme]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/16658/test_pull_request_origin_unit/9523/ yesterday

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[stevekuznetsov]

/close