Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sleep at the end of every hack/env invocation for logs #16604

Merged

Conversation

stevekuznetsov
Copy link
Contributor

When a container exits, the Docker daemon is not very faithful in
collecting all of the logs that the contained processes created before
they finished. We need to sleep at the end of every hack/env call to
ensure that we have enough time to notice what happened and we do not
lose logs.

Signed-off-by: Steve Kuznetsov [email protected]

@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 28, 2017
@openshift-ci-robot openshift-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Sep 28, 2017
@stevekuznetsov
Copy link
Contributor Author

@smarterclayton this obviously doesn't work as it gives the args to whatever command you asked -- do we want to run a shell as the entrypoint and have it do this?

When a container exits, the Docker daemon is not very faithful in
collecting all of the logs that the contained processes created before
they finished. We need to sleep at the end of every `hack/env` call to
ensure that we have enough time to notice what happened and we do not
lose logs.

Signed-off-by: Steve Kuznetsov <[email protected]>
@bparees
Copy link
Contributor

bparees commented Sep 28, 2017

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 28, 2017
@openshift-merge-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bparees

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 28, 2017
@stevekuznetsov
Copy link
Contributor Author

/retest

@stevekuznetsov
Copy link
Contributor Author

@smarterclayton why does cross run here?

@stevekuznetsov
Copy link
Contributor Author

/retest

@smarterclayton
Copy link
Contributor

Because of changes to hack

# container exit races with log collection so we
# need to sleep at the end but preserve the exit
# code of whatever the user asked for us to run
cmd=( '/bin/bash' '-c' "${cmd[*]}; return_code=\$?; sleep 1; exit \${return_code}" )
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty ugly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're not wrong

@stevekuznetsov
Copy link
Contributor Author

@deads2k in cmd i see:

Error from server (Forbidden): User "system:admin" cannot get path "/healthz": User "system:admin" cannot "get" on "/healthz"

WTF?

@soltysh
Copy link
Contributor

soltysh commented Sep 29, 2017

That's #16273, and I think this might be related to etcd problem @mfojtik was chasing recently.

@bparees
Copy link
Contributor

bparees commented Oct 4, 2017

@stevekuznetsov anything holding this up?

@stevekuznetsov
Copy link
Contributor Author

@smarterclayton are you nack or ack here

@smarterclayton
Copy link
Contributor

smarterclayton commented Oct 4, 2017 via email

@stevekuznetsov stevekuznetsov changed the title [WIP] Sleep at the end of every hack/env invocation for logs Sleep at the end of every hack/env invocation for logs Oct 4, 2017
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 4, 2017
@stevekuznetsov
Copy link
Contributor Author

/kind bug

@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Oct 4, 2017
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

1 similar comment
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@stevekuznetsov
Copy link
Contributor Author

/retest

@0xmichalis 0xmichalis removed their assignment Oct 5, 2017
@stevekuznetsov
Copy link
Contributor Author

/retest

1 similar comment
@stevekuznetsov
Copy link
Contributor Author

/retest

@openshift-merge-robot
Copy link
Contributor

Automatic merge from submit-queue (batch tested with PRs 16615, 16604).

@openshift-merge-robot openshift-merge-robot merged commit b15a4b9 into openshift:master Oct 5, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants