Bug 1247735 - prehook (and posthook?) deployment pods are deleted after exit which makes logs unavailable to users (edit)

[thoraxe]

Currently, after a prehook or posthook pod completes, it is deleted. This means that a normal OpenShift user cannot view the logs because they would need system-level access to directly view the Docker logs.

This behavior wasn't present in beta4.

Some thoughts for solutions:

1. flag in the hook itself "do not delete"
2. flag in dc/rc template "keep x lifecycle pods"
3. flag on project "keep x lifecycle pods"
4. flag on master-config "keep x lifecycle pods"

This is related to #3544 in some ways.

[ironcladlou]

Some clarification:

1. The fact that hook pods are deleted upon deployment completion is consistent with the treatment of the deployer pod itself. When a deployment completes successfully, the deployer pod and hook pods are deleted. If the deployment fails, the deployer and hook pods are preserved.
2. There's potentially surprising hook pod behavior in the context of 1: if a hook pod fails but is ignored (per failure policy) and the deployment itself completes, the hook pods are still deleted. Arguably the failed but ignored hook pods should be preserved even if the deployment completes.

[thoraxe]

There is an issue with #1 in that I may be trying to "develop" a hook. If the hook completes without error but did not do what I thought it should have done, I have no way to troubleshoot that.

I could, in those situations, always cause my hook to fail by throwing a nonzero exit code but then the deployment can not ever continue which may be a part of my hook development.

If the centralized logging story doesn't include hooks in some way, I think that we really need to allow successful hook pods to be kept via configuration.

[ironcladlou]

If the hook completes without error but did not do what I thought it should have done, I have no way to troubleshoot that.

This is reasonable, but it also applies to the primary deployer pod, not just hooks. Short of centralized logging, the only other way I'm seeing around all this would be cleanup policies for deployer pods. When discussing the upstream deployment proposal (kubernetes/kubernetes#12236) I mentioned the idea of generalizing the rolling updater cleanup policy onto the high level strategy resource itself, which people seemed somewhat agreeable to.

/cc @smarterclayton

[ironcladlou]

@thoraxe please close this in favor of https://trello.com/c/5Pt8kGwT.

[thoraxe]

WORKSFORME

[smarterclayton]

Also centralized logging is coming, and it'll be more important to make
historical logs for containers show up than it is today.

On Thu, Aug 20, 2015 at 10:48 AM, Erik M Jacobs [email protected]
wrote:

Closed #3952 #3952.

—
Reply to this email directly or view it on GitHub
#3952 (comment).

Clayton Coleman | Lead Engineer, OpenShift