oc deploy --latest can't wait for the deployment and return correct exist status

[tnozicka]

For people that are controlling OpenShift from scripts, there needs to be a way to rollout a new deployment and wait for it, returning meaningful exist code.

Right now the closest seems to be using oc deploy --latest --follow that waits for the deployment to finish, but does not tell you if it actually got deployed or failed. (It returns 0 even if the deployment fails; rightfully, because it followed just the logs and that went fine.)

This is a major problem when you want to control the deployment from a script or pipeline.

Let's consider the following use case:

1. build a new image and push it
2. oc deploy --latest to rollout the new version
   2.1 wait for it and make sure it got deployed or fail
3. run tests against the new instance

You have to know that it got deployed successfully at step 2. otherwise it makes no sense to run those tests.

This should be done in the same command with a flag like --wait. It will be quite common operation e.g. in pipelines.

(I am aware this functionality is moving to oc rollout, but I did not want to reference it since it's not merged yet. I am fine with fixing this just there, not in oc deploy.)

@Kargakis This is the one we discussed over IRC a bit.

It's pipeline related so @bparees might be interested or have some good insights.

[bparees]

the openshift jenkins plugin already supports waiting for a deployment to complete if you so choose. you can use that either as a build step, or from the jenkins pipeline plugin.

"Trigger OpenShift Deployment": performs the equivalent of an oc deploy --latest command invocation; it will monitor the resulting ReplicationController's "openshift.io/deployment.phase" annotation to confirm success.

[mfojtik]

@smarterclayton isn't think a use-case for the oc observe?

[tnozicka]

@bparees that's a bit specific just to the subset of pipelines using DSL (which we are not using); but good to know. Additionally, oc deploy --latest does not actually deploys latest when working with imagestreams, I believe. That's why we are waiting for oc rollout latest to finally make it there and use nasty hacks right now.

[0xmichalis]

@mfojtik not really. This is basically what oc rollout status (#9874) is destined to do. @tnozicka wants the wait option to be on the first command that executes the deployment (oc rollout latest / oc deploy --latest) because he's worried about races, and that's a valid concern. I don't think though that any real world deployment can finish in between oc rollout latest; oc rollout status. I am not even sure that any test deployment can finish that fast:)

[0xmichalis]

I don't think though that any real world deployment can finish in between oc rollout latest; oc rollout status. I am not even sure that any test deployment can finish that fast:)

And at the same time have a new deployment start, in order to trick oc rollout status

[0xmichalis]

@mfojtik I think you added the --wait flag for start-build. It should be the same here I guess. I am just not sure I want to duplicate the work rollout status is doing. If others find it useful to have it on rollout latest as a flag though, we can have it.

[bparees]

@bparees that's a bit specific just to the subset of pipelines using DSL (which we are not using);

no, it's usable from any pipeline, whether you use the DSL or not, and also any standard jenkins job using the "trigger a deployment" build step.

but i agree that it would be preferable to not have to use jenkins to get this behavior.

[mfojtik]

@Kargakis oc rollout latest --wait ? (will wait till the deployment is complete and also wait for the replicas to be ready)

[mfojtik]

@Kargakis yeah, --wait proved to be useful for writing extended tests (remove the need for polling the build)

[0xmichalis]

@mfojtik yes and you will need to take care of exit codes. Did you do something similar for start-build?

[mfojtik]

yes

Michal Fojtik

On 9 September 2016 at 17:25:25, Michail Kargakis ([email protected])
wrote:

@mfojtik https://github.com/mfojtik yes and you will need to take care
of exit codes. Did you do something similar for start-build?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10864 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACsaKXdq9smGbEsl9qwdsYjGlmwOBd9ks5qoXplgaJpZM4J5Izh
.

[0xmichalis]

A couple of notes and an alternative proposal:

1. The thing with --wait on deploy is that it will work for some cases and for other cases it will not. For example, think about the outcome of oc deploy dc/foo --follow --wait for the two different cases of returned logs. In the case we return logs from the deployer that's fine, but what happens in the case the deployment is already complete? --wait should be a no-op then. The less flags that behave like this we have, the better. deploy, expose, and other offenders have been a pain to maintain.

2. Since we are adding oc rollout status (Add deployment Conditions and oc rollout status #9874) which is supposed to block until a rollout completes, I prefer not to duplicate the functionality in deploy.

I am proposing a --revision flag in oc rollout status that will pin to a specific revision so we can avoid situations where you deploy N, you start watching with rollout status, N+1 is deployed in the meantime, rollout status now watches for N+1. And for easy programmatic access to the revision being deployed, we can add -o for oc deploy to accept a revision parameter (there are other use-cases for -o as well and the flag is working for all cases). Long story short, if you are a script that is being called on a deployment that is often updated by various components, you should:

REVISION=$(oc deploy dc/foo --latest -o revision)
oc rollout status --revision=$REVISION

@mfojtik @openshift/cli-review @smarterclayton thoughts?

[ksemaev]

@Kargakis please explain - is there any analog for oc rollout status for jobs?
Current oc 3.6.0 tells:
"no status viewer has been implemented for {batch Job}"

are there any workarounds - how to see the rollout status for one job ? (oc get jobs is useless with CI/CD)

[rojcosta]

Hi,

I've been facing the same situation. After reading this thread and done some tests I've realized it's possible to use oc rollout status with CI/CD to determine if the rollout was completed successfully. You can see my tests bellow:

Monitoring the status without stopping the deploy using the web console:

$ oc rollout latest httpd ; oc rollout status dc httpd ; echo $?
deploymentconfig "httpd" rolled out
Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for rollout to finish: 1 old replicas are pending termination...
Waiting for rollout to finish: 1 old replicas are pending termination...
Waiting for latest deployment config spec to be observed by the controller loop...
replication controller "httpd-43" successfully rolled out
0

Monitoring the status while I stop the deploy using the web console:

$ oc rollout latest httpd ; oc rollout status dc httpd ; echo $?
deploymentconfig "httpd" rolled out
Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for rollout to finish: 1 old replicas are pending termination...
error: rollout of replication controller "httpd-44" was cancelled
1

So, it's possible to use the code above to monitor one deploy and fail the CI/CD if necessary. An example of Jenkins's pipeline code could be:

    stage('deploy') {
        steps {
            script {
                sh "oc rollout latest httpd"
                sh "oc rollout status dc httpd"
            }
        }
    }

Besides, I believe is important to know that if you start a build it will start a deployment right after completing the build. So, you must either use something like sh "sleep <time>" or another mechanism to be sure the rollout completed. Otherwhise the oc rollout latest will fail due there is another rollout in progress.