Update deployment logs and surge on recreate #8787

smarterclayton · 2016-05-07T02:37:37Z

Don't use glog in deployment output, also surge on recreate. Make deployment truly reentrant, with the --until=[start|pre|mid|0%|50%|100%] condition to gate how far to go:

$ openshift-deploy --until=pre
--> pre: Running hook pod ...
my hook pod ran
--> pre: Success
--> pre hook completed
$ openshift-deploy --until=50%
--> pre: Hook pod already succeeded
--> Scaling deployment-2 from 0 to 5
    Scaling deployment-2 up to 3
    Scaling deployment-1 down to 2
--> Reached 50% (currently 60%)
$ sleep 5
$ curl http://myservice:8080/metrics
user metrics ok
$ openshift-deploy
--> pre: Hook pod already succeeded
--> Scaling deployment-2 from 3 to 5
    Scaling deployment-2 up to 5
    Scaling deployment-1 down to 0
--> Success

Will allow custom deployments to be scriptable.

smarterclayton · 2016-05-08T01:31:24Z

@ironcladlou @Kargakis this will allow someone to create a custom deployment that is just the openshift-deploy image and script a deployment with bash.

smarterclayton · 2016-05-09T16:10:34Z

[test]

0xmichalis · 2016-05-10T15:55:25Z

pkg/cmd/infra/deployer/deployer.go

 		}
 	}

+	if d.until == "start" {
+		return strategy.NewConditionReachedErr("Ready to start deployment")


Umm, these conditions are not errors.

They're exceptions to the normal flow - it's handled at the top level.
Effectively we have defined a structured termination condition.

On Tue, May 10, 2016 at 11:55 AM, Michail Kargakis <[email protected]

wrote:

In pkg/cmd/infra/deployer/deployer.go
#8787 (comment):

} }

if d.until == "start" {

return strategy.NewConditionReachedErr("Ready to start deployment")

Umm, these conditions are not errors.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/openshift/origin/pull/8787/files/a96fb629083f11e94f9496eb3fd5d5fa15ab9245#r62698862

0xmichalis · 2016-05-12T22:44:28Z

@ironcladlou I think that the patch in the rolling updater will help us move DeploymentConfigs to use ProgressDeadlineSeconds (obsoletes TimeoutSeconds in rolling settings and ActiveDeadlineSeconds in the deployer?). In the upstream controller we will use Conditions on the Deployment to track progress, here we can use annotations on the replication controllers.

0xmichalis · 2016-05-12T23:16:37Z

In the upstream controller we will use Conditions on the Deployment to track progress, here we can use annotations on the replication controllers.

Or we could use Conditions on the deploymentconfig by decoding it once and then getting/updating it accordingly. I think I still prefer annotations on the rcs.

0xmichalis · 2016-05-12T23:29:11Z

obsoletes TimeoutSeconds in rolling settings and ActiveDeadlineSeconds in the deployer?

Nevermind about ADS. I had a look at the rolling updater - TimeoutSeconds is almost exactly what ProgressDeadlineSeconds will be upstream. I may even change it to be exactly the same. So this patch doesn't help us the way I thought it would. But we already have PDS!:)

smarterclayton · 2016-05-13T16:05:48Z

Only other comments on this?

0xmichalis · 2016-05-13T16:16:31Z

pkg/deploy/strategy/interfaces.go

+	if !strings.HasSuffix(until, "%") {
+		return 0, false
+	}
+	until = until[:len(until)-1]


Can't you reuse PercentageBetween here?

Need to get the percentage.

Right, you can do the other way around.

0xmichalis · 2016-05-13T16:40:45Z

Is there an upstream PR for the rolling updater?

0xmichalis · 2016-05-13T16:42:21Z

Looks good in general

smarterclayton · 2016-05-13T16:44:00Z

No I'll open once once you're comfortable with the changes.

On Fri, May 13, 2016 at 12:40 PM, Michail Kargakis <[email protected]

wrote:

Is there an upstream PR for the rolling updater?

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#8787 (comment)

0xmichalis · 2016-05-14T15:20:58Z

I was thinking of exposing --until (or something similar to it that would direct the percentage of my rollout) via the deployment api. Then in combination with pausing/resuming and proportional scaling someone could do A/B with a single deploymentconfig. The deployer needs to not take into account the time it is paused.

smarterclayton · 2016-05-14T17:33:03Z

I think that would be useful. Would be worth discussing upstream.

I'm going to extract the upstream PR now and squash.

On Sat, May 14, 2016 at 11:21 AM, Michail Kargakis <[email protected]

wrote:

I was thinking of exposing --until (or something similar to it that would
direct the percentage of my rollout) via the deployment api. Then in
combination with pausing/resuming and proportional scaling someone could do
A/B with a single deploymentconfig. The deployer needs to not take into
account the time it is paused.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#8787 (comment)

smarterclayton · 2016-05-14T17:37:33Z

Actually, not going to squash since the commits are mostly independent.
Ready for final review.

On Sat, May 14, 2016 at 1:33 PM, Clayton Coleman [email protected]
wrote:

I think that would be useful. Would be worth discussing upstream.

I'm going to extract the upstream PR now and squash.

On Sat, May 14, 2016 at 11:21 AM, Michail Kargakis <
[email protected]> wrote:

I was thinking of exposing --until (or something similar to it that would
direct the percentage of my rollout) via the deployment api. Then in
combination with pausing/resuming and proportional scaling someone could do
A/B with a single deploymentconfig. The deployer needs to not take into
account the time it is paused.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#8787 (comment)

0xmichalis · 2016-05-14T17:43:17Z

Still havent played with this PR but I have a question. I am running a 50% deployment. After it finishes and the deployer exits successfully, the deployment will be marked Complete and then the deploymentconfig controller on the next resync will scale it up to 100% and scale down the old 50%, right?

0xmichalis · 2016-05-14T17:46:51Z

Actually if this is correct then the controller will not scale up the deployment but scale down the deploymentconfig because of the scaling support it provides to old clients

smarterclayton · 2016-05-14T17:51:39Z

Yeah, you'd more than likely get the old behavior.

The expectation is that you'd always finish this script with "run to
completion".

On Sat, May 14, 2016 at 1:46 PM, Michail Kargakis [email protected]
wrote:

Actually if this is correct then the controller will not scale up the
deployment but scale down the deploymentconfig because of the scaling
support it provides to old clients

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#8787 (comment)

Useful for iterative testing of deployment logic, or running an external deployer.

When re-entering deployments, we may not need to even check the scale.

Allows callers to terminate early.

Allows a user to script a deployment to reach certain conditions, then take action: $ openshift-deploy --until=pre --> pre: Running hook pod ... my hook pod ran --> pre: Success --> pre hook completed $ openshift-deploy --until=50% --> pre: Hook pod already succeeded --> Scaling deployment-2 from 0 to 5 Scaling deployment-2 up to 3 Scaling deployment-1 down to 2 --> Reached 50% (currently 60%) $ sleep 5 $ curl http://myservice:8080/metrics user metrics ok $ openshift-deploy --> pre: Hook pod already succeeded --> Scaling deployment-2 from 3 to 5 Scaling deployment-2 up to 5 Scaling deployment-1 down to 0 --> Success Will allow custom deployments

Allow customParams to be specified when type is Recreate or Rolling. Allow image to be empty, add validation for resources and environment. Add a custom deployment extended test.

Even in surge

Shows info about custom deployments that also having rolling update. Makes multiline command printing better. Adds printing tag images hooks.

openshift-bot · 2016-05-16T15:00:25Z

Evaluated for origin test up to f92fe67

0xmichalis · 2016-05-16T15:03:28Z

LGTM

smarterclayton · 2016-05-16T15:11:19Z

Will merge on green. Thanks

openshift-bot · 2016-05-16T15:56:20Z

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/3847/)

smarterclayton · 2016-05-16T15:57:48Z

[merge]

openshift-bot · 2016-05-16T16:00:26Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/3847/) (Image: devenv-rhel7_4209)

openshift-bot · 2016-05-16T16:00:27Z

Evaluated for origin merge up to f92fe67

smarterclayton force-pushed the manage_deploy_logs branch 4 times, most recently from c713dfe to f452554 Compare May 8, 2016 01:29

smarterclayton assigned ironcladlou May 8, 2016

smarterclayton force-pushed the manage_deploy_logs branch 2 times, most recently from 08aeda9 to 0c079db Compare May 8, 2016 16:19

smarterclayton force-pushed the manage_deploy_logs branch from 0c079db to a96fb62 Compare May 9, 2016 16:11

0xmichalis reviewed May 10, 2016
View reviewed changes

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 11, 2016

0xmichalis mentioned this pull request May 11, 2016

Our builders, deployers, and the router need to stop using glog #5788

Closed

smarterclayton force-pushed the manage_deploy_logs branch from a96fb62 to 9fb9ec4 Compare May 11, 2016 19:22

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 11, 2016

smarterclayton force-pushed the manage_deploy_logs branch from 9fb9ec4 to 57e2e50 Compare May 12, 2016 14:05

smarterclayton force-pushed the manage_deploy_logs branch from 00ea045 to 851d7af Compare May 13, 2016 16:05

0xmichalis reviewed May 13, 2016
View reviewed changes

smarterclayton force-pushed the manage_deploy_logs branch from 851d7af to d4ef4ca Compare May 14, 2016 17:37

smarterclayton added 9 commits May 16, 2016 10:58

Update deployment logs and surge on recreate

f705fe8

Add an annotation that skips creating deployer pods

33bcf05

Useful for iterative testing of deployment logic, or running an external deployer.

If desired conditions match, don't go into a wait loop

55919cd

When re-entering deployments, we may not need to even check the scale.

UPSTREAM: 25617: Rolling updater should indicate progress

3fdf3e9

Allows callers to terminate early.

Enable custom deployments to have rolling/recreate as well

4adfd6a

Allow customParams to be specified when type is Recreate or Rolling. Allow image to be empty, add validation for resources and environment. Add a custom deployment extended test.

Updated generated deepcopy/conversion

c6aeef3

Recreate deployment should always run the acceptor

759363d

Even in surge

Print hooks for custom deployments and multiline commands

f92fe67

Shows info about custom deployments that also having rolling update. Makes multiline command printing better. Adds printing tag images hooks.

smarterclayton force-pushed the manage_deploy_logs branch from d4ef4ca to f92fe67 Compare May 16, 2016 14:58

openshift-bot merged commit 00b3e34 into openshift:master May 16, 2016

0xmichalis self-assigned this Dec 15, 2016

0xmichalis mentioned this pull request Dec 15, 2016

Add example of custom deployment and remove dead link openshift/openshift-docs#3260

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update deployment logs and surge on recreate #8787

Update deployment logs and surge on recreate #8787

smarterclayton commented May 7, 2016 •

edited

Loading

smarterclayton commented May 8, 2016

smarterclayton commented May 9, 2016

0xmichalis May 10, 2016

smarterclayton May 10, 2016

0xmichalis commented May 12, 2016

0xmichalis commented May 12, 2016

0xmichalis commented May 12, 2016

smarterclayton commented May 13, 2016

0xmichalis May 13, 2016

smarterclayton May 13, 2016

0xmichalis May 13, 2016

0xmichalis commented May 13, 2016

0xmichalis commented May 13, 2016

smarterclayton commented May 13, 2016

0xmichalis commented May 14, 2016

smarterclayton commented May 14, 2016

smarterclayton commented May 14, 2016

0xmichalis commented May 14, 2016

0xmichalis commented May 14, 2016

smarterclayton commented May 14, 2016

openshift-bot commented May 16, 2016

0xmichalis commented May 16, 2016

smarterclayton commented May 16, 2016

openshift-bot commented May 16, 2016

smarterclayton commented May 16, 2016 via email

openshift-bot commented May 16, 2016 •

edited

Loading

openshift-bot commented May 16, 2016

Update deployment logs and surge on recreate #8787

Update deployment logs and surge on recreate #8787

Conversation

smarterclayton commented May 7, 2016 • edited Loading

smarterclayton commented May 8, 2016

smarterclayton commented May 9, 2016

0xmichalis May 10, 2016

Choose a reason for hiding this comment

smarterclayton May 10, 2016

Choose a reason for hiding this comment

0xmichalis commented May 12, 2016

0xmichalis commented May 12, 2016

0xmichalis commented May 12, 2016

smarterclayton commented May 13, 2016

0xmichalis May 13, 2016

Choose a reason for hiding this comment

smarterclayton May 13, 2016

Choose a reason for hiding this comment

0xmichalis May 13, 2016

Choose a reason for hiding this comment

0xmichalis commented May 13, 2016

0xmichalis commented May 13, 2016

smarterclayton commented May 13, 2016

0xmichalis commented May 14, 2016

smarterclayton commented May 14, 2016

smarterclayton commented May 14, 2016

0xmichalis commented May 14, 2016

0xmichalis commented May 14, 2016

smarterclayton commented May 14, 2016

openshift-bot commented May 16, 2016

0xmichalis commented May 16, 2016

smarterclayton commented May 16, 2016

openshift-bot commented May 16, 2016

smarterclayton commented May 16, 2016 via email

openshift-bot commented May 16, 2016 • edited Loading

openshift-bot commented May 16, 2016

smarterclayton commented May 7, 2016 •

edited

Loading

openshift-bot commented May 16, 2016 •

edited

Loading