deploy: remove deployer pods on cancellation

[0xmichalis]

Fixes #8439
Fixes #8500
Fixes #9289

@smarterclayton @ironcladlou @mfojtik PTAL

[0xmichalis]

Opened #9296 which is related

[smarterclayton]

[test]

[0xmichalis]

• Failure [24.149 seconds]
deploymentconfigs
/data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:309
  when run iteratively
  /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:131
    should only deploy the last deployment [Conformance] [It]
    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:130

    Expected error:
        <*errors.errorString | 0xc8201c1630>: {
            s: "found multiple running deployments: [deployment-simple-1 deployment-simple-2 deployment-simple-3 deployment-simple-4]",
        }
        found multiple running deployments: [deployment-simple-1 deployment-simple-2 deployment-simple-3 deployment-simple-4]
    not to have occurred

    /data/src/github.com/openshift/origin/test/extended/deployments/deployments.go:119

The failure is real but I think it has to do with the way the deployer pods are listed in one shot in the extended utilities for the deployments. Looking...

[0xmichalis]

Worth noting that I don't see anywhere in the output the defered function that dumps the dc and its rcs and deployer pods.

[0xmichalis]

Ok, so we cannot just transition the deployment to Failed when we delete the deployers because the deployment config controller will notice it and create a new deployment maybe before the previous deployer pod is deleted. We can transition to Failed once we make sure the deployer pod does not exist. I think we will need a pod cache to requeue on DELETE events (we will need to requeue on every event for #9296 anyway).

[0xmichalis]

Green on the extended test, broken unit test

[smarterclayton]

v(4) probably.

[smarterclayton]

Changes look ok

[0xmichalis]

yum flake, repushed to fix the glog level

[openshift-bot]

Evaluated for origin test up to 7a7770e

[smarterclayton]

Lgtm [merge]

[openshift-bot]

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/4829/) (Image: devenv-rhel7_4371)

[openshift-bot]

Evaluated for origin merge up to 7a7770e

[openshift-bot]

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/4829/)