Daemonset fault tolerance

[asm582]

This PR covers the following scenarios:

• The daemonset should continue to work after the pod restart(s), clean up dangling CIs and GIs, and delete CI and GIs when the user deletes a bunch of submitted pods.
• The PR now consumes node selectors to point pods to nodes
• The PR patches the node(s) with org.instaslice/

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asm582

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [asm582]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[asm582]

All test cases pass locally and 80 pod submissions with sleep 5 command completes in about 260 seconds.

[openshift-ci]

@asm582: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[harche]

/cc @rphillips for the next round of review.

[rphillips]

/lgtm