OCPNODE-2882: Handle unexpected node reboots #421
Conversation
Signed-off-by: Sai Ramesh Vanka <[email protected]>
openshift-ci
bot
added
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
openshift-ci
bot
added
the
size/S
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: sairameshv The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
@sairameshv not sure if I got this correctly, but reading this comment #336 (comment) I get the impression that if we inject the code in deamonset to delete all allocations during the deamonset bootstrap, would it fix the issue? |
|
by that I meant, what happens if you fetch the corresponding instaslice object and delete all allocations somewhere here, https://github.com/openshift/instaslice-operator/blob/main/internal/controller/daemonset/instaslice_daemonset.go#L98 ? |
If we delete all the previous allocations when a daemonset comes up after a reboot, can we gate the pods again and create the slices ? Any thoughts @asm582 ? |
good point, all partitions will be deleted by the hardware after reboot. I think if an operator or deployment is used to spawn pods, then we will always have new ungated pods. we need to come up with a solution for plain pods. |
sairameshv
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@sairameshv: This pull request references OCPNODE-2882 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
| return ctrl.Result{RequeueAfter: controller.Requeue1sDelay}, nil | ||
| } | ||
| for _, cond := range node.Status.Conditions { | ||
| if cond.Type == v1.NodeReady && cond.Status == v1.ConditionFalse { |
There was a problem hiding this comment.
A node can go NotReady and not be rebooted... We might have to look at the bootid changing at startup.
There was a problem hiding this comment.
I know bootId can be fetched from the Node Status. How can we determine if it is changed? Do we have to persist boot ids somewhere?
There was a problem hiding this comment.
The status on the Instaslice object might need to persist it.
|
Closing this PR in favor of #433 |
Potentially
Fixes #336
make test,make lint, pass locallymake test-e2e-kind-emulatedpasses with some tweaks to the config files./hold