Handle incorrect allocations in the system

[asm582]

• At scale, the InstaSlice controller is not able to handle allocations for short tasks as it relies on sub-resource status, which takes a while to update, causing different pods to use the same allocation in the system while servicing short tasks. This PR introduces a cache in the system that brings determinism into the system while servicing pods with 2-second tasks. E2E's pass in emulator and non-emulator mode, and existing unit tests pass.
• PR also retries InstaSlice status updates in scenarios when configmap is not present but the CR needs an update

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: asm582

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [asm582]
• internal/controller/daemonset/OWNERS [asm582]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[asm582]

/hold until we pass e2e on Openshift

[openshift-ci]

@asm582: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[harche]

/hold until we pass e2e on Openshift

Thanks, would it be okay to wait till next week for OCP related testing? MoC cluster is busy in some DRA related tests.

[asm582]

/hold until we pass e2e on Openshift

Thanks, would it be okay to wait till next week for OCP related testing? MoC cluster is busy in some DRA related tests.

Sure