Discuss daemonset state removal

[asm582]

In the current system, the daemon set sets states in instance allocation to created (when mig slice is created on GPU with configmap added to pod namespaces) and deleted (when mig slice is deleted from the GPU with configmap deleted from the namespace).
The controller can listen to config map events and move allocation to ungated when a config map create event occurs or remove allocation when a config map delete event occurs. This approach reduces the ping-pong for changing allocation status by two controllers for a pod.

[harche]

The events in k8s are not guaranteed to arrive in sequence. So in a rapidly starting and stopping, the deleted event may arrive before created one.

[sairameshv]

/assign

[sairameshv]

Here is the web sequence diagram of the existing Stateful Daemonset behavior

Following is the proposal/design to avoid updating the allocation status of the Instaslice object by the Daemonset

@asm582 , Let me know your thoughts on the new workflow

[asm582]

Thanks, @sairameshv. Excellent summary; yes, this is what we want to achieve. We should continue to follow the design of creating before deleting. @harche mentioned out-of-order event triggers like config map deletes events, that could occur before creating, which we need to handle in the controller logic.