flake: Test_Run_OneVolumeAttachAndDetachMultipleNodesWithReadWriteOnce

[jim-minter]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/16808/test_pull_request_origin_unit/4224/

=== RUN   Test_Run_OneVolumeAttachAndDetachMultipleNodesWithReadWriteOnce
--- FAIL: Test_Run_OneVolumeAttachAndDetachMultipleNodesWithReadWriteOnce (0.01s)
	reconciler_test.go:520: Warning: Wrong NewAttacherCallCount. Expected: <1> Actual: <0>. Will retry.
	reconciler_test.go:472: Expected volume "fake-plugin/volume-name" on node "node-name2" to have multiattach error
FAIL

That PR already includes #16791.

[gnufied]

This is unrelated to fix in #16791.

There is a upstream flake fix we are missing - but it probably will not fix this flake kubernetes/kubernetes#48757 . But because upstream fix(48757) introduces additional wait before checking for MultiAttachError it minimizes chances of this flake happening.

The underlying problem is, when we check for MultiAttachError there is no guarantee that second node's volume has been processed by reconciler loop. Although this is a very tight loop with no sleep in between but it appears to be still occuring. Thinking about a fix.

[gnufied]

I ran this in a tight loop for 30 minutes and I couldn't reproduce it. But I do still agree that this flake can happen.

╰─openshift/origin> while true; do date; go test -race ./vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler; done
Thu Oct 12 13:19:52 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.737s
Thu Oct 12 13:20:12 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.736s
Thu Oct 12 13:20:31 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.736s
Thu Oct 12 13:20:51 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.735s
Thu Oct 12 13:21:10 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.737s
Thu Oct 12 13:21:29 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.736s
Thu Oct 12 13:21:49 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.736s
Thu Oct 12 13:22:08 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.735s
Thu Oct 12 13:22:27 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.736s
Thu Oct 12 13:22:47 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.738s
Thu Oct 12 13:23:06 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.736s
Thu Oct 12 13:23:26 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.736s
Thu Oct 12 13:23:45 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.736s
Thu Oct 12 13:24:04 EDT 2017
ok      github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/controller/volume/attachdetach/reconciler      1.736s