Data race in grpc lib used by etcd client

[sttts]

Seen in https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18444/test_pull_request_origin_unit/9098/.

WARNING: DATA RACE
Read at 0x00c4200a11e1 by goroutine 124:
  github.com/openshift/origin/vendor/google.golang.org/grpc/transport.(*Stream).BytesReceived()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/google.golang.org/grpc/transport/transport.go:359 +0x8a
  github.com/openshift/origin/vendor/google.golang.org/grpc.invoke()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/google.golang.org/grpc/call.go:284 +0x146d
  github.com/openshift/origin/vendor/google.golang.org/grpc.Invoke()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/google.golang.org/grpc/call.go:143 +0x216
  github.com/openshift/origin/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.(*kVClient).Range()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.go:2239 +0x106
  github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*retryWriteKVClient).Range()
      <autogenerated>:1 +0xad
  github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*retryKVClient).Range.func1()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/retry.go:92 +0xfc
  github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*Client).newAuthRetryWrapper.func1()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/retry.go:61 +0x5e
  github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*retryKVClient).Range()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/retry.go:91 +0x221
  github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*kv).do()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/kv.go:140 +0x63d
  github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*kv).Do()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/kv.go:119 +0x9f
  github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3.(*kv).Get()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/github.com/coreos/etcd/clientv3/kv.go:93 +0xf2
  github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watchChan).sync()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:177 +0x16e
  github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watchChan).startWatching()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:193 +0xa49

Previous write at 0x00c4200a11e1 by goroutine 98:
  github.com/openshift/origin/vendor/google.golang.org/grpc/transport.(*http2Client).operateHeaders()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/google.golang.org/grpc/transport/http2_client.go:963 +0x93
  github.com/openshift/origin/vendor/google.golang.org/grpc/transport.(*http2Client).reader()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/google.golang.org/grpc/transport/http2_client.go:1080 +0x832

Goroutine 124 (running) created at:
  github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watchChan).run()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:129 +0x8f

Goroutine 98 (running) created at:
  github.com/openshift/origin/vendor/google.golang.org/grpc/transport.newHTTP2Client()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/google.golang.org/grpc/transport/http2_client.go:250 +0x1268
  github.com/openshift/origin/vendor/google.golang.org/grpc/transport.NewClientTransport()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/google.golang.org/grpc/transport/transport.go:439 +0xb1
  github.com/openshift/origin/vendor/google.golang.org/grpc.(*addrConn).resetTransport()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/google.golang.org/grpc/clientconn.go:829 +0x43a
  github.com/openshift/origin/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn.func1()
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/google.golang.org/grpc/clientconn.go:612 +0x41

[sttts]

/cc @liggitt @deads2k @mfojtik

Have never seen this before and couldn't reproduce in upstream master either.

[liggitt]

see kubernetes/kubernetes#54863 and kubernetes/kubernetes#53124 ?

[sttts]

Fixed by kubernetes/kubernetes#57480

[sttts]

Have bumped grpc alone, but etcd compilation breaks. So we have to bump etcd, grpc and probably grpc-gateway all at once. Do we want that? Kube 1.9.2 is also on an older version (v3.1.11), a bit younger though than ours.