Multiple auto-egress-IP fixes #18808

danwinship · 2018-03-02T14:30:50Z

OK, 3 "setup" commits followed by 2 commits with fixes:

the first and second commits only change egressip_test.go, so clearly don't break the actual functioning of auto-egress-IP
the third commit ("Have multiple egress IP ovscontroller methods rather than one confusing one") is small and fairly self-explanatory.
the fourth commit ("Fix egressip handling when a NetNamespace is updated") involves a small fix to egressip.go and then a larger update to egressip_test.go to show that the fix works. Specifically: if the EgressIPs field on a NetNamespace was changed while that egress IP was active, then we did not clean up all of the state associated with the old egress IP, because we were accidentally cleaning up the state associated with the new egress IP instead.
the last commit ("Rearrange egressip internals, add duplication tests") is very large and complicated and needs the most review, but in particular note that all of the existing tests still pass with the new code, and it adds two more tests (which don't pass with the old code). This fixes the problem that we were previously mostly doing the right thing when the user screwed up and added a duplicate EgressIP, but we weren't doing the right thing when they removed the duplicates, because we weren't keeping enough information about the global merged node/namespace EgressIP state. I think the end result is actually simpler than the old code.

Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1551028
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1547899
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1543786
Fixes https://bugzilla.redhat.com/show_bug.cgi?id=1520363

@openshift/sig-networking PTAL

This makes the tests a little bit more self-documenting, and makes it simpler to add new tests. Also, change it to only look at OVS flows in the egress table, so that when there is a mismatch, the error message is of a manageable length.

Split out one test case, add some new (already-passing) test cases, add some comments, and remove some double-checking-of-internal-state that will break with upcoming rewrites.

…ng one

(We handled delete+recreate correctly, but not update.)

knobunc

LGTM

knobunc · 2018-03-02T16:53:16Z

/assign @dcbw

dcbw · 2018-03-02T19:17:21Z

pkg/network/node/egressip.go

-		ns.assignedIP = ""
-		ns.nodeIP = ""
+	if ns.requestedIP != "" {
+		eg := eip.egressIPs[ns.requestedIP]


Are we sure eg isn't going to be nil here?

dcbw · 2018-03-02T19:21:36Z

pkg/network/node/egressip.go

-		if oldNode := eip.nodesByEgressIP[ip]; oldNode != nil {
-			utilruntime.HandleError(fmt.Errorf("Multiple nodes claiming EgressIP %q (nodes %q, %q)", ip, node.nodeIP, oldNode.nodeIP))
-			continue
+		eg := eip.egressIPs[ip]


I almost want:

eg := eip.ensureEgressIPInfo(ip) if eg == nil { return } ... } func (eip *egressIPWatcher) ensureEgressIPInfo(ip string) *egressIPInfo { if ip == "" { return nil } eg := eip.egressIPs[ip] if eg == nil { eg = &egressIPInfo{ip: ip} eip.egressIPs[ip] = eg } return eg }

which you'd also use down below.

dcbw · 2018-03-02T19:23:28Z

pkg/network/node/egressip.go

 		}

-		eip.nodesByEgressIP[ip] = node
-		eip.maybeAddEgressIP(ip)
+		if len(eg.nodes) != 0 {


Also kinda want:

func (eg *egressIPInfo) addNode(node *nodeEgress, ip string) { if len(eg.nodes) != 0 { utilruntime.HandleError(fmt.Errorf("Multiple nodes claiming EgressIP %q (nodes %q, %q)", ip, node.nodeIP, eg.nodes[0].nodeIP)) } eg.nodes = append(eg.nodes, node) } func (eg *egressIPInfo) delNode(node *nodeEgress) { for i := range eg.nodes { if eg.nodes[i] == node { eg.nodes = append(eg.nodes[:i], eg.nodes[i+1:]...) return } } }

and the corresponding functions for namespaces.

dcbw · 2018-03-02T19:29:43Z

pkg/network/node/egressip.go

+	// The egressIPInfo should have an assigned node IP if and only if the
+	// egress IP is active (ie, it is assigned to exactly 1 node and exactly
+	// 1 namespace).
+	egressIPActive := (len(eg.nodes) == 1 && len(eg.namespaces) == 1)


And if you do choose to go with struct egressIPInfo functions, some of the stuff here could be done as functions too. Obviously it can go too far, but maybe a bit cleaner. Your call.

maybe even

changed := eg.syncNodeIP(eip.masqueradeBit, eip.localIP, eip.assignEgressIP, eip.releaseEgressIP) func (eg *egressIPInfo) syncNodeIP(masqBit uint32, localIP string, assign func(string, string) error, func(string, string) error) bool { ... }

danwinship · 2018-03-05T14:11:35Z

I took most of your suggestions. I didn't change syncEgressIP much other than to split it into two subfunctions, one for iptables stuff and one for OVS stuff. The OVS stuff doesn't really split further very well because the blockedVNIDs state has to flow through it so they'd just have to be passed around if we did split it up further

There should never be multiple HostSubnets or multiple NetNamespaces claiming the same egress IP, but if there are, we need to track them carefully so we don't get out sync with reality after things are fixed.

knobunc · 2018-03-05T20:01:20Z

The changes LGTM too. @dcbw?

danwinship · 2018-03-05T20:06:04Z

/retest

dcbw · 2018-03-05T20:47:22Z

/lgtm

openshift-ci-robot · 2018-03-05T20:47:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: danwinship, dcbw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/network/OWNERS~~ [danwinship,dcbw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

danwinship · 2018-03-05T20:52:53Z

/retest

pravisankar

Added some comments, rest of the changes LGTM

pravisankar · 2018-03-05T21:53:49Z

pkg/network/node/egressip.go

 	ns.requestedIP = egressIP
-	eip.namespacesByEgressIP[egressIP] = ns
-	eip.maybeAddEgressIP(egressIP)
+	if egressIP == "" {


I think this is to handle delete case where ns is no longer referenced in eip.namespacesByVNID?
To be clear, may be we could assign ns.requestedIP = egressip after this check

yes, you're right, the ns.requestedIP assignment is irrelevant (but harmless) in this case

pravisankar · 2018-03-05T22:02:27Z

pkg/network/node/egressip.go

@@ -266,44 +233,120 @@ func (eip *egressIPWatcher) updateNamespaceEgress(vnid uint32, egressIP string)

 	ns := eip.namespacesByVNID[vnid]


Need to consider join and isolate project operations otherwise there will be some problems.
Example:

ns1 with vnid v1 requested egrees ip e1

ns2 with vnid v2 requested egress ip e2

ns2 joined with ns1, now ns2 has vnid v1

eip.namespacesByVNID[v1] will now have requestedIP=e2

ns2 got isolated

Now, v1 has only one namespace ns1 but the requested ip will be set as ep2 instead of ep1 which is incorrect.

filed #18852

pravisankar · 2018-03-05T22:15:12Z

pkg/network/node/egressip.go

+
+func (eg *egressIPInfo) addNode(node *nodeEgress) {
+	if len(eg.nodes) != 0 {
+		utilruntime.HandleError(fmt.Errorf("Multiple nodes claiming EgressIP %q (nodes %q, %q)", eg.ip, node.nodeIP, eg.nodes[0].nodeIP))


Can we clarify that only the first node that claimed the egress IP will have egress rules?
(syncEgressIPTablesState() checks eg.nodes[0].nodeIP)

actually none of them will; syncEgressIPTablesState() only looks at eg.nodes[0] when len(eg.nodes)==1

Yes, that's right. Adding a comment like https://github.com/openshift/origin/pull/18808/files#diff-77d23d75ed1c9b57573523db2f13c52aR308 somewhere would be helpful.

pravisankar · 2018-03-05T22:42:02Z

pkg/network/node/egressip.go

+	// 1 namespace).
+	egressIPActive := (len(eg.nodes) == 1 && len(eg.namespaces) == 1)
+	assignedNodeIPChanged := false
+	if egressIPActive && eg.assignedNodeIP != eg.nodes[0].nodeIP {


There could be very rare corner case when F5 is involved.
Scenario:

F5 node n1(no real backing node only hostSubnet object) requested egress ip e1

node n2 requested same egress ip e1

eg.nodes == [n1, n2], n1 claims the egress ip e1

SDN controller calls delete + create on F5 node n1 (removes AssignHostSubnetAnnoation and assigns vnid)

eg.nodes == [n2, n1] now n2 claims the egress ip e1 (side effect of how we handle F5 case)

This is theoretical case but may never happen in practice. I don't think we need to fix this case.

as above, there's no actual corner case here because egressIPActive would be false at both step 3 and step 5

danwinship · 2018-03-05T23:58:03Z

/retest

pravisankar · 2018-03-06T00:37:02Z

pkg/network/node/egressip.go

+
+func (eg *egressIPInfo) addNode(node *nodeEgress) {
+	if len(eg.nodes) != 0 {
+		utilruntime.HandleError(fmt.Errorf("Multiple nodes claiming EgressIP %q (nodes %q, %q)", eg.ip, node.nodeIP, eg.nodes[0].nodeIP))


Yes, that's right. Adding a comment like https://github.com/openshift/origin/pull/18808/files#diff-77d23d75ed1c9b57573523db2f13c52aR308 somewhere would be helpful.

danwinship · 2018-03-06T00:37:48Z

/test end_to_end

danwinship · 2018-03-06T04:09:37Z

/retest

danwinship · 2018-03-06T13:53:52Z

/retest

danwinship · 2018-03-06T15:09:47Z

/retest

danwinship · 2018-03-06T17:37:28Z

/retest

danwinship · 2018-03-06T18:41:32Z

/retest

knobunc · 2018-03-06T19:34:09Z

/retest

knobunc · 2018-03-06T19:52:03Z

Flake was on #17883

Logs: https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18808/test_pull_request_origin_extended_conformance_install/8540/

knobunc · 2018-03-06T20:09:09Z

/hold

knobunc · 2018-03-06T20:12:45Z

/hold cancel

danwinship · 2018-03-07T13:47:39Z

/retest

openshift-merge-robot · 2018-03-08T04:44:40Z

Automatic merge from submit-queue (batch tested with PRs 18780, 18802, 18391, 18832, 18808).

danwinship added 4 commits March 2, 2018 08:59

Add assertOVSChanges helper to egressip_test

6758ab7

This makes the tests a little bit more self-documenting, and makes it simpler to add new tests. Also, change it to only look at OVS flows in the egress table, so that when there is a mismatch, the error message is of a manageable length.

egressip_test updates

bbadc22

Split out one test case, add some new (already-passing) test cases, add some comments, and remove some double-checking-of-internal-state that will break with upcoming rewrites.

Have multiple egress IP ovscontroller methods rather than one confusi…

c854216

…ng one

Fix egressip handling when a NetNamespace is updated

b10a138

(We handled delete+recreate correctly, but not update.)

danwinship added kind/bug Categorizes issue or PR as related to a bug. component/networking sig/networking labels Mar 2, 2018

openshift-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 2, 2018

openshift-ci-robot requested review from rajatchopra and smarterclayton March 2, 2018 14:30

danwinship removed the request for review from smarterclayton March 2, 2018 14:31

danwinship force-pushed the egressip-fixes branch from 25a0ddd to b23627e Compare March 2, 2018 14:50

knobunc approved these changes Mar 2, 2018

View reviewed changes

openshift-ci-robot assigned dcbw Mar 2, 2018

knobunc self-assigned this Mar 2, 2018

dcbw reviewed Mar 2, 2018

View reviewed changes

danwinship force-pushed the egressip-fixes branch from b23627e to 7e5a3c0 Compare March 5, 2018 14:10

Rearrange egressip internals, add duplication tests

a21509b

There should never be multiple HostSubnets or multiple NetNamespaces claiming the same egress IP, but if there are, we need to track them carefully so we don't get out sync with reality after things are fixed.

danwinship force-pushed the egressip-fixes branch from 7e5a3c0 to a21509b Compare March 5, 2018 15:00

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Mar 5, 2018

pravisankar reviewed Mar 5, 2018

View reviewed changes

pravisankar approved these changes Mar 6, 2018

View reviewed changes

danwinship mentioned this pull request Mar 6, 2018

[3.9] Multiple auto-egress-IP fixes #18861

Merged

openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 6, 2018

openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Mar 6, 2018

danwinship mentioned this pull request Mar 6, 2018

Minor fixup to auto egress IP deletion fix #18866

Closed

openshift-merge-robot merged commit 6b99107 into openshift:master Mar 8, 2018

danwinship deleted the egressip-fixes branch July 20, 2018 13:20

		@@ -266,44 +233,120 @@ func (eip *egressIPWatcher) updateNamespaceEgress(vnid uint32, egressIP string)

		ns := eip.namespacesByVNID[vnid]

Multiple auto-egress-IP fixes #18808

Multiple auto-egress-IP fixes #18808

Conversation

danwinship commented Mar 2, 2018 • edited Loading

knobunc left a comment

Choose a reason for hiding this comment

knobunc commented Mar 2, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danwinship commented Mar 5, 2018

knobunc commented Mar 5, 2018

danwinship commented Mar 5, 2018

dcbw commented Mar 5, 2018

openshift-ci-robot commented Mar 5, 2018

danwinship commented Mar 5, 2018

pravisankar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pravisankar Mar 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danwinship commented Mar 5, 2018

Choose a reason for hiding this comment

danwinship commented Mar 6, 2018

danwinship commented Mar 6, 2018

danwinship commented Mar 6, 2018

danwinship commented Mar 6, 2018

danwinship commented Mar 6, 2018

danwinship commented Mar 6, 2018

knobunc commented Mar 6, 2018

knobunc commented Mar 6, 2018

knobunc commented Mar 6, 2018

knobunc commented Mar 6, 2018

danwinship commented Mar 7, 2018

openshift-merge-robot commented Mar 8, 2018

danwinship commented Mar 2, 2018 •

edited

Loading

pravisankar Mar 5, 2018 •

edited

Loading