Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sa: make docker registry service controller check all secrets #19788

Conversation

mfojtik
Copy link
Contributor

@mfojtik mfojtik commented May 21, 2018

This change will make the docker registry service controller check all secrets regardless
whether the docker registry service location changed or not. This should effectively fix
all secrets in the cluster that are missing the registry DNS in the dockercfg content.

The drawback here is that the controller will now need to decode all dockercfg secrets
to perform the check, which might increase the load when the controller starts.

However, the controller resync is zero which means it won't run very often, so I think the CPU time here is worth to spend to make sure the dockercfg content in the secret is correct ;-)

Should fix https://bugzilla.redhat.com/show_bug.cgi?id=1578195

/cc @soltysh
/cc @deads2k

@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mfojtik

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 21, 2018
@@ -241,18 +241,14 @@ func getDockerRegistryLocations(lister listers.ServiceLister, location serviceLo
func (e *DockerRegistryServiceController) syncRegistryLocationChange(key string) error {
newLocations := e.getDockerRegistryLocations()
newDockerRegistryLocations := sets.NewString(newLocations...)
existingURLs := e.getRegistryURLs()
if existingURLs.Equal(newDockerRegistryLocations) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can limit the full scan to the "empty to something" transition, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you mean? this is just checking if the docker-registry service moved I believe (the ClusterIP or name?)... we will still have to go secret by secret to decode the dockercfg and determine whether we have all endpoints we want there...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you mean? this is just checking if the docker-registry service moved I believe (the ClusterIP or name?)... we will still have to go secret by secret to decode the dockercfg and determine whether we have all endpoints we want there...

The link you linked me to before. Like this: #19791

@mfojtik
Copy link
Contributor Author

mfojtik commented May 21, 2018

@deads2k need more thoughts to make the unit test pass... it seems like when you remove the registry service the secret is still present where it should be cleared up, will keep digging tomorrow.

@mfojtik mfojtik force-pushed the sa-10-fix-docker-registry-secrets branch from b94c3bd to a0255f5 Compare May 22, 2018 11:19
@openshift-ci-robot openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 22, 2018
@mfojtik
Copy link
Contributor Author

mfojtik commented May 22, 2018

@deads2k i picked yours (#19791) but after a while I realized that your change is making the unit tests somehow racy...

Alternative approach... Lets not hide the intent to run an initial sweep of secrets when the controller starts and have a special variable to indicate that instead of using the registry urls vars (which I guess is also used by other controllers...). This way we don't need to change the unit tests, actually I can write a new one to exercise the initial sweep. WDYT?

}

t.Logf("deleting %s service", registryService.Name)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will keep these as they proved to be helpful when debugging

@mfojtik mfojtik force-pushed the sa-10-fix-docker-registry-secrets branch from a0255f5 to bba8265 Compare May 22, 2018 11:28
Copy link
Contributor

@soltysh soltysh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, I'll let David have final saying

@mfojtik mfojtik force-pushed the sa-10-fix-docker-registry-secrets branch from bba8265 to d7738c7 Compare May 22, 2018 11:36
@mfojtik
Copy link
Contributor Author

mfojtik commented May 22, 2018

/retest

@mfojtik mfojtik force-pushed the sa-10-fix-docker-registry-secrets branch 2 times, most recently from 010c619 to 590bb9a Compare May 22, 2018 12:41
@mfojtik
Copy link
Contributor Author

mfojtik commented May 22, 2018

/retest

1 similar comment
@mfojtik
Copy link
Contributor Author

mfojtik commented May 22, 2018

/retest

newLocations := e.getDockerRegistryLocations()
newDockerRegistryLocations := sets.NewString(newLocations...)
existingURLs := e.getRegistryURLs()
if existingURLs.Equal(newDockerRegistryLocations) {
glog.V(4).Infof("No effective update: %v", newDockerRegistryLocations)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: seems like you may want to keep this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed, good catch

@@ -229,8 +254,6 @@ func TestUpdateOldStyleSecretWithKey(t *testing.T) {
t.Fatalf("failed to become ready")
}

fakeWatch.Modify(registryService)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was because the test previous had to kick something and doesn't need to anymore?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, proving the initial secret check should work regardless of the registry change

@deads2k
Copy link
Contributor

deads2k commented May 22, 2018

optional nit, one question, lgtm. Feel free to tag with or without the nit.

@mfojtik mfojtik force-pushed the sa-10-fix-docker-registry-secrets branch from 590bb9a to 99e40f3 Compare May 23, 2018 08:10
@mfojtik mfojtik added the lgtm Indicates that a PR is ready to be merged. label May 23, 2018
@openshift-merge-robot openshift-merge-robot merged commit 1a293b3 into openshift:master May 23, 2018
@mfojtik mfojtik deleted the sa-10-fix-docker-registry-secrets branch September 5, 2018 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants