Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug 1508061: Fix panic during openshift controller options initialization #17127

Merged
merged 1 commit into from
Nov 1, 2017
Merged

Bug 1508061: Fix panic during openshift controller options initialization #17127

merged 1 commit into from
Nov 1, 2017

Conversation

mfojtik
Copy link
Contributor

@mfojtik mfojtik commented Nov 1, 2017 •
edited
Loading

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1508061

Basically newKubeControllerManager mutates the cmdLineArgs in parallel to getOpenshiftControllerOptions which is trying to read it.

@deads2k @sttts PTAL, i consider this 3.7 blocker.

@mfojtik mfojtik added kind/bug Categorizes issue or PR as related to a bug. kind/delivery-blocker labels Nov 1, 2017
@mfojtik mfojtik added this to the 3.7.0 milestone Nov 1, 2017
@openshift-ci-robot openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 1, 2017
@openshift-merge-robot openshift-merge-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 1, 2017
@mfojtik mfojtik force-pushed the fix-initialization-panic branch from a29cfa5 to dfbd95f Compare November 1, 2017 09:33
@mfojtik mfojtik requested a review from deads2k November 1, 2017 09:39
@0xmichalis 0xmichalis removed their assignment Nov 1, 2017
sttts
sttts reviewed Nov 1, 2017
View reviewed changes
pkg/cmd/server/start/start_master.go Outdated
// Wait for the runEmbeddedKubeControllerManager to mutate the
// ControllerArguments to prevent getOpenshiftControllerOptions from
// panicking because of concurrent read/write access.
<-controllerArgumentsInitialized
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this smells. Do the openshift controller really have to see the mutated flags? Or was this only an accident?

@mfojtik
Copy link
Contributor Author

mfojtik commented Nov 1, 2017

Alternative without chan: mfojtik@4ce220b

@mfojtik mfojtik force-pushed the fix-initialization-panic branch from dfbd95f to 3b3a5d6 Compare November 1, 2017 10:14
@openshift-ci-robot openshift-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 1, 2017
@mfojtik
Copy link
Contributor Author

mfojtik commented Nov 1, 2017

@sttts updated.

@sttts
Copy link
Contributor

sttts commented Nov 1, 2017

I pretty much prefer the deep-copy variant over the channel and the setup...-func one. This really looked as an accident to mutate an incoming param.

@sttts
Copy link
Contributor

sttts commented Nov 1, 2017

Waiting for tests....

@mfojtik
Copy link
Contributor Author

mfojtik commented Nov 1, 2017

/retest

@mfojtik mfojtik force-pushed the fix-initialization-panic branch from 3b3a5d6 to 36e4f29 Compare November 1, 2017 12:24
@openshift-ci-robot openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 1, 2017
deads2k
deads2k reviewed Nov 1, 2017
View reviewed changes
pkg/cmd/server/start/start_kube_controller_manager.go Outdated
for _, item := range v {
newVal = append(newVal, item)
}
cmdLineArgs[k] = newVal
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: append([]string{}, v...)

@mfojtik mfojtik force-pushed the fix-initialization-panic branch from 36e4f29 to 8c7ba88 Compare November 1, 2017 12:26
@openshift-ci-robot openshift-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 1, 2017
@deads2k
Copy link
Contributor

deads2k commented Nov 1, 2017

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 1, 2017
@simo5
Copy link
Contributor

simo5 commented Nov 1, 2017

/lgtm

liggitt
liggitt reviewed Nov 1, 2017
View reviewed changes
pkg/cmd/server/start/start_kube_controller_manager.go
func newKubeControllerManager(kubeconfigFile, saPrivateKeyFile, saRootCAFile, podEvictionTimeout, recyclerImage string, dynamicProvisioningEnabled bool, cmdLineArgs map[string][]string) (*controlleroptions.CMServer, []func(), error) {
if cmdLineArgs == nil {
cmdLineArgs = map[string][]string{}
func newKubeControllerManager(kubeconfigFile, saPrivateKeyFile, saRootCAFile, podEvictionTimeout, recyclerImage string, dynamicProvisioningEnabled bool, controllerArgs map[string][]string) (*controlleroptions.CMServer, []func(), error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same issue exists in newScheduler

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liggitt i don't think getOpenshiftControllerOptions iterate over m.config.KubernetesMasterConfig.SchedulerArguments so we are probably safe there but yeah, we should deep copy to be sure there as well. I will update this PR

@mfojtik mfojtik force-pushed the fix-initialization-panic branch from 8c7ba88 to 9b2f23d Compare November 1, 2017 14:51
@openshift-ci-robot openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 1, 2017
@openshift-merge-robot openshift-merge-robot removed the lgtm Indicates that a PR is ready to be merged. label Nov 1, 2017
@liggitt
Copy link
Contributor

liggitt commented Nov 1, 2017

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 1, 2017
@openshift-merge-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k, liggitt, mfojtik, simo5

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@mfojtik
Copy link
Contributor Author

mfojtik commented Nov 1, 2017

/retest

2 similar comments
@eparis
Copy link
Member

eparis commented Nov 1, 2017

/retest

@eparis
Copy link
Member

eparis commented Nov 1, 2017

/retest

@openshift-merge-robot
Copy link
Contributor

Automatic merge from submit-queue.

@openshift-merge-robot openshift-merge-robot merged commit 7968b96 into openshift:master Nov 1, 2017
@mfojtik mfojtik deleted the fix-initialization-panic branch September 5, 2018 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sttts sttts sttts left review comments

@liggitt liggitt liggitt left review comments

@deads2k deads2k deads2k left review comments

Assignees

@liggitt liggitt

@deads2k deads2k

@simo5 simo5

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. kind/delivery-blocker lgtm Indicates that a PR is ready to be merged. queue/critical-fix size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
v3.7
Development

Successfully merging this pull request may close these issues.
9 participants
@mfojtik @sttts @deads2k @simo5 @liggitt @openshift-merge-robot @eparis @0xmichalis @openshift-ci-robot