New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add kube-apiserver wiring #20578

Merged

openshift-merge-robot merged 4 commits into openshift:master from deads2k:server-13-split-kubeapiserver

Aug 23, 2018

Contributor

deads2k commented Aug 8, 2018 •

edited

Loading

@sttts I've come around. This is where my head's at. Not pretty, but I think we can fit most everything into those two spots.

openshift-ci-robot added do-not-merge/work-in-progress approved size/XXL labels

openshift-ci-robot requested review from mfojtik and squeed

August 8, 2018 18:54

Contributor Author

deads2k commented Aug 8, 2018

/retest

Contributor

sttts commented Aug 9, 2018

Commented on the follow-up PR.

sttts reviewed

View reviewed changes

vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go Outdated

               		return nil, err
               	}
+              	if err := PatchKubeAPIServerConfig(kubeAPIServerConfig, sharedInformers, versionedInformers, &pluginInitializer); err != nil {

Contributor

sttts Aug 9, 2018

👍

deads2k force-pushed the server-13-split-kubeapiserver branch from 35dcbe2 to f9b0ce8 Compare

August 9, 2018 17:09

Contributor Author

deads2k commented Aug 9, 2018

@liggitt I plan to kill the limitsecretreferences option. Any objection to just straight out removing it? I don't think anyone uses it.

Contributor

liggitt commented Aug 9, 2018

@liggitt I plan to kill the limitsecretreferences option. Any objection to just straight out removing it? I don't think anyone uses it.

no objection

deads2k force-pushed the server-13-split-kubeapiserver branch 2 times, most recently from 470fdac to db4e9a2 Compare

August 17, 2018 13:28

openshift-ci-robot added size/XL and removed size/XXL labels

deads2k force-pushed the server-13-split-kubeapiserver branch from db4e9a2 to d9dde2e Compare

August 17, 2018 19:57

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/openshiftkubeapiserver/patch_handlerchain.go Outdated

-              func BuildHandlerChain(genericConfig *genericapiserver.Config, kubeInformers informers.SharedInformerFactory, kubeAPIServerConfig *configapi.MasterConfig, stopCh <-chan struct{}) (func(apiHandler http.Handler, kc *genericapiserver.Config) http.Handler, map[string]genericapiserver.PostStartHookFunc, error) {
-              	webconsoleProxyHandler, err := newWebConsoleProxy(genericConfig, kubeInformers, kubeAPIServerConfig)
+              func BuildHandlerChain(genericConfig *genericapiserver.Config, kubeInformers informers.SharedInformerFactory, kubeAPIServerConfig *configapi.MasterConfig) (func(apiHandler http.Handler, kc *genericapiserver.Config) http.Handler, map[string]genericapiserver.PostStartHookFunc, error) {
+              	extraPostStarkHooks := map[string]genericapiserver.PostStartHookFunc{}

Contributor

sttts Aug 20, 2018

typo: start

deads2k mentioned this pull request

switch our last two special cased admission plugins to config #20699

Merged

deads2k force-pushed the server-13-split-kubeapiserver branch from d9dde2e to 107c2d1 Compare

August 20, 2018 18:24

openshift-ci-robot added size/XXL and removed size/XL labels

deads2k force-pushed the server-13-split-kubeapiserver branch 2 times, most recently from 2f21baf to 4480027 Compare

August 21, 2018 17:40

openshift-ci-robot commented Aug 21, 2018

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: deads2k

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [deads2k]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

deads2k force-pushed the server-13-split-kubeapiserver branch from 28b9c41 to 28b885a Compare

August 21, 2018 21:01

deads2k changed the title ~~[WIP] add kube-apiserver wiring~~ add kube-apiserver wiring

Contributor Author

deads2k commented Aug 21, 2018

@sttts this is ready for a real review. We can bring in the old SCC as a later step for cluster up

Contributor Author

deads2k commented Aug 21, 2018

@smarterclayton in 4.0 are we ready to drop SCC in /api/v1? If so (and I think we should), I'd like to drop it from cluster up in 3.11.

deads2k force-pushed the server-13-split-kubeapiserver branch from 28b885a to 1d2c1a7 Compare

August 22, 2018 12:12

openshift-ci-robot added size/XL and removed size/XXL labels

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/openshiftkubeapiserver/flags.go Outdated

+              		return nil, err
+              	}
+              	// these flags are overridden by a patch

Contributor

sttts Aug 22, 2018

who maintains this list? not sure of the value

Contributor Author

deads2k Aug 22, 2018

who maintains this list? not sure of the value

We need this list so that we can create a struct to represent it in order to manage it. We'll need to write a test of some sort to make sure we don't lose them.

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/openshiftkubeapiserver/flags.go

+              	setIfUnset(args, "secure-port", portString)
+              	var keys []string
+              	for key := range args {

Contributor

sttts Aug 22, 2018

keys := append([]string(nil), args...)

Contributor Author

deads2k Aug 22, 2018

keys := append([]string(nil), args...)

args is a map

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/openshiftkubeapiserver/flags.go Outdated

+              				return nil, err
+              			}
+              			// if the config isn't a DefaultAdmissionConfig, then assume we're enabled (we were called after all)
+              			// if the config *is* a DefaultAdmissionConfig and it explicitly said

Contributor

sttts Aug 22, 2018

"it explicitly said" ?

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/openshiftkubeapiserver/flags.go Outdated

+              			} else if defaultConfig.Disable {
+              				forceOff = append(forceOff, pluginName)
+              				continue

Contributor

sttts Aug 22, 2018

no need for the continues if "else" is used.

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/openshiftkubeapiserver/flags.go

+              	tempFile.Close()
+              	setIfUnset(args, "admission-control-config-file", tempFile.Name())
+              	setIfUnset(args, "disable-admission-plugins", forceOff...)

Contributor

sttts Aug 22, 2018

these are additive to the default values. Maybe we should append?

Contributor Author

deads2k Aug 22, 2018

these are additive to the default values. Maybe we should append?

We've always stomped admission with our choices. I only want one way to set via config.

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/openshiftkubeapiserver/patch.go Outdated

@@ @@ -1,23 +1,27 @@ @@
               package openshiftkubeapiserver
               import (
+              	"github.com/openshift/origin/pkg/admission/namespaceconditions"

Contributor

sttts Aug 22, 2018

Goland still hasn't learned our import order.

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/openshiftkubeapiserver/patch.go

               		if err != nil {
               			return nil, err
               		}
               		clusterQuotaMappingController := openshiftapiserver.NewClusterQuotaMappingController(kubeAPIServerInformers.InternalKubernetesInformers.Core().InternalVersion().Namespaces(), kubeAPIServerInformers.InternalOpenshiftQuotaInformers.Quota().InternalVersion().ClusterResourceQuotas())
-              		kubeClient, err := kubernetes.NewForConfig(config.GenericConfig.LoopbackClientConfig)
+              		patchContext.postStartHooks["quota.openshift.io-clusterquotamapping"] = func(context genericapiserver.PostStartHookContext) error {
+              			go clusterQuotaMappingController.Run(5, context.StopCh)

Contributor

sttts Aug 22, 2018

any deeper reason why this is now a post start hook and not a plugin initializer?

Contributor Author

deads2k Aug 22, 2018

any deeper reason why this is now a post start hook and not a plugin initializer?

There's a controller piece that neesd to run

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/openshiftkubeapiserver/patch.go

               		}
               		return nil
               	})
-              	server.GenericAPIServer.AddPostStartHookOrDie("openshift.io-restmapperupdater", func(context genericapiserver.PostStartHookContext) error {

Contributor

sttts Aug 22, 2018

where is this gone? Why don't we need that anymore?

Contributor Author

deads2k Aug 22, 2018

where is this gone? Why don't we need that anymore?

we use the upstream restmapper now.

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/openshiftkubeapiserver/patch_handlerchain.go Outdated

               	return func(apiHandler http.Handler, genericConfig *genericapiserver.Config) http.Handler {
               			// Machinery that let's use discover the Web Console Public URL
               			accessor := newWebConsolePublicURLAccessor(genericConfig.LoopbackClientConfig)
-              			go accessor.Run(stopCh)
+              			extraPostStartHooks["openshift.io-webconsolepublicurl"] = func(context genericapiserver.PostStartHookContext) error {

Contributor

sttts Aug 22, 2018 •

edited

Loading

this is (and was) dirty and deserves a big, bold explanation why we do this here.

Contributor Author

deads2k Aug 22, 2018

this is (and was) dirty and deserves a big, bold explanation why we do this here.

No worse than before, but I can comment it.

Contributor Author

deads2k commented Aug 22, 2018

comments addressed in a separate commit.

sttts reviewed

View reviewed changes

pkg/cmd/openshift-kube-apiserver/server.go

+              	bootstrappolicy.ClusterRoles = bootstrappolicy.OpenshiftClusterRoles
+              	bootstrappolicy.ClusterRoleBindings = bootstrappolicy.OpenshiftClusterRoleBindings
+              	options.AllOrderedPlugins = originadmission.CombinedAdmissionControlPlugins

Contributor

sttts Aug 22, 2018

ugly to override global slice vars meant as constants. But I guess we have no choice.

Contributor

sttts Aug 22, 2018

a carry patch might be more explicit.

Contributor Author

deads2k Aug 22, 2018

a carry patch might be more explicit.

doing so doesn't allow running in both modes and exposes us to greater conflicts

sttts reviewed

View reviewed changes

vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/aggregator.go Outdated

@@ @@ -78,7 +79,8 @@ func createAggregatorConfig( @@
               	// copy the etcd options so we don't mutate originals.
               	etcdOptions := *commandOptions.Etcd
               	etcdOptions.StorageConfig.Codec = aggregatorscheme.Codecs.LegacyCodec(v1beta1.SchemeGroupVersion, v1.SchemeGroupVersion)
-              	genericConfig.RESTOptionsGetter = &genericoptions.SimpleRestOptionsFactory{Options: etcdOptions}
+              	aggregatorinstall.Install(legacyscheme.Scheme)
+              	//genericConfig.RESTOptionsGetter = &genericoptions.SimpleRestOptionsFactory{Options: etcdOptions}

Contributor

sttts Aug 22, 2018

why? This needs a comment

Contributor Author

deads2k Aug 22, 2018

why? This needs a comment

added

sttts reviewed

View reviewed changes

vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/apiextensions.go Outdated

@@ @@ -30,6 +30,10 @@ import ( @@
               	"k8s.io/kubernetes/cmd/kube-apiserver/app/options"
               )
+              func createAPIExtensionsServer(apiextensionsConfig *apiextensionsapiserver.Config, delegateAPIServer genericapiserver.DelegationTarget) (*apiextensionsapiserver.CustomResourceDefinitions, error) {
+              	return apiextensionsConfig.Complete().New(delegateAPIServer)

Contributor

sttts Aug 22, 2018

If I trust my eyes, this is only an (unnecessary) move?

Contributor Author

deads2k Aug 22, 2018

If I trust my eyes, this is only an (unnecessary) move?

fixed

sttts reviewed

View reviewed changes

vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go Outdated

               		return nil, err
               	}
+              	if err := PatchKubeAPIServerServer(kubeAPIServer); err != nil {
+              	}

Contributor

sttts Aug 22, 2018

drop if or handle error

Contributor Author

deads2k Aug 22, 2018

drop if or handle error

fixed

sttts reviewed

View reviewed changes

vendor/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go

               		return nil, fmt.Errorf("error in initializing storage factory: %s", err)
               	}
+              	storageFactory.SetResourceEtcdPrefix(schema.GroupResource{Group: "apiregistration.k8s.io", Resource: "apiservices"}, "apiservices")

Contributor

sttts Aug 22, 2018

what's the history of this? Did we ship an earlier version with a wrong etcd path?

Contributor Author

deads2k Aug 22, 2018

what's the history of this? Did we ship an earlier version with a wrong etcd path?

Yes, in 3.6

deads2k added 4 commits

August 22, 2018 09:58


          UPSTREAM: <carry>: patch in a non-standard location for apiservices


          UPSTREAM: <carry>: allow injection of kube-apiserver options

33a71af


          generated

d323ddd


          wire hypershift openshift-kube-apiserver to use minimal patching stra…

5e13f18

…tegy

deads2k force-pushed the server-13-split-kubeapiserver branch from f10076b to 5e13f18 Compare

August 22, 2018 13:58

Contributor

sttts commented Aug 22, 2018

Still quite some ugly hacks. But it's a good step into the right direction 👍

deads2k added the lgtm label

Contributor Author

deads2k commented Aug 22, 2018

/retest

1 similar comment

Contributor Author

deads2k commented Aug 22, 2018

/retest

openshift-merge-robot merged commit 7840b4d into openshift:master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm size/XL