Allow nodes to bootstrap and add `oc cluster join` for nodes #9547

smarterclayton · 2016-06-24T14:58:12Z

Leverage the alpha CSR endpoint in Kube to allow nodes to bootstrap themselves. On startup, the node now accepts --bootstrap which instructs it to perform the following steps:

Connect to the master using --kubeconfig and attempt to create a CSR for a client cert on system:nodes, with the common name system:node:NAME
Given a successful client cert, use that cert to request a serving cert for the node hostnames
Given that, attempt to download the ConfigMap node-config in namespace openshift-infra if it exists
Create a default node-config.yaml if none was download (if the config map does not exist)
Merge any command line flags with the config file
Start

The server creates a new infra service account on startup - node-bootstrap in openshift-infra. This account is granted the ability to create client CSRs (and nothing else). The CSR controller is initialized (if no defaults are set) to allow the system:service accounts:openshift-infra group to request client CSRs and the system:nodes group to request server CSRs. The controller can be disabled by setting an invalid or unused group.

Add a new command oc serviceaccounts create-kubeconfig which generates a kubeconfig file for a service account (to make it easy to generate a kubeconfig that can be used to bootstrap the node).

Add a new command oc cluster join which launches a container that acts as a node.

smarterclayton · 2016-11-20T01:34:59Z

@liggitt @deads2k this is ready for a once over.

smarterclayton · 2016-11-20T02:10:23Z

Created a node-bootstrap.kubeconfig on creation that is a client cert for the account. This simplifies small clusters (because you can just use that file as your bootstrap). However, it's not the ideal outcome, and most people should use oc sa create-kubeconfig -n openshift-infra node-bootstrap > as their node-bootstrap.kubeconfig.

smarterclayton · 2016-11-20T02:10:29Z

[test]

smarterclayton · 2016-11-20T02:11:57Z

@dgoodwin this is similar to the upstream work, but leverages our existing crypto infra and is capable of starting existing nodes. I think future work may converge, but will be some difference for a while.

dgoodwin

Just some initial comments from trying to test this out.

dgoodwin · 2016-11-22T12:27:22Z

docs/man/man1/oc-cluster-join.1

+Add a new node to an existing OpenShift cluster
+
+.PP
+Uses an existing connection to a Docker daemon to start an OpenShift node. You must provide a secret to connect to the master. Before running command, ensure that you can execute docker commands successfully (ie. 'docker ps').


Could we get more specific on what that secret is and how you obtain it? I am assuming it's kubeconfig output from create-kubeconfig but per below I don't seem to have a node-bootstrap SA.

It might be very useful to spell out exactly how to do this in the output from oc cluster up, or in the "by example" docs somewhere.

Do you expect that the secret might one day also allow you to use the discovery bits likely landing in 1.6? We would need to plan for how you find an API server if so.

The secret is a kubeconfig file, that has the ability to request CSR. Expectation here is that your file contains discovery info (we're not focused on self-hosting in OpenShift until it's had time to bake in Kube, so the discovery bits aren't really necessary).

dgoodwin · 2016-11-22T12:28:46Z

pkg/bootstrap/docker/join.go

+`)
+)
+
+// NewCmdJoin creates a command that joins an existing OpenShift cluster.


FWIW we found kubeadm reset pretty handy for both devs and users, oc cluster leave might be a nice addition at some point to just clear everything out and allow user to try again.

dgoodwin · 2016-11-22T12:59:00Z

pkg/cmd/server/admin/default_certs.go

+			KeyFile:  path.Join(certDir, "node-bootstrap.key"),
+		},
+		UnqualifiedUser: "node-bootstrap",
+		User:            "system:serviceaccount:openshift-infra:node-bootstrap",


This is supposed to be created by default after oc cluster up? (this did not seem to happen for me) Anything I might be missing that needs to be done to get it created?

This is created automatically on a new cluster - you would need to rebuild the image if you haven't.

dgoodwin · 2016-11-22T13:33:15Z

pkg/cmd/server/bootstrappolicy/infra_sa_policy.go

+	)
+	if err != nil {
+		panic(err)
+	}


To above question, this looks like it should be creating a node-bootstrap SA, I do see other SA's defined here, but not this one.

dgoodwin · 2016-11-22T13:46:48Z

Where does this fit in larger scope of things, is oc cluster up/join becoming less focused on dev experimentation and headed for something supported?

deads2k · 2016-11-22T16:02:05Z

vendor/k8s.io/kubernetes/pkg/controller/certificates/group_approver.go

+			signClient = true
+			break
+		}
+		if len(a.approveKubeletServingCSRsForGroup) > 0 && g == a.approveKubeletServingCSRsForGroup {


If you sign serving certificates without confirming that my identity actually owns the hostname that you're signing for, then I can exploit the API to gain a serving cert for another node which in combination with an owned node network will allow me to steal any used secret in the system even if we close the API authorizer gaps.

I haven't dug into the pull, but if you allowed me to get signed cert for the address of the master, I could also steal any node credentials I wanted.

smarterclayton · 2016-11-28T22:17:19Z

Where does this fit in larger scope of things, is oc cluster up/join becoming less focused on dev experimentation and headed for something supported?

No, it would still be focused on the simplest possible workflow that can lead a cluster.

liggitt · 2017-01-23T05:24:55Z

pkg/cmd/server/start/bootstrap_node.go

+			Name: fmt.Sprintf("node-bootstrapper-server-%s", safeSecretName(o.NodeArgs.NodeName)),
+		},
+		Spec: certificates.CertificateSigningRequestSpec{
+			Request: csrData,


TODO to limit to server usage in 1.6

liggitt · 2017-01-23T05:26:01Z

pkg/cmd/server/start/bootstrap_node.go

+		if err != nil {
+			return err
+		}
+		if err := o.NodeArgs.MergeSerializeableNodeConfig(nodeConfig); err != nil {


have you sanity-checked if there's any difference between this and start node --write-config?

yes, but will do so again.

smarterclayton · 2017-01-24T02:27:42Z

#11772 [test]

…

On Mon, Jan 23, 2017 at 9:13 PM, OpenShift Bot ***@***.***> wrote: continuous-integration/openshift-jenkins/test FAILURE ( https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/13212/) (Base Commit: fac9b78 <fac9b78> ) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#9547 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p-1_mOMZ6AGZwTCFPSalVHc8spWfks5rVV4_gaJpZM4I92KL> .

liggitt · 2017-01-24T02:32:37Z

things we'll want to fix before this is no longer marked experimental:

make the cli work with a version of the certificates api other than v1alpha1
switch the auto-sign to gate on a permission check, not a hard-coded supergroup (I plan to do this upstream shortly)
reduce magic filenames in bootstrap file

liggitt · 2017-01-24T02:33:03Z

make sure it's clearly marked experimental, make sure there are follow-ups for the items above. no other comments.

smarterclayton · 2017-01-24T02:34:50Z

My execution plan for this is:

merge
extend openshift-ansible to have a role that sets up the node agent as a bootstrapper rather than a service model
change ci.openshift.org to use that config model when baking nodes (prove baking nodes works)
build an auto approver for ci.openshift.org that uses observe + cert data
get experience in all the failure modes
in 1.6 or 1.7 update node configuration as necessary, and possibly support "real" kube bootstrap mode (which looks like 1.7 at the earliest)

When the --bootstrap flag is provided to `start node`, require --kubeconfig, connect to the master and perform a CSR request for a client cert and a server cert. The node then attempts to read a node-config ConfigMap from openshift-infra - if no value exists, it uses the default config. If the CSR is not approved within a few minutes, the node will crash loop. On the master, enable the cert signer and grant the node-bootstrapper service account the appropriate permission to request CSRs.

smarterclayton · 2017-01-24T04:03:52Z

#12635, marked experimental, [merge]

…

On Mon, Jan 23, 2017 at 10:50 PM, OpenShift Bot ***@***.***> wrote: continuous-integration/openshift-jenkins/test SUCCESS ( https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/13214/) (Base Commit: b0f2c58 <b0f2c58> ) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#9547 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p_qNHC1PbihgMjKefxu0t9t8WvUqks5rVXUigaJpZM4I92KL> .

openshift-bot · 2017-01-24T04:05:43Z

Evaluated for origin test up to 2525370

openshift-bot · 2017-01-24T04:05:53Z

Evaluated for origin merge up to 2525370

openshift-bot · 2017-01-24T05:30:56Z

continuous-integration/openshift-jenkins/test SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/13220/) (Base Commit: c971421)

openshift-bot · 2017-01-24T10:53:31Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pr_origin/13232/) (Base Commit: 22d4240) (Image: devenv-rhel7_5761)

debianmaster · 2017-03-22T09:18:35Z

can you please add a simple readme on how to use this feature?

debianmaster · 2017-07-15T15:56:02Z

still not clear how to generate the boostrap secret. can you please add simple readme or post commands here i will write a READMe i tried multiple options but could not get it right

smarterclayton · 2017-07-15T16:24:27Z

oc serviceaccounts create-kubeconfig -n openshift-infra node-boostrapper > FILE On Jul 15, 2017, at 11:56 AM, Chakradhar Rao Jonagam < [email protected]> wrote: still not clear how to generate the boostrap secret. can you please add simple readme or post commands here i will write a READMe i tried multiple options but could not get it right — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#9547 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p0MWT1_2co2MqXlNRDQMfY3qi0woks5sOOEVgaJpZM4I92KL> .

debianmaster · 2017-07-16T22:34:04Z

looks like a typo in node-boostrappe im going go try this and let you know

smarterclayton · 2017-07-16T22:46:51Z

You didn't spell bootstrapper correctly On Jul 16, 2017, at 6:28 PM, Chakradhar Rao Jonagam < [email protected]> wrote: @smarterclayton <https://github.com/smarterclayton> this is what i get version information at the bottom. $ oc serviceaccounts create-kubeconfig -n openshift-infra node-boostrapper Error from server (NotFound): serviceaccounts "node-boostrapper" not found $ oc version oc v3.6.0-rc.0+98b3d56 kubernetes v1.6.1+5115d708d7 features: Basic-Auth Server https://192.168.99.100:8443 openshift v3.6.0-rc.0+98b3d56 kubernetes v1.6.1+5115d708d7 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9547 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p0vRdrtFOHG_AfYDvBTMT4RzFJXHks5sOo5sgaJpZM4I92KL> .

debianmaster · 2017-07-16T23:00:33Z

Tried with correct spelling now.

Here is what i tried on mac.

master

oc cluster up --docker-machine --create-machine 
oc serviceaccounts create-kubeconfig -n openshift-infra node-bootstrapper > secret

node1

node1
oc cluster join  --docker-machine=n1 --create-machine --secret='conentent of secret'

Failed with following error on node1 docker-machine

-- Finding server IP ...
   Using docker-machine IP 192.168.99.101 as the host IP
   Using 192.168.99.101 as the server IP
-- Joining OpenShift cluster ...
   Starting OpenShift Node using container 'origin'
FAIL
   Error: could not start OpenShift container "origin"
   Details:
     No log available from "origin" container

Davidsoff · 2017-07-20T14:36:00Z

same issue here using:

master

oc cluster up --use-existing-config --host-data-dir=/var/lib/origin/openshift.local.etcd/ --host-volumes-dir=/var/lib/origin/openshift.local.volumes/ --host-pv-dir=/var/lib/orign/openshift.local.pv/

node

oc login -t <token>
oc serviceaccounts create-kubeconfig -n openshift-infra node-bootstrapper > secret 
oc cluster join  --secret='conentent of secret'

result

failed on the node with the following error:

-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v3.6.0-rc.0 image ... OK
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ...
   Using nsenter mounter for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ...
   Using 127.0.0.1 as the server IP
-- Joining OpenShift cluster ...
   Starting OpenShift Node using container 'origin'
FAIL
   Error: could not start OpenShift container "origin"
   Details:
     No log available from "origin" container

debianmaster · 2017-07-20T14:38:08Z

cc/ @smarterclayton

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 20, 2016

openshift-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Oct 22, 2016

openshift-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Nov 2, 2016

smarterclayton force-pushed the joincommand branch from c0f4599 to 9407251 Compare November 18, 2016 16:46

openshift-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 18, 2016

smarterclayton force-pushed the joincommand branch 3 times, most recently from f6480bd to d9c6a4a Compare November 20, 2016 01:25

smarterclayton force-pushed the joincommand branch from d9c6a4a to 988c0bb Compare November 20, 2016 02:09

smarterclayton changed the title ~~WIP - enable oc cluster join an existing cluster~~ Allow nodes to bootstrap and add oc cluster join for nodes Nov 20, 2016

smarterclayton added this to the 1.5.0 milestone Nov 20, 2016

smarterclayton force-pushed the joincommand branch 3 times, most recently from c6b970f to d6e1c68 Compare November 21, 2016 23:00

dgoodwin reviewed Nov 22, 2016

View reviewed changes

deads2k reviewed Nov 22, 2016

View reviewed changes

openshift-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 26, 2016

smarterclayton force-pushed the joincommand branch from d6e1c68 to d970ddb Compare November 28, 2016 22:16

liggitt reviewed Jan 23, 2017

View reviewed changes

smarterclayton force-pushed the joincommand branch from 10b924a to dcdb29c Compare January 24, 2017 00:50

smarterclayton added 5 commits January 23, 2017 23:03

Add oc cluster join

3f42e19

Blacklist pkg/bootstrap/run from govet

1e80200

Add the certificate command into admin

2f7f2ce

generated: docs

2525370

smarterclayton force-pushed the joincommand branch from dcdb29c to 2525370 Compare January 24, 2017 04:03

openshift-bot merged commit c5868ac into openshift:master Jan 24, 2017

coreydaley mentioned this pull request Jan 25, 2017

oc cluster up won't start after adding the check for port 53 availability #12646

Closed

jawnsy mentioned this pull request Mar 2, 2017

Openshift adding new nodes to cluster failed #12359

Closed

LorbusChris mentioned this pull request Mar 29, 2017

oc cluster join -- how to use? #13336

Closed

Allow nodes to bootstrap and add oc cluster join for nodes #9547

Allow nodes to bootstrap and add oc cluster join for nodes #9547

Conversation

smarterclayton commented Jun 24, 2016 • edited Loading

smarterclayton commented Nov 20, 2016

smarterclayton commented Nov 20, 2016

smarterclayton commented Nov 20, 2016

smarterclayton commented Nov 20, 2016

dgoodwin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgoodwin commented Nov 22, 2016

Choose a reason for hiding this comment

smarterclayton commented Nov 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarterclayton commented Jan 24, 2017 via email

liggitt commented Jan 24, 2017

liggitt commented Jan 24, 2017

smarterclayton commented Jan 24, 2017

smarterclayton commented Jan 24, 2017 via email

openshift-bot commented Jan 24, 2017

openshift-bot commented Jan 24, 2017

openshift-bot commented Jan 24, 2017

openshift-bot commented Jan 24, 2017 • edited Loading

debianmaster commented Mar 22, 2017

debianmaster commented Jul 15, 2017

smarterclayton commented Jul 15, 2017 via email

debianmaster commented Jul 16, 2017

smarterclayton commented Jul 16, 2017 via email

debianmaster commented Jul 16, 2017 • edited Loading

master

node1

Davidsoff commented Jul 20, 2017 • edited Loading

master

node

result

debianmaster commented Jul 20, 2017

Allow nodes to bootstrap and add `oc cluster join` for nodes #9547

Allow nodes to bootstrap and add `oc cluster join` for nodes #9547

smarterclayton commented Jun 24, 2016 •

edited

Loading

openshift-bot commented Jan 24, 2017 •

edited

Loading

debianmaster commented Jul 16, 2017 •

edited

Loading

Davidsoff commented Jul 20, 2017 •

edited

Loading