XFS quota for emptyDir volumes #9261

mburke5678 · 2018-05-15T14:44:25Z

Seth:
As of 3.10, we no longer use the atomic-openshift-node binary. All the functions it wrapped (ovn, ovs, kubelet) have all been broken out into separate services. The kubelet is now the hyperkube binary built from openshift/kubernetes.

This means all the custom stuff we had wrapping the kubelet for our purposes is no long in effect. The node-config.yaml file is not used directly by anything. The "openshift start --write-flags" command is used to convert the node-config.yaml into a list of flags appropriate for the kubelet. Among these patched-in features that don't translate in the new setup is XFS quota via the LocalQuota option in the node-config.yaml

https://docs.openshift.com/container-platform/3.9/install_config/master_node_configuration.html#master-node-config-volume-config

https://docs.openshift.com/container-platform/3.9/install_config/master_node_configuration.html#node-configuration-files

I converted that patch to a carry in our openshift/kubernetes in this PR
openshift/origin#19533

However, because the kubelet does not understand the node-config.yaml and there is no flag by which the XFS quota configuration can be conveyed, the patch reads a new config file at a fixed location to configure the XFS quota.

The new config file path is
/var/lib/origin/openshift.local.volumes/volume-config.yaml

The file has the format:

apiVersion: kubelet.config.openshift.io/v1
kind: VolumeConfig
localQuota:
perFSGroupInGiB: 1

This feature is very lightly documented. The only two places I can find it mentioned are the links above. The primary user, to our knowledge, is Online.

Michael, can you open a PR for 3.10 that updates the documentation to reflect this change?

Thanks and let me know if you have any questions.

FYI @sjenning

sjenning · 2018-05-15T16:35:00Z

install_config/master_node_configuration.adoc

+You can use the link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-xfs[XFS quota subsystem] 
+to limit the size of `emptyDir` volumes  and volumes based on an `emptyDir` volume, such as secrets and configmaps, on each node.
+
+Enable a quota on local storage use per unique FSGroup ID usin g the *_/var/lib/origin/openshift.local.volumes/volume-config.yaml_* file.


s/usin g/using/

sjenning · 2018-05-15T16:36:37Z

install_config/master_node_configuration.adoc

+---
+
+<1> Contains options for controlling local volume quota on the node.
+<2> Set this value to a resource quantity representing the desired quota per 


Due to some complications in the code, this value is now numeric, not a resource quantity. It is the quota in GBs.

sjenning · 2018-05-15T16:37:28Z

install_config/master_node_configuration.adoc

+<1> Contains options for controlling local volume quota on the node.
+<2> Set this value to a resource quantity representing the desired quota per 
+xref:../install_config/persistent_storage/pod_security_context.adoc#fsgroup[FSGroup], per node (such as 1Gi, 512Mi, etc). 
+Requires the *volumeDirectory* to be on an XFS filesystem mounted with the 'gquota' option. The matching security context contraint *fsGroup* type must be 


s/gquota/grpquota/

gquota is a alias for grpquota, but grpquota is the real option

sjenning · 2018-05-15T16:40:44Z

Thanks @mburke5678 for the quick response on this!
/cc @derekwaynecarr

Look good overall. I had a few comments.

mburke5678 · 2018-05-16T19:41:15Z

@sjenning Is the perFSGroupInGiB setting in GB (gigabyte) or GiB (gibibyte) as the parameter has GiB in its name (or, does it not really matter for our purposes?).

sjenning · 2018-05-18T15:51:18Z

standby on this. we might be changing how this is done.

https://bugzilla.redhat.com/show_bug.cgi?id=1572285#c5

i'll keep you posted.

sjenning · 2018-05-18T19:59:41Z

follow-on PR openshift/origin#19773

Transforms perFSGroupInGB back to perFSGroup with a Quantity value.
Changes the config file location to /etc/origin/node/volume-config.yaml so that it can be a second key to the existing node-config-compute configmap and everything will Just Work (tm).

mburke5678 · 2018-05-18T20:43:54Z

@sjenning I changed the parameter to perFSGroup: 50 and the text to: "Sets the maximum number of emptyDir volumes for each unique FSGroup ID." Also replaced the path to the yaml.
PTAL

mburke5678 · 2018-05-18T21:03:19Z

@mdshuai Can someone on your team review this PR?

mburke5678 · 2018-05-25T17:06:47Z

@mdshuai Can someone on your team review this PR?

sjenning · 2018-05-25T19:10:20Z

day_two_guide/topics/node_maintenance.adoc

@@ -262,7 +262,7 @@ be configured:
 NOTE: If the XFS filesystem hosting that folder is mounted with the
 `gquota` option in the `/etc/fstab`
 * the matching security context contraint’s `fsGroup` type set to `MustRunAs`
-* The `/etc/origin/node/node-config.yml` is configured as:
+* The xref:../install_config/master_node_configuration.adoc#node-config-volume-config`/var/lib/origin/openshift.local.volumes/volume-config.yaml` is configured] as:


This file is now /etc/origin/node/volume-config.yaml

sjenning · 2018-05-25T19:11:32Z

install_config/master_node_configuration.adoc

+apiVersion: kubelet.config.openshift.io/v1
+kind: VolumeConfig
+localQuota: <1>
+  perFSGroup: 50 <2>


needs a unit. use 1Gi or 512Mi as an example

sjenning · 2018-05-25T19:14:02Z

install_config/master_node_configuration.adoc

+---
+
+<1> Contains the options for controlling local volume quota on the node.
+<2> Sets the maximum number of `emptyDir` volumes for each unique FSGroup ID. 


this is not right.

Specifies the maximum amount of storage that can be used for disk-based emptyDir volumes on the node for each unique FSGroup ID

sjenning · 2018-05-31T20:48:42Z

We just got tasked with auditing the docs for references to /etc/origin/node/node-config.yaml because that file should no longer be directly edited.

Instead, cluster ops should edit the node-config.yaml key in the node-config-<role> configmap in openshift-node namespace.

Really the same thing applies for this as it will be deployed using the volume-config.yaml key in the same configmap (openshift/openshift-ansible#8450).

sjenning · 2018-05-31T20:52:11Z

install_config/master_node_configuration.adoc

+to limit the number of `emptyDir` volumes and volumes based on an `emptyDir` volume, such as secrets and configmaps, on each node.
+
+Enable a quota on local storage for each unique xref:../architecture/additional_concepts/authorization.adoc#security-context-constraints[FSGroup ID] 
+using the *_/etc/origin/node/volume-config.yaml_* file.


Probably add this here:

NOTE: The volume-config.yaml file should not be edited directly as it is created from the node-config-compute configmap in the openshift-node namespace. Creating or editing volume-config.yaml should be done via the configmap and not directly on the node.

@sjenning What does the code in the config map configure perFSGroup look like?

An example:

apiVersion: kubelet.config.openshift.io/v1 kind: VolumeConfig localQuota: perFSGroup: 1Gi

mburke5678 · 2018-06-01T17:59:29Z

@sjenning What does the code in the config map configure perFSGroup look like?

mburke5678 · 2018-06-01T18:51:44Z

@chao007 PTAL

chao007 · 2018-06-04T09:49:02Z

install_config/master_node_configuration.adoc

-<2> Set this value to a resource quantity representing the desired quota per 
-xref:../install_config/persistent_storage/pod_security_context.adoc#fsgroup[FSGroup], per node (such as 1Gi, 512Mi, etc). 
-Requires the *volumeDirectory* to be on an XFS filesystem mounted with the 'gquota' option. The matching security context contraint *fsGroup* type must be 
+<2> Set ththe desired quota per 


"ththe"? spelling error?

mburke5678 · 2018-06-04T17:11:53Z

@chao007 I do not see ththe in the source. Can you verify if ththe is no longer present?

sjenning · 2018-06-04T21:40:11Z

install_config/master_node_configuration.adoc

+
+[NOTE]
+====
+You should not edit the *_/var/lib/origin/openshift.local.volumes/volume-config.yaml_* file directly. The file is created from the *_node-config-compute_* configuration map. 


/etc/origin/node/volume-config.yaml

chao007 · 2018-06-05T06:38:12Z

install_config/master_node_configuration.adoc

+kind: VolumeConfig
+localQuota: <1>
+  perFSGroupInGiB: 1Gi <2>
+----


Is this updated to "perFSGroup" ？

@chao007 Fixed. Changed to perFSGroup: 1Gi per Seth's comment. #9261 (comment)

sjenning · 2018-06-05T20:37:50Z

lgtm, thanks 👍

mburke5678 · 2018-06-06T15:52:42Z

@openshift/team-documentation PTAL

kalexand-rh

I'm slightly confused by some of the legacy language and have a few picks.

kalexand-rh · 2018-06-06T16:20:13Z

day_two_guide/topics/node_maintenance.adoc

@@ -260,9 +260,9 @@ is lost. The location of the data in the *{rhocp}* node is located in
 be configured:

 NOTE: If the XFS filesystem hosting that folder is mounted with the
-`gquota` option in the `/etc/fstab`
+`grpquota` option in the `/etc/fstab`
 * the matching security context contraint’s `fsGroup` type set to `MustRunAs`


The structure of this list is confusing, and I'm not sure what you mean by "The using the ... configuration map."

Maybe

grpquota option in the /etc/fstab file:

Set the matching security context contraint’s fsGroup type to MustRunAs.

xref:../install_config/master_node_configuration.adoc#node-config-volume-config[Configure the volume] using the node-config-compute configuration map in the openshift-node project.
(and remove the example volumeconfig because it's not really helping here.)

@kalexand-rh Looking around, I don't think this topic is being used at this time. It was existing text. I made your changes, but did not remove the example. I am not sure why it is there.
I did a search and found gquota and changed to grpquota per Seth's comment.

Thanks, @mburke5678!
@bfallonf, do you know if this file should be used or removed? It looks like it was commented out of the day_two_guide/host_level_tasks topic.

kalexand-rh · 2018-06-06T16:25:03Z

install_config/master_node_configuration.adoc

+
+<1> Contains options for controlling local volume quota on the node.
+<2> Set this value to a resource quantity representing the desired quota per 
+[FSGroup], per node, such as 1Gi, 512Mi, and so forth. 


I'd set the values off in backticks to emphasize that they're example parameter values. (Otherwise, you need spaces between the value and abbreviation because of Style.)

kalexand-rh · 2018-06-06T16:25:23Z

install_config/master_node_configuration.adoc

+
+[NOTE]
+====
+You should not edit the *_/etc/origin/node/volume-config.yaml_* file directly. The file is created from the *_node-config-compute_* configuration map. 


s/You should/Do

kalexand-rh · 2018-06-06T16:25:37Z

install_config/master_node_configuration.adoc

+[NOTE]
+====
+You should not edit the *_/etc/origin/node/volume-config.yaml_* file directly. The file is created from the *_node-config-compute_* configuration map. 
+Use the *_node-config-compute_* configuration map to create or edit the paramaters in the *_volume-config.yaml_*.


s/volume-config.yaml/volume-config.yaml file

kalexand-rh · 2018-06-06T16:31:01Z

install_config/master_node_configuration.adoc

@@ -1082,13 +1082,10 @@ If -1, there is no limit on requests.
 |A boolean to enable or disable dynamic provisioning. Default is *true*.

 |FSGroup


I'd consider adding emphasis on FSGroup.

kalexand-rh · 2018-06-06T16:31:04Z

install_config/master_node_configuration.adoc

@@ -1082,13 +1082,10 @@ If -1, there is no limit on requests.
 |A boolean to enable or disable dynamic provisioning. Default is *true*.

 |FSGroup
-|Can be specified to enable a quota on local storage use per unique FSGroup ID.
+|Can be specified to xref:node-config-volume-config[enable a quota on local storage] use per unique FSGroup ID.


This is confusing. Do you mean "Can be specified to xref:node-config-volume-config[enable a quota on local storage]. You must enable quota for each unique FSGroup ID." ?

The feature sets a blanket quota for all FSgroups on each node.
"Enables xref:node-config-volume-config[local storage quotas] on each node for each FSGroup."

mburke5678 · 2018-06-07T20:55:22Z

/cherrypick enterprise-3.10

openshift-cherrypick-robot · 2018-06-07T20:55:30Z

@mburke5678: new pull request created: #9907

In response to this:

/cherrypick enterprise-3.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 15, 2018

mburke5678 added the branch/enterprise-3.10 label May 15, 2018

mburke5678 added this to the Future Release milestone May 15, 2018

sjenning reviewed May 15, 2018

View reviewed changes

openshift-ci-robot requested a review from derekwaynecarr May 15, 2018 16:40

mburke5678 changed the title ~~[WIP] XFS quota for emptyDir volumes~~ XFS quota for emptyDir volumes May 16, 2018

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 16, 2018

sjenning reviewed May 25, 2018

View reviewed changes

sjenning reviewed May 31, 2018

View reviewed changes

mburke5678 force-pushed the XFS-quota branch from 3e75d2a to dd701e6 Compare June 1, 2018 17:46

chao007 reviewed Jun 4, 2018

View reviewed changes

sjenning reviewed Jun 4, 2018

View reviewed changes

chao007 reviewed Jun 5, 2018

View reviewed changes

mburke5678 force-pushed the XFS-quota branch from e4588da to b62fae2 Compare June 5, 2018 13:46

mburke5678 force-pushed the XFS-quota branch 3 times, most recently from fe6aab1 to 1eca68c Compare June 6, 2018 15:48

kalexand-rh reviewed Jun 6, 2018

View reviewed changes

kalexand-rh added the peer-review-done Signifies that the peer review team has reviewed this PR label Jun 6, 2018

XFS quota for emptyDir volumes

f5fb200

mburke5678 force-pushed the XFS-quota branch from f5785bb to f5fb200 Compare June 7, 2018 16:05

mburke5678 merged commit ee6e798 into openshift:master Jun 7, 2018

openshift-cherrypick-robot mentioned this pull request Jun 7, 2018

[enterprise-3.10] XFS quota for emptyDir volumes #9907

Merged

mburke5678 deleted the XFS-quota branch June 7, 2018 20:55

vikram-redhat modified the milestones: Future Release, OCP 3.10 GA Jul 31, 2018

		@@ -1082,13 +1082,10 @@ If -1, there is no limit on requests.
		\|A boolean to enable or disable dynamic provisioning. Default is true.

		\|FSGroup

XFS quota for emptyDir volumes #9261

XFS quota for emptyDir volumes #9261

Conversation

mburke5678 commented May 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjenning May 15, 2018 • edited Loading

Choose a reason for hiding this comment

sjenning commented May 15, 2018

mburke5678 commented May 16, 2018

sjenning commented May 18, 2018

sjenning commented May 18, 2018 • edited Loading

mburke5678 commented May 18, 2018 • edited Loading

mburke5678 commented May 18, 2018

mburke5678 commented May 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjenning commented May 31, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mburke5678 commented Jun 1, 2018 • edited Loading

mburke5678 commented Jun 1, 2018

Choose a reason for hiding this comment

mburke5678 commented Jun 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjenning commented Jun 5, 2018

mburke5678 commented Jun 6, 2018

kalexand-rh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mburke5678 Jun 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mburke5678 commented Jun 7, 2018

openshift-cherrypick-robot commented Jun 7, 2018

sjenning May 15, 2018 •

edited

Loading

sjenning commented May 18, 2018 •

edited

Loading

mburke5678 commented May 18, 2018 •

edited

Loading

mburke5678 commented Jun 1, 2018 •

edited

Loading

mburke5678 Jun 6, 2018 •

edited

Loading