Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add integration test for persistent-volume-size config option #4186

Open
anjannath opened this issue May 30, 2024 · 7 comments · May be fixed by #4623
Open

Add integration test for persistent-volume-size config option #4186

anjannath opened this issue May 30, 2024 · 7 comments · May be fixed by #4623
Assignees
@rohanKanojia
Labels
kind/task Workable task

Comments

@anjannath
Copy link
Member

this is to be used with the disk-size config option, a user has to set. crc config set disk-size 35 then crc config set persistent-volume-size 4 can be set as default disk size in 31G and the extra increase of 4G can now be used for increasing the topolvm partition

this config option is currently only used with microshift preset but after #4117 is resolved will also be used for the openshift preset
@anjannath anjannath added the kind/task Workable task label May 30, 2024
@rohanKanojia rohanKanojia self-assigned this Feb 7, 2025
@rohanKanojia rohanKanojia moved this to Work In Progress in Project planning: crc Feb 11, 2025
@rohanKanojia
Copy link
Contributor

I'm observing strange behavior while writing e2e test for this scenario.

No matter what value I set in persistent-volume-size configuration option, I'm always seeing 15GB size when I ssh into the partition.

This is the config I'm using:

$ crc config view
- consent-telemetry                     : no
- disk-size                             : 35
- persistent-volume-size                : 40
- preset                                : microshift

When I start the cluster and ssh into the VM and execute lsblk command, I see this output:

NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda           252:0    0   35G  0 disk 
├─vda1        252:1    0  200M  0 part /boot/efi
├─vda2        252:2    0  800M  0 part /boot
├─vda3        252:3    0    1M  0 part 
├─vda4        252:4    0    1M  0 part 
└─vda5        252:5    0   34G  0 part 
  └─rhel-root 253:0    0   15G  0 lvm  /var/lib/containers/storage/overlay
                                       /var
                                       /sysroot/ostree/deploy/rhel/var
                                       /usr
                                       /etc
                                       /
                                       /sysroot

While debugging I'm observing that sizeToIncrease is being evaluated as a negative integer value here:

crc/pkg/crc/machine/start.go

Lines 200 to 205 in 2499dd2

sizeToIncrease := expectedLVSize - lvSize
lvPath := fmt.Sprintf("/dev/%s", lvFullName)
if sizeToIncrease > 1 {
logging.Info("Extending and resizing '/dev/rhel/root' logical volume")
if _, _, err := sshRunner.RunPrivileged("Extending and resizing the logical volume(LV)", "/usr/sbin/lvextend", "-L", fmt.Sprintf("+%db", sizeToIncrease), lvPath, "--devices", rootPart); err != nil {
return err

Is this behavior expected? Am I looking at the right place?

@anjannath
Copy link
Member Author

Is this behavior expected? Am I looking at the right place?

yes kind of expected as the persistent-volume-size cannot be larger then disk-size since its a lvm volume and exists inside the disk whose size is controlled by the disk-size value

we should also add a config validation that checks that persistent-volume-size is always smaller than disk-size

@rohanKanojia
Copy link
Contributor

yes kind of expected as the persistent-volume-size cannot be larger then disk-size since its a lvm volume and exists inside the disk whose size is controlled by the disk-size value

Even if I set some smaller value, I'm not able to see lvm size change other than 15G

we should also add a config validation that checks that persistent-volume-size is always smaller than disk-size

@anjannath : There is a validation check in place for this config option to not receive any value less than 15

crc/pkg/crc/validation/validation.go

Lines 47 to 50 in 457b653

func ValidatePersistentVolumeSize(value int) error {
if value < constants.DefaultPersistentVolumeSize {
return fmt.Errorf("requires disk size in GiB >= %d", constants.DefaultPersistentVolumeSize)
}

@praveenkumar
Copy link
Member

@rohanKanojia if this is not working as expectation then it is a bug.

@rohanKanojia
Copy link
Contributor

@praveenkumar @anjannath : I've tried with several combinations but I can't get it to change lvm size value to other than 15G

No Config for Persistent Volume size

❯ crc config view
- consent-telemetry                     : no
- disk-size                             : 35
- preset                                : microshift

[core@api ~]$ lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda           252:0    0   35G  0 disk 
├─vda1        252:1    0  200M  0 part /boot/efi
├─vda2        252:2    0  800M  0 part /boot
├─vda3        252:3    0    1M  0 part 
├─vda4        252:4    0    1M  0 part 
└─vda5        252:5    0   34G  0 part 
  └─rhel-root 253:0    0   15G  0 lvm  /var/lib/containers/storage/overlay
                                       /var
                                       /sysroot/ostree/deploy/rhel/var
                                       /usr
                                       /etc
                                       /
                                       /sysroot

20G Persistent volume size

❯ crc config view
- consent-telemetry                     : no
- disk-size                             : 35
- persistent-volume-size                : 20
- preset                                : microshift

[core@api ~]$ lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda           252:0    0   35G  0 disk 
├─vda1        252:1    0  200M  0 part /boot/efi
├─vda2        252:2    0  800M  0 part /boot
├─vda3        252:3    0    1M  0 part 
├─vda4        252:4    0    1M  0 part 
└─vda5        252:5    0   34G  0 part 
  └─rhel-root 253:0    0   15G  0 lvm  /var/lib/containers/storage/overlay
                                       /var
                                       /sysroot/ostree/deploy/rhel/var
                                       /usr
                                       /etc
                                       /
                                       /sysroot

25G persistent volume size

- consent-telemetry                     : no
- disk-size                             : 35
- persistent-volume-size                : 25
- preset                                : microshift

NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda           252:0    0   35G  0 disk 
├─vda1        252:1    0  200M  0 part /boot/efi
├─vda2        252:2    0  800M  0 part /boot
├─vda3        252:3    0    1M  0 part 
├─vda4        252:4    0    1M  0 part 
└─vda5        252:5    0   34G  0 part 
  └─rhel-root 253:0    0   15G  0 lvm  /var/lib/containers/storage/overlay
                                       /var
                                       /sysroot/ostree/deploy/rhel/var
                                       /usr
                                       /etc
                                       /
                                       /sysroot

@anjannath
Copy link
Member Author

Did some testing to figure out how the size of the partition and volumes is determined, and i observed the following, first i tried to change both the disk-size and persistent-volume-size configs

 % crc config view
- consent-telemetry                     : no
- disk-size                             : 41
- host-network-access                   : true
- persistent-volume-size                : 20
- preset                                : microshift

 % ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no -oIdentitiesOnly=yes -i ~/.crc/machines/crc/id_ed25519 [email protected] -p 2222 -- lsblk
Warning: Permanently added '[127.0.0.1]:2222' (ED25519) to the list of known hosts.
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda           252:0    0   41G  0 disk
├─vda1        252:1    0  200M  0 part /boot/efi
├─vda2        252:2    0  800M  0 part /boot
├─vda3        252:3    0    1M  0 part
├─vda4        252:4    0    1M  0 part
└─vda5        252:5    0   40G  0 part
  └─rhel-root 253:0    0   20G  0 lvm  /var/lib/containers/storage/overlay
                                       /var
                                       /sysroot/ostree/deploy/rhel/var
                                       /usr
                                       /etc
                                       /
                                       /sysroot

Observations

The amount of space we want to increase for persistent-volume-size it has to be first added to the disk-size, in the above example we wanted the persistent-volume-size to be 20G which is +6G increase from the default size because initially actual persistent-volume-size is 19G, i.e 34-15 = 19

this is where we have a bug although the persistent-volume-size value is 15 by default, we have 19G of space for PV in actual, i.e 34-15 = 19 (34G is the whole vda5 partition, 15G is for the rhel-root volume and remaining (19G) can be utilized for PV creation)

so it seems the persistent-volume-size actually defines the size of the rhel-root volume and the actual persistent-volume-size that is available for PV creation is the amount of free space left on vda5 partition

this is the calculation: available for pv-size = disk-size - 21G (this 21G is because we have 1G for boot, efi partitions (/boot + /boot/efi) and 15G for the root (/sysroot) partition all of these are lvm volumes, in the default configuration we have 35G as disk-size of which 1G goes for boot+efi partition and from the remaining 34G, 15G is for the rhel-root volume and the remaining 19G is free space that can be used for PV

Note

when we increase the persistent-volume-size what it actually does is it tries to increase the amount of free space and make the free space left equal to the value of persistent-volume-size with a constraint that rhel-root volume cannot be less then 15G

in the above example with disk-size set to 41G we can set the persistent-volume-size to a maximum of 25G (i.e 41 - (1 + 15) = 25

% crc config view
- consent-telemetry                     : no
- disk-size                             : 41
- host-network-access                   : true
- persistent-volume-size                : 25
- preset                                : microshift

% ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no -oIdentitiesOnly=yes -i ~/.crc/machines/crc/id_ed25519 [email protected] -p 2222 -- lsblk
Warning: Permanently added '[127.0.0.1]:2222' (ED25519) to the list of known hosts.
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda           252:0    0   41G  0 disk
├─vda1        252:1    0  200M  0 part /boot/efi
├─vda2        252:2    0  800M  0 part /boot
├─vda3        252:3    0    1M  0 part
├─vda4        252:4    0    1M  0 part
└─vda5        252:5    0   40G  0 part
  └─rhel-root 253:0    0   15G  0 lvm  /var/lib/containers/storage/overlay
                                       /var
                                       /sysroot/ostree/deploy/rhel/var
                                       /usr
                                       /etc
                                       /
                                       /sysroot

if i try to increase the persistent-volume-size by more than that then it is not effective:

 % crc config view
- consent-telemetry                     : no
- disk-size                             : 41
- host-network-access                   : true
- persistent-volume-size                : 27
- preset                                : microshift

% ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=no -oIdentitiesOnly=yes -i ~/.crc/machines/crc/id_ed25519 [email protected] -p 2222 -- lsblk
Warning: Permanently added '[127.0.0.1]:2222' (ED25519) to the list of known hosts.
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda           252:0    0   41G  0 disk
├─vda1        252:1    0  200M  0 part /boot/efi
├─vda2        252:2    0  800M  0 part /boot
├─vda3        252:3    0    1M  0 part
├─vda4        252:4    0    1M  0 part
└─vda5        252:5    0   40G  0 part
  └─rhel-root 253:0    0   15G  0 lvm  /var/lib/containers/storage/overlay
                                       /var
                                       /sysroot/ostree/deploy/rhel/var
                                       /usr
                                       /etc
                                       /
                                       /sysroot

@rohanKanojia
Copy link
Contributor

@anjannath : Thanks, I can see the change in lvm disk size when I increase disk-size .

The amount of space we want to increase for persistent-volume-size it has to be first added to the disk-size

I see. So the value of lvm partition depends on disk-size rather than persistent-volume-size .

I'll try to add integration test with increased disk-size

@rohanKanojia rohanKanojia moved this to In Progress in Eclipse JKube Feb 14, 2025
rohanKanojia added a commit to rohankanojia-forks/crc that referenced this issue Feb 14, 2025
@rohanKanojia
test (e2e) : add step for setting persistent-volume-size config opt…
54913de 
…ion (crc-org#4186)

Add step for `persistent-volume-size` config option in story_microshift.feature

Signed-off-by: Rohan Kumar <[email protected]>
This was referenced Feb 14, 2025
Open
Open
rohanKanojia added a commit to rohankanojia-forks/crc that referenced this issue Feb 18, 2025
@rohanKanojia
test (e2e) : add step for setting persistent-volume-size config opt…
8ec1320 
…ion (crc-org#4186)

Add step for `persistent-volume-size` config option in story_microshift.feature

Signed-off-by: Rohan Kumar <[email protected]>
rohanKanojia added a commit to rohankanojia-forks/crc that referenced this issue Feb 18, 2025
@rohanKanojia
test (e2e) : add step for setting persistent-volume-size config opt…
5f1bbae 
…ion (crc-org#4186)

Add step for `persistent-volume-size` config option in story_microshift.feature

Signed-off-by: Rohan Kumar <[email protected]>
rohanKanojia added a commit to rohankanojia-forks/crc that referenced this issue Feb 18, 2025
@rohanKanojia
test (e2e) : add step for setting persistent-volume-size config opt…
e505f6a 
…ion (crc-org#4186)

Add step for `persistent-volume-size` config option in story_microshift.feature

Signed-off-by: Rohan Kumar <[email protected]>
@rohanKanojia rohanKanojia moved this from Work In Progress to Ready for review in Project planning: crc Feb 19, 2025
@rohanKanojia rohanKanojia moved this from In Progress to Review in Eclipse JKube Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rohanKanojia rohanKanojia

Labels
kind/task Workable task
Projects
Project planning: crc
Status: Ready for review
Milestone
No milestone
3 participants
@praveenkumar @anjannath @rohanKanojia