[Feature Request] Hot-plug vCPUs

[jeromegn]

Feature Request

We'd like to be able to add (and remove) vCPUs from running firecracker microvms. It appears to be possible to do that with KVM. Examples I've seen define how many vCPUs at most a VM might use and then the actual number it will be using at boot. Then you can add vCPUs via virsh.

https://www.unixarena.com/2015/12/linux-kvm-how-to-add-remove-vcpu-to-guest-on-fly.html/

This would allow use to add a "burst" feature when CPU usage spikes.

Describe the desired solution

An API to modify a running microvm's vCPUs count.

This should notify the guest VM of the change:

Dec 16 12:48:28 UA-KVM1 kernel: CPU1 has been hot-added
Dec 16 12:48:28 UA-KVM1 kernel: SMP alternatives: switching to SMP code
Dec 16 12:48:57 UA-KVM1 kernel: smpboot: Booting Node 0 Processor 1 APIC 0x1
Dec 16 12:48:57 UA-KVM1 kernel: kvm-clock: cpu 1, msr 0:3ff87041, secondary cpu clock
Dec 16 12:48:57 UA-KVM1 kernel: TSC synchronization [CPU#0 -> CPU#1]:
Dec 16 12:48:57 UA-KVM1 kernel: Measured 906183720569 cycles TSC warp between CPUs, turning off TSC clock.
Dec 16 12:48:57 UA-KVM1 kernel: tsc: Marking TSC unstable due to check_tsc_sync_source failed
Dec 16 12:48:57 UA-KVM1 kernel: KVM setup async PF for cpu 1
Dec 16 12:48:57 UA-KVM1 kernel: kvm-stealtime: cpu 1, msr 3fd0d240
Dec 16 12:48:57 UA-KVM1 kernel: microcode: CPU1 sig=0x206c1, pf=0x1, revision=0x1
Dec 16 12:48:57 UA-KVM1 kernel: Will online and init hotplugged CPU: 1

Describe possible alternatives

We could give every firecracker microvm access to all cores and only use cgroups to limit actual scheduling time. This is not great though as it might create a lot of CPU steal. We prefer to give full cores when possible.

Checks

• Have you searched the Firecracker Issues database for similar requests?
• Have you read all the existing relevant Firecracker documentation?
• Have you read and understood Firecracker's core tenets?

[KarthikNedunchezhiyan]

@jeromegn does vertical scaling is the only solution to the problem you are to trying to fix? Does horizontal scaling like multiple instances not helped. Just curious to understand the usecase.

[raduiliescu]

Hi @jeromegn!

You might also want to look at cpu online/offline kernel feature - https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-devices-system-cpu.
On the downside you will always need to start the microVM with the max number of CPUs, and you will need an agent inside the microVM to write the values in /proc, but the overhead of keeping CPUs offline is better than in the cgroup case.

[jeromegn]

@KarthikNedunchezhiyan we need to support a large variety of workloads. More VMs isn't always the solution, but we're already doing that.

@raduiliescu thanks! That could work, but our users have root access to the VM and could bring up any number of CPUs.

[AlexandruCihodaru]

We need to think about it for a bit, we will get back to you.

[wearyzen]

marking this as parked right now but we will track this as part of our roadmap. We consider this as well while working on this.

[roypat]

Hey all,
Just to leave a quick update here, we've prototyped vCPU hotplug in the feature/vcpu-hotplug feature branch. The results of performance testing there was that the end-to-end latency of hotplugging and on-lining an addition vCPU was ~30ms per vCPU, which is prohibitively expensive for the post-restore scenario (where we want to be able to resume guest execution within single digit milliseconds). As such, we're moving this ticket back to the backlog for now :(

[wjordan]

Thanks for the update on this. Just to clarify, 30ms per vCPU sounds like an acceptable amount of latency for a vCPU hotplug process for most use cases, but it sounds like there's an issue in the prototype that requires re-adding previously-hotplugged vCPUs after a snapshot-restore, which is why this latency applies to the post-restore scenario.

In the linked branch, it sounds like the issue has something to do with the vCPU config not persisting after a restore:

firecracker/docs/vcpu-hotplug.md

Lines 127 to 132 in c41ceb5

	### Snapshotting
	vCPU hot-plugging after snapshot restore is currently disabled. This is due to
	the vCPU config not persisting after restore. Trying to hot-plug vCPUs after a
	snapshot restore will return `HotplugVcpuError::RestoredFromSnapshot` until this
	has been fixed.

Is this understanding correct? Does the next step in the implementation involve updating the snapshot state format to include the relevant vCPU hotplug info in the microVM state, or was there some other issue in the implementation that requires vCPU hotplug latency in the snapshot restore process?

[roypat]

Hi Will,

Just to clarify, 30ms per vCPU sounds like an acceptable amount of latency for a vCPU hotplug process for most use cases, but it sounds like there's an issue in the prototype that requires re-adding previously-hotplugged vCPUs after a snapshot-restore, which is why this latency applies to the post-restore scenario.

We only implemented pre-snapshot hotplugging back in July to get an idea of the latencies involved, but for the usecase we were looking at, we actually want to hotplug vCPUs only after restore (e.g. no hotplug ever happens before a snapshot is taken).

Is this understanding correct? Does the next step in the implementation involve updating the snapshot state format to include the relevant vCPU hotplug info in the microVM state, or was there some other issue in the implementation that requires vCPU hotplug latency in the snapshot restore process?

Essentially, the entire ACPI hotplug device would need to be persisted in the snapshot, but generally there shouldn't be much of an issue with that.