fix: Retry KVM_CREATE_VM on EINTR #5046
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The error variant is used when creating a new VM from KVM FD via KVM_CREATE_VM call. On success, VM FD is returned. The previous error message doesn't tell a lie but is a bit unintuitive. Signed-off-by: Takahiro Itazuri <[email protected]>
zulinx86
added
the
Status: Awaiting review
|
Question:
|
zulinx86
force-pushed
the
EINTR_on_KVM_CREATE_VM
branch
from
f292393 to
dd04187
Compare
roypat
reviewed
Feb 21, 2025
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #5046 +/- ##
==========================================
- Coverage 83.20% 83.19% -0.02%
==========================================
Files 247 247
Lines 26631 26641 +10
==========================================
+ Hits 22158 22163 +5
- Misses 4473 4478 +5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
zulinx86
force-pushed
the
EINTR_on_KVM_CREATE_VM
branch
from
dd04187 to
52127c1
Compare
Manciukic
reviewed
Feb 21, 2025
zulinx86
force-pushed
the
EINTR_on_KVM_CREATE_VM
branch
2 times, most recently
from
028f3f6 to
fb23c4f
Compare
It is known that KVM_CREATE_VM fails with EINTR on heavily loaded machines with many VMs. It might be a kernel bug but apparently has not been fixed. To mitigate it, QEMU does an infinitely retry on EINTR. Similar, do retries up to 5 times. Signed-off-by: Takahiro Itazuri <[email protected]>
zulinx86
force-pushed
the
EINTR_on_KVM_CREATE_VM
branch
from
fb23c4f to
be26374
Compare
ShadowCurse
approved these changes
Feb 21, 2025
| Ok(fd) => return Ok(fd), | ||
| Err(e) if e.errno() == libc::EINTR && attempt < MAX_ATTEMPTS => { | ||
| info!("Attemp #{attempt} of KVM_CREATE_VM returned EINTR"); | ||
| // Exponential backoff (1us, 2us, 4us, and 8us => 15us in total) |
There was a problem hiding this comment.
Shouldn't it be 16us in the end? 2^4?
There was a problem hiding this comment.
The sleep duration is 2^(attempt - 1). The attempt starts from 1 to 5, but if attempt == 5 it goes to the third hand of the match expression. So 2^(1 - 1) = 2^0 = 1, 2^(2 - 1) = 2^1 = 2, 2^(3 - 1) = 2^2 = 4, and 2^(4 - 1) = 2^3 = 8.
There was a problem hiding this comment.
Ok, so it is the sum.
Btw, why do we even need the backoff?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reason & Changes
It is known that KVM_CREATE_VM occasionally fails with EINTR on heavily loaded machines
with many VMs.
The behavior itself that KVM_CREATE_VM can return EINTR is intentional. This is because
the KVM_CREATE_VM path includes mm_take_all_locks() that is CPU intensive and all CPU
intensive syscalls should check for pending signals and return EINTR immediately to allow
userland to remain interactive.
https://lists.nongnu.org/archive/html/qemu-devel/2014-01/msg01740.html
However, it is empirically confirmed that, even though there is no pending signal,
KVM_CREATE_VM returns EINTR.
https://lore.kernel.org/qemu-devel/[email protected]/
To mitigate it, QEMU does an inifinite retry on EINTR that greatly improves reliabiliy:
Similarly, we do retries up to 5 times. Although Firecracker clients are also able to
retry, they have to start Firecracker from scratch. Doing retries in Firecracker makes
recovery faster and improves reliability.
License Acceptance
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.PR Checklist
tools/devtool checkstyleto verify that the PR passes theautomated style checks.
how they are solving the problem in a clear and encompassing way.
[ ] I have updated any relevant documentation (both in code and in the docs)in the PR.
[ ] I have mentioned all user-facing changes inCHANGELOG.md.[ ] If a specific issue led to this PR, this PR closes the issue.[ ] When making API changes, I have followed theRunbook for Firecracker API changes.
integration tests.
TODO.rust-vmm.