Streaming Encoding for LIST Responses

[serathius]

Enhancement Description

• One-line enhancement description (can be used as a release note): Streaming Response Encoding
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/5116-streaming-response-encoding/README.md
• Previous discussion: Don't copy whole response during response marshalling  kubernetes#129304, Streaming json list encoder kubernetes#129334
• Primary contact (assignee): serathius@
• Responsible SIGs: api-machinery
• Enhancement target (which target equals to which milestone):
  • Beta release target (x.y): 1.33
  • Stable release target (x.y): 1.34
• Beta
  • KEP (k/enhancements) update PR(s): KEP-5116: Add KEP (Streaming response encoding) #5119
  • Code (k/k) update PR(s):
    • Implement tests for deferredResponseWriter to validate behavior on multiple writer calls. kubernetes#130168
    • Implement chunking for gzip encoder in deferredResponseWriter kubernetes#130264
    • Implement LIST benchmark for different content types (Json, Proto, CBOR, YAML) kubernetes#130169
    • Implement LIST benchmarks for different API kinds (Configmap, Pod, CustomResource)
    • Implement static analysis to validate json and proto tags of builtin List types kubernetes#130216
    • Add tests for encoding collections in JSON for KEP-5116 kubernetes#130220
    • Implement unit tests for Proto encoder
    • Implement JSON streaming encoder
    • Implement Proto streaming encoder
    • Reduce memory allocated for control plane in benchmarks
  • Docs (k/website) update(s):
    • No planning doc update as feature is not user facing.

[serathius]

/sig api-machinery

[chenk008]

I'm glad to see this proposal. We have also implemented similar capabilities in our inner repo and are preparing to push this part to upstream. We have submitted a CFP for the upcoming KubeCon China conference.

In our implementation, we use sync.Pool to efficiently manage memory allocation and cache the serialized results of each item. When the buffer reaches a certain size, we execute a flush operation to parallelize the serialization processing and write to http2.

Additionally, we have added support for gzip compression, which is only enabled when the first batch of cached data reaches 128 * 1024.

For json serialization, we have customized the StreamMarshal method for unstructuredList.

As for protobuf, we generate code through a generator to ensure reverse protobuf marshalling compatibility.

type StreamMarshaller interface {
	// return the object size and the item size slice
	StreamSize() (uint64, []int)

	StreamMarshal(w stream.Writer, itemSize []int) error
}

And it has conducted extensive testing with large datasets and have obtained comparative results. @yulongfang Can you share some benchmark results?

[yulongfang]

Thank @chenk008 for your introduction. We have many large-scale clusters in Alibaba Cloud. When the controllers of these large-scale clusters are restarted, they will initiate a full list request to the apiserver, which will have a certain impact on the stability of the cluster. We have to use larger machines to run the apiserver, resulting in a waste of resources.

In this context, we adopted the method to carry out relevant optimization and achieved the following results.

list json format return data stress test scenario description:

• apiserver version: 1.30
• apiserver specification: 32c 128GB
• apiserver replica number: 1 replica
• number of stock resources: build 10,000 100kb cr information
• stress test scenario: increase pressure according to the gradient of qps 0.1 / 0.5

list json format return data related stress test data:

qps 0.05

• before optimization: cpu 35.7 c mem 89Gb
• stream json after optimization: cpu 6.22 c mem 60 Gb

qps 0.1

• before optimization: cpu 11 c mem 146Gb
• stream json after optimization: cpu 7.45 c mem 97 Gb

list protobuf Format Returned data Stress test scenario description:

• apiserver version: 1.30
• apiserver specification: 32c 128GB
• apiserver replica number: 1 replica
• Number of existing resources: Build 10,000 configmaps information of size 100kb
• Stress test scenario: Increase pressure according to the gradient of qps 0.1 / 0.5

list configmaps format Returned data Related stress test data:

qps 0.05

• Before optimization: cpu 16.8 c mem 54.3Gb
• After stream json optimization: cpu 16.8 c mem 16.1 Gb

qps 0.1

• Before optimization: cpu 42 c mem 122Gb
• After stream json optimization: cpu 42 c mem 18 Gb

[BenTheElder]

FYI: Technical details are usually discussed in KEP PRs or elsewhere, with the KEP issue serving as a place to link back work.

@chenk008 @yulongfang you might consider reviewing #5119

[serathius]

Hey @chenk008 @yulongfang please see the previous discussion in kubernetes/kubernetes#129304 and kubernetes/kubernetes#129334.
We also have already done a performance analysis of our changes in kubernetes/kubernetes#129304 (comment).

We also added running a automatic benchmark of list requests. You can see the results in https://perf-dash.k8s.io/#/?jobname=benchmark%20list&metriccategoryname=E2E&metricname=Resources&Resource=memory&PodName=kube-apiserver-benchmark-list-master%2Fkube-apiserver

We currently run it in JSON + configmap with RV="" configuration, hope to expand it to include Proto, Pods, CustomResources and other types of LIST request. Would be awesome if you can contribute.

[serathius]

/milestone v1.33

[pacoxu]

@jpbetz @dipesh-rawat this is target to v1.33 and the KEP was merged.
Should the lead-opted-in and tracked label be added and tracked by release team?

[dipesh-rawat]

@serathius @pacoxu Unfortunately, the enhancement freeze deadline has passed, and this KEP issue was not lead-opted-in, so it wasn’t added to the tracking board for the v1.33 release. Post-freeze, we've disabled the automated sync job for KEP issues to the tracking board.

To move forward, we’ll need a short exception request filed so the team can add the lead-opted-in label and manually include this in the tracking board.

If you still wish to progress this enhancement in v1.33, please file an exception request as soon as possible, within three days. If you have any questions, you can reach out in the #release-enhancements channel on Slack and we'll be happy to help. Thanks!

(cc v1.33 Release Lead @npolshakova)

[serathius]

Ups, @jpbetz is OOO. @deads2k can you take a look?

[serathius]

Sent https://groups.google.com/g/kubernetes-sig-release/c/fDI9FdlClnA

[dipesh-rawat]

@serathius Since the release team has APPROVED the exception request here. This will be considered to be added to the milestone for v1.33 release.

[dipesh-rawat]

Hello @serathius 👋, v1.33 Enhancements team here.

This enhancement is targeting stage beta for v1.33 (correct me, if otherwise)
/stage beta

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.33. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline on Thursday 6th February 2025 so that the PRR team has enough time to review your KEP.

With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

Could we please link the KEP README in the issue description.

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/5116-streaming-response-encoding/README.md

The status of this enhancement is marked as Tracked for enhancements freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

/label tracked/yes

[dipesh-rawat]

I've manually added this KEP to the tracking board and marked it as tracked for enhancements freeze🚀

Could one of the sig leads add the lead-opted-in label? @deads2k, would you be able to help with this or point me to someone who can? Thanks!

[dipesh-rawat]

Could one of the sig leads add the lead-opted-in label?

@serathius Would you be able to assist with the above request? It would be great to get the label added as work is being done in this v1.33 release.

[serathius]

I'm not a SIG api-machinery lead, so I don't think I should use it. I can ask nicely on Slack.

[deads2k]

/label lead-opted-in
/milestone v1.33