kops no longer supplies content-md5 header when writing to S3

[akloss-cibo]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.

Client version: 1.30.4 (git-v1.30.4)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

NA. Can't provision a cluster (but I'm trying 1.30.9).

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

Note that the state bucket needs object lock configured to reproduce this error.

KOPS_STATE_STORE=s3://<state-bucket-redacted> kops create cluster --name=k8s-cluster.example.com --zones=us-east-1a
I0204 17:54:07.483215    2059 new_cluster.go:1454] Cloud Provider ID: "aws"
I0204 17:54:08.174376    2059 subnets.go:224] Assigned CIDR 172.20.0.0/16 to subnet us-east-1a
Error: error writing updated configuration: error writing Cluster "k8s-cluster.example.com": error writing configuration file s3://tech-cibo-k8s-infra-test/k8s-cluster.example.com/config: error writing s3://tech-cibo-k8s-infra-test/k8s-cluster.example.com/config: operation error S3: PutObject, https response error StatusCode: 400, RequestID: PDGZS3Q92SG52NPZ, HostID: AkmnoIOaC7VpTMrHxKcnTJq5rVxlyS6TFFTgUQnLtxgMxWyDcXl2/86bkyiccxYRELoYhG/UtR8=, api error InvalidRequest: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters
%

5. What happened after the commands executed?

No cluster configured. See above.

6. What did you expect to happen?

A cluster to be configured

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

NA. See example above.

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

The interesting part:

I0204 17:55:42.419879    2070 populate_instancegroup_spec.go:76] Populating instance group spec for "control-plane-us-east-1a"
I0204 17:55:42.420179    2070 populate_instancegroup_spec.go:76] Populating instance group spec for "nodes-us-east-1a"
I0204 17:55:42.422567    2070 s3fs.go:385] Reading file "s3://bucket-name-redacted/k8s-cluster.example.com/config"
I0204 17:55:42.431761    2070 s3fs.go:303] Writing file "s3://bucket-name-redacted/k8s-cluster.example.com/config"
I0204 17:55:42.431781    2070 s3context.go:259] Checking default bucket encryption for "bucket-name-redacted"
I0204 17:55:42.431792    2070 s3context.go:264] Calling S3 GetBucketEncryption Bucket="bucket-name-redacted"
I0204 17:55:42.444596    2070 s3context.go:284] bucket "bucket-name-redacted" has default encryption set to true
I0204 17:55:42.444611    2070 s3fs.go:323] Calling S3 PutObject Bucket="bucket-name-redacted" Key="k8s-cluster.example.com/config" SSE="DefaultBucketEncryption" ACL=""
Error: error writing updated configuration: error writing Cluster "k8s-cluster.example.com": error writing configuration file s3://bucket-name-redacted/k8s-cluster.example.com/config: error writing s3://bucket-name-redacted/k8s-cluster.example.com/config: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 3352CJ5E5RK3YW84, HostID: ulkqNpzUWym8pJKy0yur7jwGSUe2eigqHjsfY0N1ndaA3Ja/xRX6qx35eUvg9Ox9Mk5y3xGnnQM=, api error InvalidRequest: Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters

9. Anything else do we need to know?

This has been reported as a problem for the PHP AWS SDK and Amazon indicated it is still a user/client concern to supply the request parameters to include the relevant headers.