Custom labels are only collected from gke.gcr.io/pause@sha256, not from all containers in the pod.

[asbbx]

Hello!

I'm running cAdvisor as a DaemonSet in GKE (version 1.29.6). While trying to collect custom labels from all containers in my pod, I'm only getting them from the gke.gcr.io/pause@sha256 image in pod.

Args from cadvisor:

args:

- -housekeeping_interval=10s
- -max_housekeeping_interval=15s
- -event_storage_event_limit=default=0
- -event_storage_age_limit=default=0
- -enable_metrics=app,cpu,disk,diskIO,memory,network,process
- -docker_only
- -store_container_labels=false
- -whitelisted_container_labels=io.kubernetes.container.name,io.kubernetes.pod.name,io.kubernetes.pod.namespace,mycustomlabel/somevalue

Could you help me identify what I might be missing? I would appreciate any insights.

[skgsergio]

We've faced the same issue and found that the problem here is that containerd doesn't have the labels in all the containers of the pod and cadvisor scrapes at containerd level. Not sure if much can be done.

For example, note our missing my.redacted.kabel/* in the helper container:

sh-5.1# ctr -n k8s.io c ls labels.io.kubernetes.pod.name==redacted-pod-name-xxm53eou
CONTAINER                                                           IMAGE                                                                                                                   RUNTIME
10de4d5704415a7bbd18abb1e54ebed216547b2fff89c1290b316fc9ff117bff    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx                                                io.containerd.runc.v2
e5c8ba6db57ec604df59361f6f7bda31308f6a7b541ae839d162bf0b7151e192    gke.gcr.io/gke-release/gke-release/pause:3.8@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830    io.containerd.runc.v2

sh-5.1# ctr -n k8s.io c info e5c8ba6db57ec604df59361f6f7bda31308f6a7b541ae839d162bf0b7151e192 | jq .Labels
{
  "INCLUDES_SOURCE": "/source.tar.xz",
  "SOURCES_INCLUDED": "./build/pause/",
  "io.cri-containerd.kind": "sandbox",
  "io.kubernetes.pod.name": "redacted-pod-name-xxm53eou",
  "io.kubernetes.pod.namespace": "redacted-namespace",
  "io.kubernetes.pod.uid": "f10421d7-d8f5-42bd-b99d-d0eb246a2c7d",
  "my.redacted.label/id": "1337",
  "my.redacted.label/name": "this_value_is_redacted",
  "pod": "redacted-pod-name"
}

sh-5.1# ctr -n k8s.io c info 10de4d5704415a7bbd18abb1e54ebed216547b2fff89c1290b316fc9ff117bff | jq .Labels
{
  "io.cri-containerd.kind": "container",
  "io.kubernetes.container.name": "helper",
  "io.kubernetes.pod.name": "redacted-pod-name-xxm53eou",
  "io.kubernetes.pod.namespace": "redacted-namespace",
  "io.kubernetes.pod.uid": "f10421d7-d8f5-42bd-b99d-d0eb246a2c7d"
}