Degrade performance of the k8s cluster scanning in v0.59

[unmade]

Description

I noticed a significant difference in performance for k8s cluster scanning between v0.58 and v0.59. The v0.59 takes approx. 10x more time to complete.

Here is the command I'm running:

trivy \
  --report all \
  --format json \
  --scanners vuln \
  k8s \
  "$SCAN_CONTEXT" \
  --include-namespaces "$SCAN_NAMESPACES" \
  --exclude-kinds replicationcontrollers,daemonsets,cronjobs,jobs,services,configmaps,roles,rolebindings,networkpolicies,ingresses,resourcequotas,limitranges,clusterroles,clusterrolebindings \
  --disable-node-collector \
  --debug

Trivy is granted the following role:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cluster-trivy-scanner
rules:
- apiGroups:
    - ""
    - "apps"
  resources:
    - deployments
    - namespaces
    - nodes
    - pods
    - replicasets
    - secrets
    - serviceaccounts
    - statefulsets
  verbs:
    - get
    - list
    - watch

I tried local runs as well as runs from within k8s cluster.

I don't get much information from the output other than on v0.59 the waiting time is significantly more than for v0.58. Any help into how to debug/profile where the bottleneck would be appreciated!

Desired Behavior

Performance remains the same for k8s cluster scanning in version 0.59

Actual Behavior

Performance degraded the same for k8s cluster scanning in version 0.59

Reproduction Steps

Have a k8s cluster with few deployments, replicasets, statefulsets. Grant Trivy Role as in the example above. 

1. Run the scan with Trivy v0.58.3
2. Run the scan with Trivy v0.59.1

Target

Kubernetes

Scanner

Vulnerability

Output Format

JSON

Mode

Standalone

Debug Output

There is no much difference in the output between 0.58 and 0.59, other that 0.59 takes much longer to complete.

Operating System

macOS / Docker container deployed to k8s

Version

$ trivy --version
Version: 0.58.3

$ trivy --version
Version: 0.59.1

Checklist

• Run trivy clean --all
• Read the troubleshooting

[simar7]

@unmade can you share some examples of what the degradation looks like? Does it just take longer? Consume more memory? CPU? All of the above?

cc @afdesk

[unmade]

I didn't notice change in resource consumption, only that it just takes way longer. Looks like it can be related to --include/exclude-kinds flag and please check my reply to the comment below. T

he longer execution can be explained that much more resource are now being scanned, which I guess is not a performance downgrade per se

[afdesk]

@unmade thanks for the report.
I'll take a look

update:

@unmade Could you share more details about time?
I tried to reproduce it quickly on my minikube instance and couldn't do it (

$ time trivy k8s --scanners vuln -f json --report all   --exclude-kinds replicationcontrollers,daemonsets,cronjobs,jobs,services,configmaps,roles,rolebindings,networkpolicies,ingresses,resourcequotas,limitranges,clusterroles,clusterrolebindings
...
trivy k8s --scanners vuln -f json --report all --exclude-kinds   0.65s user 0.51s system 8% cpu 14.132 total

$ time ~/.opt/trivy-58-2 k8s --scanners vuln -f json --report all   --exclude-kinds replicationcontrollers,daemonsets,cronjobs,jobs,services,configmaps,roles,rolebindings,networkpolicies,ingresses,resourcequotas,limitranges,clusterroles,clusterrolebindings
...
~/.opt/trivy-58-2 k8s --scanners vuln -f json --report all --exclude-kinds   0.64s user 0.38s system 9% cpu 10.910 total

There was changed a way to include/exclude kinds, so I believe that it may contain some issues.

also, in your case, why do use long exclude-kinds instead of include-kinds?

$ trivy k8s --scanners vuln --include-kinds pods,deployments,replicasets,statefulsets,serviceaccounts --disable-node-collector --report summary

[afdesk]

also I have another idea.
how many SCAN_NAMESPACES do you have?

[afdesk]

IIUC your case, scan without --include-namespaces "$SCAN_NAMESPACES" may be faster )
@unmade could you check it? thanks!

[unmade]

Thank you for quick replies and providing some ideas! And apologies it took me a bit longer than expected to reply.

why do use long exclude-kinds instead of include-kinds

I guess it was just my preference at the time, no particular reason really

how many SCAN_NAMESPACES do you have?
IIUC your case, scan without --include-namespaces "$SCAN_NAMESPACES" may be faster

For test runs I explicitly provide exactly one namespace, it is the same namespace for each run. Unfortunately, providing more namespace or not specifying it all takes much longer and makes it harder to debug

---

Yesterday I did a few test runs with different Trivy versions. Indeed, longer time seems to be related to --inlcude/exclude-kinds behavior now. Here are are my results:

Args	v0.55.0	v0.56.2	v0.57.1	v0.58.2	v0.59.1
no --include/exclude-kinds provided	55s	43s	43s	41s	42s
--include-kinds provided	-	-	-	empty result*	396s
--exclude-kinds provided	-	-	-	40s	393s

--inlcude/exclude-kinds flags were provided as follows:

• --include-kinds pods,deployments,replicasets,statefulsets,serviceaccounts
• --exclude-kinds replicationcontrollers,daemonsets,cronjobs,jobs,services,configmaps,roles,rolebindings,networkpolicies,ingresses,resourcequotas,limitranges,clusterroles,clusterrolebindings

The strange thing is that on v0.58.2 when I provide --include-kinds option I get empty results (e.g. {'ClusterName': 'test-cluster'}. This is not the case if I provide --exclude-kinds.

Additionally, on v0.59.1 I noticed that DaemonSet pods are scanned and included in report as well. When I explicitly exclude pods via --exclude-kinds, I see that ReplicaSet pods are scanned and included in report anyway. The scan took 241s instead of 393s.

For my case I think I can just omit --include/exclude-kinds flag completely and it will perform as before.

[simar7]

Thanks @unmade for the detailed report.

Actually we made a few fixes in (v0.59.1), one of the specifically targeting resources that get included/excluded with the flags. It is documented in more detail here aquasecurity/trivy-kubernetes#395

Having said that, the longer times are most likely IMO due to the fact that we're actually picking up the correct resources now and not because of a slowdowns. This might explain your empty result in the case of v0.58.2.

Let us know if you'd like us to investigate further.

[unmade]

@simar7

the longer times are most likely IMO due to the fact that we're actually picking up the correct resources

Agree, for me it was the case and simply omitting these flags completely works just as before. I feel like it is more or less clear now and I'm closing the discussion.

Thank you guys, for your help and for maintaining Trivy!