No license information for binutils-aarch64-linux-gnu

[albe19029]

Description

Good day, have next issue and need help to resolve it.
I have next base ubuntu image with installed binutils package there.

Dockerfile:

FROM ubuntu
RUN apt-get -y update && apt-get -y install binutils

But when I try to get license information for binutils-aarch64-linux-gnu package - I don't have it.
I use next command:

trivy image --format cyclonedx --output 1.json {my_image} --license-full --debug --license-confidence-level=0.1

So as you can see google classifier (https://github.com/google/licenseclassifier) should check the content of /usr/share/doc/binutils-aarch64-linux-gnu/copyright file

But if I use identify_license (v2) it produce me correct information:

2025/02/12 14:50:25 Classifying license(s): /usr/share/doc/binutils-aarch64-linux-gnu/copyright
2025/02/12 14:50:25 Finished Classifying License "/usr/share/doc/binutils-aarch64-linux-gnu/copyright": 3.04325ms
/usr/share/doc/binutils-aarch64-linux-gnu/copyright GPL-3.0 (variant: a.txt, confidence: 0.9017857142857143, start: 17, end: 30)end: 30)

As I can see both projects assets.DefaultClassifier()

So I can't understand why identify_license can get license information, and trivy don't

Desired Behavior

trivy find license information for binutils-aarch64-linux-gnu

Actual Behavior

trivy don't find license information for binutils-aarch64-linux-gnu

Reproduction Steps

1. run: trivy image --format cyclonedx --output 1.json {my_image} --license-full --debug --license-confidence-level=0.1
2. run: identify_license /usr/share/doc/binutils-aarch64-linux-gnu/copyright (headers option must be set to true)
3. identify_license find license, trivy don't.
...

Target

None

Scanner

None

Output Format

None

Mode

None

Debug Output

trivy image --format cyclonedx --output /Users/albe/1.json albe_image --license-full --debug --license-confidence-level=0.1
2025-02-12T15:28:48+03:00	DEBUG	No plugins loaded
2025-02-12T15:28:48+03:00	DEBUG	Default config file "file_path=trivy.yaml" not found, using built in values
2025-02-12T15:28:48+03:00	DEBUG	Cache dir	dir="/Users/albe/Library/Caches/trivy"
2025-02-12T15:28:48+03:00	DEBUG	Cache dir	dir="/Users/albe/Library/Caches/trivy"
2025-02-12T15:28:48+03:00	DEBUG	Parsed severities	severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2025-02-12T15:28:48+03:00	DEBUG	Ignore statuses	statuses=[]
2025-02-12T15:28:48+03:00	INFO	"--format cyclonedx" disables security scanning. Specify "--scanners vuln" explicitly if you want to include vulnerabilities in the "cyclonedx" report.
2025-02-12T15:28:48+03:00	DEBUG	[pkg] Package types	types=[os library]
2025-02-12T15:28:48+03:00	DEBUG	[pkg] Package relationships	relationships=[unknown root workspace direct indirect]
2025-02-12T15:28:48+03:00	DEBUG	Initializing scan cache...	type="fs"
2025-02-12T15:28:48+03:00	DEBUG	[image] Detected image ID	image_id="sha256:cbdc5bd9e29ebf8921c9ecafea2fb3f94eba51385dd23a550023096df46b5d04"
2025-02-12T15:28:48+03:00	DEBUG	[image] Detected diff ID	diff_ids=[sha256:375990b2a90a8d8f332d9b9422d948f7068a3313bf5a1c9fbb91ff2d29046130 sha256:b1378c57a3b9d6c70f56a179300f20978a34ab64f5bf7d7a5ae909c901a1ebe7]
2025-02-12T15:28:48+03:00	DEBUG	[image] Detected base layers	diff_ids=[sha256:375990b2a90a8d8f332d9b9422d948f7068a3313bf5a1c9fbb91ff2d29046130]
2025-02-12T15:28:48+03:00	DEBUG	[image] Missing image ID in cache	image_id="sha256:cbdc5bd9e29ebf8921c9ecafea2fb3f94eba51385dd23a550023096df46b5d04"
2025-02-12T15:28:48+03:00	DEBUG	[image] Missing diff ID in cache	diff_id="sha256:375990b2a90a8d8f332d9b9422d948f7068a3313bf5a1c9fbb91ff2d29046130"
2025-02-12T15:28:48+03:00	DEBUG	[image] Missing diff ID in cache	diff_id="sha256:b1378c57a3b9d6c70f56a179300f20978a34ab64f5bf7d7a5ae909c901a1ebe7"
2025-02-12T15:28:48+03:00	DEBUG	[dpkg] Unable to parse the available file	file_path="var/lib/dpkg/available" err="file open error: open var/lib/dpkg/available: file does not exist"
2025-02-12T15:28:49+03:00	DEBUG	Loading the default license classifier...
2025-02-12T15:28:49+03:00	DEBUG	No secrets found in container image config
2025-02-12T15:28:49+03:00	INFO	Detected OS	family="ubuntu" version="24.04"
2025-02-12T15:28:49+03:00	INFO	Number of language-specific files	num=0
2025-02-12T15:28:49+03:00	DEBUG	Specified ignore file does not exist	file=".trivyignore"
2025-02-12T15:28:49+03:00	DEBUG	[vex] VEX filtering is disabled

Operating System

Mac

Version

0.59.1

Checklist

• Run trivy clean --all
• Read the troubleshooting

[DmitriyLewen]

Helllo @albe19029
Thanks for your report!

For deb packages Trivy detects licenses from /usr/share/doc/*/copyright files.
But file for binutils-aarch64-linux-gnu is link:

➜ docker run -it --rm 8391 ls -ahl /usr/share/doc | grep binutils-aarch64-linux-gnu
lrwxrwxrwx 1 root root   11 Aug  7  2024 binutils-aarch64-linux-gnu -> libbinutils

Trivy doesn't currently support links - #5356

Regards, Dmitriy

[albe19029]

Thanks for the clarification

[albe19029]

As I understand here we skip symlinks ((https://github.com/aquasecurity/trivy/blob/main/pkg/fanal/walker/tar.go#L74)

But can we modify dpkgLicenseAnalyzer to make it PostAnalyzer? In this case we can implement Required method to get all /usr/share/doc/*/copyright files, and then in Analyze we will have all files content to read it without new tar iteration?

And we only need to pass symlink and process it here: (https://github.com/aquasecurity/trivy/blob/main/pkg/fanal/artifact/image/image.go#L385)

Maybe I missed something, correct me if I am wrong. In other case can we do that?

[albe19029]

Maybe we should create WaitGroup and there will be a separate thread for parsing logic, but PostAnalyze should be called after all layers parsed, with resolved compositeFS files from other filesystem?

[DmitriyLewen]

this is the difficulty of implementing it.

you can add new logic to PostAnalyze only if the original file and the link are in the same layer.

If the original file is added in one layer, and the link is added in another - you will have 2 different PostAnalyze functions running

[DmitriyLewen]

Maybe we should create WaitGroup and there will be a separate thread for parsing logic, but PostAnalyze should be called after all layers parsed, with resolved compositeFS files from other filesystem?

To do this, we need to change the entire caching logic, since we store each layer in a separate bucket.

[albe19029]

Sorry, can you please explain in more details about separate bucket?

[DmitriyLewen]

We check each layer separately and save it to cache:

trivy/pkg/fanal/artifact/image/image.go

Lines 332 to 338 in 73bd20d

	layerInfo, err := a.inspectLayer(ctx, layer, disabledAnalyzers)
	if err != nil {
	return nil, xerrors.Errorf("failed to analyze layer (%s): %w", layer.DiffID, err)
	}
	if err = a.cache.PutBlob(layerKey, layerInfo); err != nil {
	return nil, xerrors.Errorf("failed to store layer: %s in cache: %w", layerKey, err)
	}

So cache contains separate bucket for each layer.
e.g.

Cache uses BlobInfo struct (that you can understand what info we save)

[albe19029]

But I don't want to change this logic, I just want to resolve symlinks from other layes, by coping directory content from other lower layers filesystems.

So from my perspective a.walker.Walk will execute for every layer. But then they all will wait to all layers. And having in case symlink is on another layer - files will be copied from another compositeFile. but as I will copy it with symlink name all layers will have only their files, but with resolve symlinks.

No need to change logic with caches.

[albe19029]

Thanks a lot. Never knew about it, now it clear to me how it works.

From what I see:

1. we still need to read information from lower layers to get base information
2. we need to save links information the same way as OpaqueDirs and WhiteoutFiles. In this case having symlinks, we can compare it with new Files, and having base information decide if it valid, or was deleted or modified.

But to start I need to understand - do you think that we can resolve symlink from lower layers? As without this action there will be no information for future comparation in ApplyLayers layer. For now want to implement it for layer.Licenses, as other Analyzer need more time to investigate logic.

[DmitriyLewen]

do you think that we can resolve symlink from lower layers?

As i wrote before - i think we can't do that.
Possible cases when we will deceive users.
We should take info from all layers.

[albe19029]

Ok, now it is clear for me that with this kind of cache it is impossible to resolve the problems, as each layer is incomplete to get 100% sure result.

But why then in this configuration I also fail?

echo 'FROM ${image}' > Dockerfile
docker build -o rootfs .
trivy filesystem ./rootfs --format cyclonedx --output 1.json --license-full

Here in roots I have all layers combined in a single layers. So I have full view to analyze. And I use filesystem instead of image. And still don't have result. Is symlinks also don't work with filesystem?

[albe19029]

As I see you use - filepath.WalkDir. And from documentation "WalkDir does not follow symbolic links."

[albe19029]

scan.txt
This is the only way how to fix it to get license information for all packages.

[albe19029]

How do you think is it possible to provide original filesystem fs.FS in analyzer.PostAnalysisInput (in filesystem type). In this case it will be possible to read file having package name without iteration on filesystem. So if for example we detected package binutils-aarch64-linux-gnu we can check /usr/share/doc/binutils-aarch64-linux-gnu/copyright, even if it is a symlink. It will not fix image type, but at least filesystem type will produce correct information, as it has only one layer.

[albe19029]

At least grype reads copyright information like this and resolve it, when package name was detected and it knows license file location without filesystem iteration. So if we will have os.DirFS, it will be possible to read content when packages were detected in dpkgAnalyzer

[DmitriyLewen]

I am currently focused on other higher priority tasks and cannot give a correct answer.
You can try to implement this. If it works, we will review your PR.

[albe19029]

#8424

[albe19029]

I have much better implementation now. #8432
This code can work for all scanners (for now it was just implemented for image scan)
I also need help to fix cache version change correctly, as added new field to it.

So basic idea:

1. I push symlinks to analyzers and having it I can predict where information about license can be
2. As information can be on different layers and in cache, so I merge it only in ApplyLayers code where I have all layers.
3. If prediction is valid - I get information from source file of another package.