Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Event Based" replication fails to replicate all tags, "Scheduled" or "Manual" replicates properly. #21623

Open
JonTheNiceGuy opened this issue Feb 12, 2025 · 4 comments · May be fixed by #21660
Open

"Event Based" replication fails to replicate all tags, "Scheduled" or "Manual" replicates properly. #21623

JonTheNiceGuy opened this issue Feb 12, 2025 · 4 comments · May be fixed by #21660
Assignees
@wy65701436 @chlins
Labels
area/replication kind/bug needs/follow-up target/2.13.0 issues that are targeting v2.13.0

Comments

@JonTheNiceGuy
Copy link

Expected behavior and actual behavior:

I have created a Push replication between two harbor instances. The replication when set to "Event" replication fails to copy all of the tags, however a "scheduled" or "manual" replication will copy all the tags. Here are the two sets of logs:

2025-02-07T10:46:12Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: harbor, URL: [http://harbor-core:80](http://harbor-core/), insecure: true] created
2025-02-07T10:46:12Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: harbor, URL: https://backup.service.com/, insecure: false] created
2025-02-07T10:46:12Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying infrastructure/cipa:[fb2b069bf317df7a6d69fcdb63db22206be4bd2f](source registry) to infrastructure/cipa:[fb2b069bf317df7a6d69fcdb63db22206be4bd2f](destination registry)...
2025-02-07T10:46:12Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying infrastructure/cipa:fb2b069bf317df7a6d69fcdb63db22206be4bd2f(source registry) to infrastructure/cipa:fb2b069bf317df7a6d69fcdb63db22206be4bd2f(destination registry)...
2025-02-07T10:46:12Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact infrastructure/cipa:fb2b069bf317df7a6d69fcdb63db22206be4bd2f ...
2025-02-07T10:46:13Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact infrastructure/cipa:fb2b069bf317df7a6d69fcdb63db22206be4bd2f pulled
2025-02-07T10:46:14Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:a5fe02fc724ab71e1b940ab28099d7a0cf4e19cf84927ea9059d8f0522ead485(the 1th running)...
2025-02-07T10:46:15Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:a5fe02fc724ab71e1b940ab28099d7a0cf4e19cf84927ea9059d8f0522ead485 completed
2025-02-07T10:46:15Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:fce7cf3b093c38c2b925d509ff746cf17fc58f52052ced37a9c167d2534c6d34(the 1th running)...
2025-02-07T10:46:16Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:fce7cf3b093c38c2b925d509ff746cf17fc58f52052ced37a9c167d2534c6d34 completed
2025-02-07T10:46:16Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:63ef709101fd354930798471618ead195a468dfc5b4b4a078bc302db40f7b215(the 1th running)...
2025-02-07T10:46:19Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:63ef709101fd354930798471618ead195a468dfc5b4b4a078bc302db40f7b215 completed
2025-02-07T10:46:19Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:79a7e77018a9ebf49d78b30fe3696de894c29fb805eeb00574c50a3ae1972557(the 1th running)...
2025-02-07T10:46:26Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:79a7e77018a9ebf49d78b30fe3696de894c29fb805eeb00574c50a3ae1972557 completed
2025-02-07T10:46:26Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:c80635538dfb2b2872f4713d817d0c4bd25b43ad807f92c62088b8a13ac21580(the 1th running)...
2025-02-07T10:46:28Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:c80635538dfb2b2872f4713d817d0c4bd25b43ad807f92c62088b8a13ac21580 completed
2025-02-07T10:46:28Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:84c8ace7e446971f0f3c421737984bfa719fe2fd30bac698c6a353cfbf00b1a7(the 1th running)...
2025-02-07T10:46:29Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:84c8ace7e446971f0f3c421737984bfa719fe2fd30bac698c6a353cfbf00b1a7 completed
2025-02-07T10:46:29Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:fa690c137bbecb52df371fc918aca760e835a93e8c8f46ba4f9ca642684336b7(the 1th running)...
2025-02-07T10:46:30Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:fa690c137bbecb52df371fc918aca760e835a93e8c8f46ba4f9ca642684336b7 completed
2025-02-07T10:46:30Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:76841a5479b672525de2c94e5301abb3de6f5489a2cf5d3f23232393dbfbfb22(the 1th running)...
2025-02-07T10:46:31Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:76841a5479b672525de2c94e5301abb3de6f5489a2cf5d3f23232393dbfbfb22 completed
2025-02-07T10:46:31Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:6b5650109a9c71db30d0fb0e9a1b7721c1e34fc6a3440f6cbe3e7cb889c6e4b2(the 1th running)...
2025-02-07T10:46:32Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:6b5650109a9c71db30d0fb0e9a1b7721c1e34fc6a3440f6cbe3e7cb889c6e4b2 completed
2025-02-07T10:46:32Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:6cdc9bb94d4abb57305b212e4b5bfd300fa24c16d8263598c7563c67521fdfaa(the 1th running)...
2025-02-07T10:46:34Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:6cdc9bb94d4abb57305b212e4b5bfd300fa24c16d8263598c7563c67521fdfaa completed
2025-02-07T10:46:34Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:48f90a15f7b9c2f326f4d0e58a679fb2a2795804985a68dccaab3dd46b71bd05(the 1th running)...
2025-02-07T10:46:35Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:48f90a15f7b9c2f326f4d0e58a679fb2a2795804985a68dccaab3dd46b71bd05 completed
2025-02-07T10:46:35Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact infrastructure/cipa:fb2b069bf317df7a6d69fcdb63db22206be4bd2f ...
2025-02-07T10:46:35Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact infrastructure/cipa:fb2b069bf317df7a6d69fcdb63db22206be4bd2f pushed
2025-02-07T10:46:35Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy infrastructure/cipa:fb2b069bf317df7a6d69fcdb63db22206be4bd2f(source registry) to infrastructure/cipa:fb2b069bf317df7a6d69fcdb63db22206be4bd2f(destination registry) completed
2025-02-07T10:46:35Z [INFO] [/controller/replication/transfer/image/transfer.go:204]: copy infrastructure/cipa:[fb2b069bf317df7a6d69fcdb63db22206be4bd2f](source registry) to infrastructure/cipa:[fb2b069bf317df7a6d69fcdb63db22206be4bd2f](destination registry) completed

By comparison, here is the start of the scheduled trigger (with the same result for a manual trigger):

2025-02-07T19:26:51Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: harbor, URL: [http://harbor-core:80](http://harbor-core/), insecure: true] created
2025-02-07T19:26:51Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: harbor, URL: https://backup.service.com/, insecure: false] created
2025-02-07T19:26:51Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying infrastructure/cipa:[20250207,latest,fb2b069bf317df7a6d69fcdb63db22206be4bd2f,20250206,d182bfb39aa08d0bf698505ceed16d310f702bcc](source registry) to infrastructure/cipa:[20250207,latest,fb2b069bf317df7a6d69fcdb63db22206be4bd2f,20250206,d182bfb39aa08d0bf698505ceed16d310f702bcc](destination registry)...

As you can see, here we have all the tags identified correctly, compared to the individual tag with the event-based one.

Image

I had this discussion thread in the CNCF Slack: https://cloud-native.slack.com/archives/CC1E09J6S/p1738925400265139

Steps to reproduce the problem:

  1. Create two Harbor instances (although the same issue occurs with other replication targets)
  2. Create a project in the "source" Harbor instance and one in the "target" instance with the same name.
  3. Create a replication job between "source" and "target" instances, set to "push based", trigger mode "Event based" and specifying project/** as the "Source resource filter", Matching: ** as the "Tag" in the Source resource filter.
  4. Create a Docker image, tagged with at least two tags, e.g. docker build -t harbor.example.org/replication/repo:latest -t harbor.example.org/replication/repo:abc123 .
  5. Push the tagged image to your source harbor instance docker push harbor.example.org/replication/repo:latest && docker push harbor.example.org/replication/repo:abc123
  6. Note that only the last tag pushed (abc123) will be pushed to the target instance.
  7. Create a second replication job, still push based, but this time set to manual and trigger the replication.
  8. Note that both tags are replicated. Remove the pushed image, force GC.
  9. Create a third replication job which is schedule based, wait for the replication schedule to trigger.
  10. Note that both tags are replicated.

Versions:

  • harbor version: [goharbor/harbor-core:2.11.1]
  • kubernetes: [1.18.20 but also 1.31.4 presents the same issue]
  • buildah: [1.28.2 (image-spec 1.1.0-rc2, runtime-spec 1.0.2-dev)]
  • podman: [4.3.1]
  • buildctl: [github.com/moby/buildkit v0.19.0 3637d1b15a13fc3cdd0c16fcf3be0845ae68f53d]
  • docker: [4.3.1]
@ianseyer
Copy link

Seeing this exact behavior on 2.12

@wy65701436
Copy link
Contributor

I will have a reproduce, and get back to you guys, thanks.

@wy65701436 wy65701436 added kind/bug target/2.13.0 issues that are targeting v2.13.0 labels Feb 17, 2025
@riyas-rawther
Copy link

We are also having the same issue.

@ianseyer
Copy link

ianseyer commented Feb 19, 2025 •
edited
Loading

Interestingly, I have also found that this behavior holds true for webhooks. For push events, only one event is emitted even when multiple tags are pushed for the same digest.

edit: here is a sketch of a PR to address this issue: https://github.com/ianseyer/harbor/pull/3/files

It appears that tag.Ensure simply does not fire an event.

can someone who knows more about the notificaton/tag event infra tell me if this reasonable?

@ianseyer ianseyer linked a pull request Feb 21, 2025 that will close this issue
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wy65701436 wy65701436

@chlins chlins

Labels
area/replication kind/bug needs/follow-up target/2.13.0 issues that are targeting v2.13.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Iseyer/replicate tags ianseyer/harbor
5 participants
@JonTheNiceGuy @wy65701436 @ianseyer @chlins @riyas-rawther