Garbage Collection Stuck in Pending status

[mdavid01]

jobservice.zip

• We have 2 harbor environments running under Broadcom's TKG platform and harbor helm. One runs in AWS Gov West and one AWS east. AWS Gov environment has 16,000+ repos and 90TB of s3 storage. AWS East has 12,000+ repos and 90TB of s3 storage. Both environs run 5 Job Service pods.

• AWS Gov garbage collection runs as expected daily and takes approximately 2 hrs; a check on candidate GC items shows 16 candidate objects

• AWS East GC never leaves Pending status; candidate GC items shows 190,000+ candidate objects

• In both cases, Issue 15445 was basis for identifying candidates:

SELECT b.id, b.digest, b.content_type, b.status, b.version, b.size FROM blob AS b LEFT JOIN project_blob pb ON b.id = pb.blob_id WHERE pb.id IS NULL AND b.update_time <= now() - interval '2 hours';

Expected behavior and actual behavior:
Expect GC to run

Steps to reproduce the problem:

1. Stop scheduled but never-ending GC
2. Set worker to 1
3. Run GC now
4. Run kubectl logs $pod -n harbor-prod -f > ($pod + ".txt") on all 5 job service pods to find any GC Job incoming entry (none are found)

Versions:
Please specify the versions of following systems.

• harbor version: [Version v2.11.1-6b7ecba1]
• docker engine version: [TBD but likely latest version -1 (26)]
• docker-compose version: [TBD but likely latest version -1 (2.32 or 2.31)]

Additional context:

• Nowhere in the job service log files is there a 'job incoming' for Garbage Collection
• Tried deleting all execution and task jobs and restarting GC; GC did not start.
• Questions:

1. Is GC automatically set to 'pending' upon clicking GC now button (ref: tracker.go, line 42, "// Pending is default status when creating job, so no need to switch")
2. Are all the "task with job ID ... not found" errors in the attached job service log a related problem?
3. Is 20741 a strong possible root cause and would upgrade to 2.12 fix the problem?```
   wy65701436 on Jul 15, 2024
   Let me clarify, it is not the an issue of GC. And the problem will be fixed in the next harbor patch releases.
   PR: fix: update the execution sweep sql to resolve the exec deletion conflict #20603

4. is [issue 21416](https://github.com/goharbor/harbor/issues/21416) a strong possibility for root cause?
5. What conditions are required for a pending task to be set to running?
6. What module contains the code that checks the pending status and conditions necessary to release the job?