[AZURE][PRE SIGNED URL] Support Azure blob (SAS) pre signed URL as for S3 and SWIFT

[Kilz78]

Is your feature request related to a problem? Please describe.
I just saw on a harbor configuration with azure blob storage that during docker push and docker pull operations, all layers will go though harbor to be stored by Harbor itself to the blob container.

In a S3 configuration, it's the docker client that will stored itself docker layers to the S3 bucket thanks to pre signed URLs.

Describe the solution you'd like
Support blob sas pre signed URL to allow docker client to push and pull directly from the blob storage

Describe the main design/architecture of your solution
It seems blob support already this approach with SAS (shared access signature) token or maybe something else ?

Is it possible and/or plan to add this feature to Harbor ?

Access to harbor instances are limited by a proxy max size : docker push operation are blocked by this

Describe the development plan you've considered
Support blob sas pre signed URL to allow docker client to push and pull directly from the blob storage

Additional context
I have a proxy in front of all my harbor instances, and those proxies are configured with a proxy max size header. Also those proxies are well dimensionned to support web traffic as to access to harbor UI but a big weakness for big network traffic as during docker pull push operations.

WHY
WHY ?

All networks traffics going though Harbor can cause latencies
Object storage infra are better dimensionned to store fastly (big) datas
Better to have same configuration between aws/azure
Managing access trafic with storage flow is difficult : how to limit it ? in term of volumetry, rbac ....
Proxy max size limiting/blocking the size of push operations

[Vad1mo]

This is out of the scope and capabilities of Harbor. pre signed URL only work on pull not on push.
The push request needs to go via Harobr core; otherwise we will not be able to track what is pushed, and we need to track it.

There are likely options to work around that (partially via harbor and partially direct so bucket), but it would also likely require changes on upstream distribution, and it would be quite some effort ~100-500h

Its more economic to update your proxy setting

[Kilz78]

Hello @Vad1mo thanks for your quick answer.

Hm, are you sure about this ? because i tested to push an image from my network to my AWS harbor instance, and i was not able to do because i can't resolve the S3 backend from my network. How can we explicate this error ?

`$ docker push my_instance/my_project/blackbox-exporter:v0.24.0

The push refers to repository [my_instance/my_project/blackbox-exporter]

db1e9faccd5a: Pushing [==================================================>] 3.584kB

672ab1aa1d73: Pushing [==================================================>] 21.17MB/21.17MB

272a705ae6fe: Pushing [==================================================>] 2.602MB

37fd1cc2fad4: Pushing [==================================================>] 1.47MB

dial tcp: lookup s3-my_instace-my_env.s3.eu-west-1.amazonaws.com on my_dns_ip:53: no such host`

However, It works well with Azure, and I can't resolve too my blob endpoint.

Also, with azure I can pull without resolving the blob storage, is that means pulling operation are going though the harbor core also?
-> for pull operation is something plan to use pre signed url with azure blob ?

[Vad1mo]

If it works for S3 and not Azure, then it is related to https://github.com/distribution/distribution. Take a look there regarding Azure and pre signed URLs.

If its not supported there we don't support it.