Failed to push image when mirror and private repositry running in the same container

[hungrysnake]

The issue is that when I run both mirror (config.yml remoteurl: https://registry-1.docker.io") and private registry in the same contianer, the mirror part will work but the private reigstry part refused to let docker host to push images to it.

eg.
docker_host pull/push-----> registry:5002 for both mirror and priv_registry

however, everything works when those 2 functions are seperated.
eg.
docker_host pull-----> registry:5003 for mirror
docker_host push-----> registry:5004 for private registry

curl shows that the mirror is working and caching the image pull by other docker host

docker_host$ curl http://10.1.1.13:5002/v2/_catalog
{"repositories":["library/hello-world"]}

however, when other docker host is pushing to it, it got stuck in preparing > retry and loop forever

docker_host$ docker image tag hello-world 10.1.1.13:5002/my-hello-world
docker_host$$ docker push 10.1.1.13:5002/my-hello-world
The push refers to repository [10.1.1.13:5002/my-hello-world]
9c27e219663c: **Retrying** in 10 seconds

the logs from the registry showing that it denied the request. denied: requested access to the resource is denied

store_1                 | 10.1.1.14 - - [24/Jun/2020:04:21:44 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/5.4.0-29-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \\(linux\\))"
store_1                 | time="2020-06-24T04:21:44.980866629Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="10.1.1.13:5002" http.request.id=8472c60d-861f-41ba-bf17-f611ed143e8e http.request.method=GET http.request.remoteaddr="10.1.1.14:56230" http.request.uri="/v2/" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/5.4.0-29-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.007405ms http.response.status=200 http.response.written=2
store_1                 | 10.1.1.14 - - [24/Jun/2020:04:21:44 +0000] "HEAD /v2/my-hello-world/blobs/sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689 HTTP/1.1" 500 264 "" "docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/5.4.0-29-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \\(linux\\))"
store_1                 | time="2020-06-24T04:21:45.729920307Z" level=error msg="response completed with error" err.code=unknown err.detail="errors:
store_1                 | denied: requested access to the resource is denied
store_1                 | error parsing HTTP 401 response body: unexpected end of JSON input: ""
store_1                 | " err.message="unknown error" go.version=go1.11.2 http.request.host="10.1.1.13:5002" http.request.id=72279df1-f817-480c-95bd-60e47b8ef0f7 http.request.method=HEAD http.request.remoteaddr="10.1.1.14:56232" http.request.uri="/v2/my-hello-world/blobs/sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/5.4.0-29-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=735.642742ms http.response.status=500 http.response.written=264 vars.digest="sha256:0e03bdcc26d7a9a57ef3b6f1bf1a210cff6239bff7c8cac72435984032851689" vars.name=my-hello-world
store_1                 | time="2020-06-24T04:21:45.733986416Z" level=error msg="response completed with error" err.code=unsupported err.message="The operation is unsupported." go.version=go1.11.2 http.request.host="10.1.1.13:5002" http.request.id=cd4d833a-fe91-4780-8d88-535202497f54 http.request.method=POST http.request.remoteaddr="10.1.1.14:56234" http.request.uri="/v2/my-hello-world/blobs/uploads/" http.request.useragent="docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/5.4.0-29-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \(linux\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.481004ms http.response.status=405 http.response.written=78 vars.name=my-hello-world
store_1                 | 10.1.1.14 - - [24/Jun/2020:04:21:45 +0000] "POST /v2/my-hello-world/blobs/uploads/ HTTP/1.1" 405 78 "" "docker/19.03.12 go/go1.13.10 git-commit/48a66213fe kernel/5.4.0-29-generic os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.12 \\(linux\\))"

here is my config on both docker_host and registry

docker_host$ cat /etc/docker/daemon.json
{
  "registry-mirrors": ["http://10.1.1.13:5002"]
}

my registry docker-compose file

registry$ cat docker-compose.yml
store:
    image: registry
    ports:
      - "5002:5000"
    volumes:
      - type: bind
        source: ${PWD}/store/registry-data
        target: /var/lib/registry
    command:  /var/lib/registry/config.yml
    restart: always

[csawyerYumaed]

I believe the docker registry code does not allow a single registry instance to be both a docker cache and a private registry. We ended up running 2 instances and haven't had any trouble now.

1. private registry

2. docker hub cache

Maybe that will fix your issue also?