Stuck pushing my image to minikube

[sebastian-philipp]

I'm using a Mac and trying to get things working. According to https://minikube.sigs.k8s.io/docs/handbook/pushing/ there are 8 different ways to push images to minkube:

1. Pushing directly to the in-cluster Docker daemon (docker-env)

Doesn't work:

LANG=C minikube docker-env

❌  Exiting due to MK_USAGE: the docker-env command only supports the docker and containerd runtimes

2. Push images using ‘cache’ command.

LANG=C minikube cache add --logtostderr  localhost/controller:latest
I0108 14:50:14.926410   63099 out.go:345] Setting OutFile to fd 1 ...
I0108 14:50:14.927600   63099 out.go:397] isatty.IsTerminal(1) = true
I0108 14:50:14.927603   63099 out.go:358] Setting ErrFile to fd 2...
I0108 14:50:14.927606   63099 out.go:397] isatty.IsTerminal(2) = true
I0108 14:50:14.927739   63099 root.go:338] Updating PATH: /Users/sebastian/.minikube/bin
W0108 14:50:14.931146   63099 out.go:270] ❗  "minikube cache" will be deprecated in upcoming versions, please switch to "minikube image load"
❗  "minikube cache" will be deprecated in upcoming versions, please switch to "minikube image load"
I0108 14:50:14.932374   63099 config.go:182] Loaded profile config "minikube": Driver=podman, ContainerRuntime=crio, KubernetesVersion=v1.31.0
I0108 14:50:14.932411   63099 cache.go:107] acquiring lock: {Name:mk864d3b01e4c88e8ff8e38c301da9c82de3ea33 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0108 14:50:14.932945   63099 image.go:135] retrieving image: localhost/controller:latest
I0108 14:50:14.936875   63099 image.go:174] found index.docker.io/localhost/controller:latest locally: &{ref:{Repository:{Registry:{insecure:false registry:index.docker.io} repository:localhost/controller} tag:latest original:localhost/controller:latest} opener:0x14000826000 tarballImage:<nil> computed:false id:0x140003f4220 configFile:<nil> once:{done:{_:{} v:0} m:{state:0 sema:0}} err:<nil>}
I0108 14:50:14.936889   63099 cache.go:162] opening:  /Users/sebastian/.minikube/cache/images/arm64/localhost/controller_latest
I0108 14:50:14.939341   63099 cache.go:96] cache image "localhost/controller:latest" -> "/Users/sebastian/.minikube/cache/images/arm64/localhost/controller_latest" took 6.936209ms
I0108 14:50:14.939559   63099 out.go:201] 

W0108 14:50:14.939578   63099 out.go:270] ❌  Exiting due to MK_CACHE_LOAD: Failed to cache and load images: save to dir: caching images: caching image "/Users/sebastian/.minikube/cache/images/arm64/localhost/controller_latest": write: unable to calculate manifest: Error response from daemon: docker.io/localhost/controller:latest: image not known

Seems as if minikube prepends a wrong index.docker.io/to the image tag.

3. Pushing directly to in-cluster CRI-O. (podman-env)

See kubernetes/minikube#15036 (comment)

4. Pushing to an in-cluster using Registry addon

This would be a mess due to required renaming of the image. Might be something as a last resort, but not yet willing to do that.

5. Building images inside of minikube using SSH

docker@minikube:~$ podman build .
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available 
WARN[0000] For using systemd, you may need to login using an user session 
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root) 
WARN[0000] Falling back to --cgroup-manager=cgroupfs    
Error: command required for rootless mode with multiple IDs: exec: "newuidmap": executable file not found in $PATH

6. Pushing directly to in-cluster containerd (buildkitd)

Only works with ContainerD

7. Loading directly to in-cluster container runtime

LANG=C minikube image load --logtostderr localhost/controller:latest
I0108 14:57:30.399930   64167 out.go:345] Setting OutFile to fd 1 ...
I0108 14:57:30.401042   64167 out.go:397] isatty.IsTerminal(1) = true
I0108 14:57:30.401045   64167 out.go:358] Setting ErrFile to fd 2...
I0108 14:57:30.401049   64167 out.go:397] isatty.IsTerminal(2) = true
I0108 14:57:30.401168   64167 root.go:338] Updating PATH: /Users/sebastian/.minikube/bin
I0108 14:57:30.405122   64167 config.go:182] Loaded profile config "minikube": Driver=podman, ContainerRuntime=crio, KubernetesVersion=v1.31.0
I0108 14:57:30.405177   64167 cache.go:107] acquiring lock: {Name:mk864d3b01e4c88e8ff8e38c301da9c82de3ea33 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0108 14:57:30.405949   64167 image.go:135] retrieving image: localhost/controller:latest
I0108 14:57:30.410139   64167 image.go:174] found index.docker.io/localhost/controller:latest locally: &{ref:{Repository:{Registry:{insecure:false registry:index.docker.io} repository:localhost/controller} tag:latest original:localhost/controller:latest} opener:0x140007a7490 tarballImage:<nil> computed:false id:0x14000ca6080 configFile:<nil> once:{done:{_:{} v:0} m:{state:0 sema:0}} err:<nil>}
I0108 14:57:30.410155   64167 cache.go:162] opening:  /Users/sebastian/.minikube/cache/images/arm64/localhost/controller_latest
I0108 14:57:30.411706   64167 cache.go:96] cache image "localhost/controller:latest" -> "/Users/sebastian/.minikube/cache/images/arm64/localhost/controller_latest" took 6.540083ms
I0108 14:57:30.411923   64167 out.go:201] 

W0108 14:57:30.411942   64167 out.go:270] ❌  Exiting due to GUEST_IMAGE_LOAD: Failed to load image: save to dir: caching images: caching image "/Users/sebastian/.minikube/cache/images/arm64/localhost/controller_latest": write: unable to calculate manifest: Error response from daemon: docker.io/localhost/controller:latest: image not known
❌  Exiting due to GUEST_IMAGE_LOAD: Failed to load image: save to dir: caching images: caching image "/Users/sebastian/.minikube/cache/images/arm64/localhost/controller_latest": write: unable to calculate manifest: Error response from daemon: docker.io/localhost/controller:latest: image not known
W0108 14:57:30.411946   64167 out.go:270] 

W0108 14:57:30.413207   64167 out.go:293] ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                                                        │
│    😿  If the above advice does not help, please let us know:                                                          │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                                                        │
│                                                                                                                        │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.                                 │
│    Please also attach the following file to the GitHub issue:                                                          │
│    - /var/folders/ml/4lvq5psn6813v431js9sd7k80000gs/T/minikube_image_4ff35744b649b4ab6eba2e8821bbe6f1472fe93f_0.log    │
│                                                                                                                        │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Again, seems as if minikube prepends a wrong index.docker.io/to the image tag.

8. Building images to in-cluster container runtime

minikube image build -t my_image .
[1/2] STEP 1/11: FROM golang:1.22 AS builder
[2/2] STEP 1/5: FROM gcr.io/distroless/static:nonroot
time="2025-01-08T13:41:28Z" level=warning msg="missing \"TARGETOS\" build argument. Try adding \"--build-arg TARGETOS=<VALUE>\" to the command line"
time="2025-01-08T13:41:28Z" level=warning msg="missing \"TARGETARCH\" build argument. Try adding \"--build-arg TARGETARCH=<VALUE>\" to the command line"
Trying to pull gcr.io/distroless/static:nonroot...
Error: error creating build container: short-name "golang:1.22" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

Can'f find any documentation for this: https://www.google.com/search?q=%22minikube+image+build%22+%22TARGETOS%22

Essentially I'm stuck now.

[benoitf]

Hello, did you try the command from Podman Desktop ?

[sebastian-philipp]

Hello, did you try the command from Podman Desktop ?

Intersting, this works!

root@minikube:/# podman images   
REPOSITORY                               TAG                 IMAGE ID      CREATED        SIZE
localhost/controller                     latest              8e9b841d919d  6 hours ago    71.3 MB

Do you do something different?

[benoitf]

it saves the image to a file and then load the image from this file

https://github.com/podman-desktop/extension-minikube/blob/main/src/image-handler.ts

[sebastian-philipp]

Nope. This is not really working:

 Warning  Failed         51s (x12 over 3m)  kubelet            Error: ImageInspectError
  Warning  InspectFailed  39s (x13 over 3m)  kubelet            Failed to inspect image "controller:latest": rpc error: code = Unknown desc = short-name "controller:latest" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

As if you're bypassing the usual workflows and runtimes.

Edit: Yes, prepending localhost/ to the pod's container image, avoid s the ImageInspectError. Seems to be related to the issues I faces with the broken docker.io prefix in my original post.

Edit: I suspect some unintended interactions between podman adding a localhost/ prefix, and neither minikube nor minikube's runtime can properly handle this prefix.

[benoitf]

I would avoid to use untagged / :latest tag for the images, because if you deploy such images on a Kubernetes cluster, the default policy for this images will be to pull first an image
Unless you specify the IfNotPresent policy (which might be your case ?)

https://kubernetes.io/docs/concepts/containers/images/#imagepullpolicy-defaulting

so try to build the image giving them a tag name

[sebastian-philipp]

I would avoid to use untagged / :latest tag for the images

I agree. Was probably the easiest way for kubebuilder to come up with something: https://github.com/kubernetes-sigs/kubebuilder/blob/458615a6d5c20c93f82fcf60767d132c1e6d4e7c/pkg/plugins/common/kustomize/v2/scaffolds/init.go#L33

[sebastian-philipp]

Seems I found a way:

minikube machine ssh
sed -i 's!"localhost/controller:latest"!"controller:latest"!g' /var/lib/containers/storage/overlay-images/images.json
^D
LANG=C minikube image load --logtostderr controller:latest

will check tomorrow, if that breaks anything

[sebastian-philipp]

yeah, not working. the dreaded "localhost" comes back as soon as I push it to minikube.

root@minikube:/var/lib/containers# crictl images | grep controller
localhost/controller                      latest               c2c3ed5934912       75.9MB

[afbjorklund]

@sebastian-philipp unfortunately the cri-o configuration got broken in a later version of minikube:

• CRIO: add docker.io to the unqualified-search-registries kubernetes/minikube#19396

Kubernetes is supposed to be able to find images that are under the default registry (i.e. docker.io)

The fake registry ("localhost") is supposed to be silent, but you can also use yourownregistry.com

[afbjorklund]

Pushing directly to the in-cluster Docker daemon (docker-env)
the docker-env command only supports the docker and containerd runtimes

The docker-env should be able to handle cri-o (podman) also, but originally it used podman-env

With a little imagination, it should be possible to translate the CONTAINER_HOST into a DOCKER_HOST

---

minikube podman-env

export CONTAINER_HOST="ssh://[email protected]:32772/run/podman/podman.sock"
export CONTAINER_SSHKEY="/home/anders/.minikube/machines/minikube/id_rsa"
export MINIKUBE_ACTIVE_PODMAN="minikube"

would translate into
export DOCKER_HOST="ssh://[email protected]:32772"
ssh-add /home/anders/.minikube/machines/minikube/id_rsa
minikube ssh sudo ln -s /run/podman/podman.sock /var/run/docker.sock

DOCKER_BUILDKIT=0 docker

---

EDIT: Added a PR for minikube to do it automatically:

• Add support for using podman with docker-env kubernetes/minikube#20260

[afbjorklund]

5. Building images inside of minikube using SSH

Note that you have to use sudo podman build

---

You could also set up a podman system connection, to have it connect to the system socket instead of the user socket.

podman system connection add default --default unix:///run/podman/podman.sock (also need to use podman --remote)

[afbjorklund]

Final remark is that it is not minikube that is adding docker.io as the default search registry, it is actually Kubernetes...

The recommendation is to either use docker.io for your own builds, or invent your own registry (whether it exists or not)

[sebastian-philipp]

Ok, I got stuck with minikube and tried Kind and got a very similar problem:

➜  cloud-extensions git:(sw/tcp-services-cm) ✗ k describe pod
...
Events:
  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  10s              default-scheduler  Successfully assigned cloud-extensions-system/cloud-extensions-controller-manager-7c9c4688b5-qh6p9 to kind-cluster-control-plane
  Normal   Pulled     9s (x2 over 9s)  kubelet            Container image "localhost/controller:latest" already present on machine
  Warning  Failed     9s (x2 over 9s)  kubelet            Error: failed to get image from containerd "sha256:ad8e0380560012ed1279961633945ee7ba18b846978ffd789bb6518e26b97e69": image "docker.io/library/controller:latest": not found

Seems as if this use case is in general not intended to work at all!

[afbjorklund]

The same issue exists with containerd and buildkit, shortnames are not configurable (hardcoded)

That's why they add the docker.io/library to the images by default, if you don't specify a registry.