LinkLocalIPs not honored (with docker-compose)

[entrope]

Issue Description

When running docker-compose up with a compose.yaml that contains a link_local_ips directive for one of the services, the additional IP(s) are not assigned to that service's container.

This seems to be because pkg/api/handlers/compat/containers_create.go translates the IPv4Address and IPv6Address from endpoint.IPAMConfig but not LinkLocalIPs.

Steps to reproduce the issue

Steps to reproduce the issue

1. git clone -b link_local_ips https://github.com/entrope/scratch.git
2. cd scratch && podman compose up
3. Run podman exec -it simple-boss-1 /bin/sh and within that ip addr.

This is a reduced test case; the fuller original repository is at https://github.com/entrope/testnet.git .

Describe the results you received

/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if459: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 76:58:7d:c8:74:8b brd ff:ff:ff:ff:ff:ff
    inet 10.11.12.2/24 brd 10.11.12.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::7458:7dff:fec8:748b/64 scope link 
       valid_lft forever preferred_lft forever

Describe the results you expected

I expected a line for 10.11.12.5 similar to the line for 10.11.12.2.

podman info output

host:
  arch: arm64
  buildahVersion: 1.35.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc39.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.65
    systemPercent: 0.13
    userPercent: 0.22
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "39"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.7.7-200.fc39.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 13204197376
  memTotal: 15566413824
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.20240312103946045036.main.18.g8377c0a.fc39.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.11.0-dev
    package: netavark-1.10.1-1.20240319111419242122.main.46.gcc3f35d.fc39.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.11.0-dev
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.20240302220834691516.main.10.g64ee22c.fc39.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: feb70bc2ab11944a6443e4f5d2eb96a22f186b80
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc39.aarch64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc39.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  rootlessNetworkCmd: ""
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 115h 46m 24.00s (Approximately 4.79 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 99252940800
  graphRootUsed: 6317051904
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 52
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.0.0-dev-8a643c243
  Built: 1710720000
  BuiltTime: Sun Mar 17 20:00:00 2024
  GitCommit: ""
  GoVersion: go1.21.8
  Os: linux
  OsArch: linux/arm64
  Version: 5.0.0-dev-8a643c243

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

The above podman info is from an installation of Podman Desktop v1.15.0 on macOS Sequoia 15.2 with docker-compose v2.32.1. I would expect similar results on a native Podman install, although my x86-64 Linux system (running Debian testing) fails when running podman compose up:

mdpoole@sanosuke:src/scratch% podman compose up
>>>> Executing external compose provider "/home/mdpoole/bin/docker-compose". Please see podman-compose(1) for how to disable this message. <<<<

[+] Running 2/2
 ✔ Network simple_inner     Created                                                                                                                                       0.0s 
 ✔ Container simple-boss-1  Created                                                                                                                                       0.4s 
Attaching to boss-1
Gracefully stopping... (press Ctrl+C again to force)
Error response from daemon: IPAM error: requested ip address 10.11.12.2 is already allocated to container ID b35487165917ae368d8f53a9aebd1529308f845f1556c6c12771991fceaece7c
Error: executing /home/mdpoole/bin/docker-compose up: exit status 1

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

We do not support setting static link local addresses at all. So this isn't really an API problem as netavark would first need to support that.

What is the use case for static link local addresses?

[entrope]

My use case does not require specifically link-local addresses. I'm writing tests for an Internet Relay Chat (IRC) server that (in some cases) need many clients to connect and interact in a coordinated way, and IRC servers normally limit how many clients can connect from a single IP address. I could instantiate a lot of proxies on separate containers, but that is extra work and overhead compared to running them from a single container.

At least for rootless mode, "root" in a container cannot use ip addr add to assign additional IPs to the virtual network interface, so that is not a workaround.

The handling in containers_create.go just adds the static addresses to netOpts.StaticIPs -- there is nothing about them being link-local or not. Conversely, the Compose spec only has link_local_ips as an analogous feature; there is no explicit way to specify more than one static non-link-local IP per container.

[Luap99]

At least for rootless mode, "root" in a container cannot use ip addr add to assign additional IPs to the virtual network interface, so that is not a workaround.

you need to add CAP_NET_ADMIN then you can do that

The handling in containers_create.go just adds the static addresses to netOpts.StaticIPs -- there is nothing about them being link-local or not. Conversely, the Compose spec only has link_local_ips as an analogous feature; there is no explicit way to specify more than one static non-link-local IP per container.

Then it still will not do what you want, one ip per network subnet is pretty much enforced everywhere. If you want more ips specify more subnets when you create the network.

$ podman network create --subnet 10.0.0.0/24 --subnet 10.0.1.0/24 --subnet 10.0.2.0/24 testnet
testnet
$ podman run --rm --network testnet quay.io/libpod/testimage:20241011 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host proto kernel_lo 
       valid_lft forever preferred_lft forever
2: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 96:92:2e:c1:fd:3d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.0.2/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.1.2/24 brd 10.0.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.2.2/24 brd 10.0.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::9492:2eff:fec1:fd3d/64 scope link tentative proto kernel_ll 
       valid_lft forever preferred_lft forever

The fact that link local is not link-local makes zero sense to me. Using that like this seems like a bug and not a feature.
Are there any docker docs that actually explain the purpose of that option? I cannot find any.

[entrope]

Thanks, adding CAP_NET_ADMIN allowed me to do what I wanted to do.

I looked before filing this report, but I was unable to find any documentation (or even forum posts on places like Stack Exchange) about the intended application or function of link_local_ips beyond what is in the Compose spec. So if you want to close this on the basis of a workaround being available, and the "correct" behavior being both unclear and unspecified, I have no objection.