[5.3.1] Once again unable to connect to host from container via host.containers.internal

[andreaippo]

Issue Description

This is what I get in my application (running in podman) when it tries to send an HTTP request to another application which is listening on host port 8080.

connection timed out after 3000 ms: host.containers.internal/169.254.1.2:8080

The URL I set on the first application to target the second is host.containers.internal:8080

Nothing is "answering" from 169.254.1.2, and yet if I curl localhost:8080 on the host, I get a reply

I have already struggled with this in the past, but applying a workaround worked fine before podman 5.3.0.

The workaround consisted in adding this snippet to ~/.config/containers/containers.conf:

[network]
pasta_options = ["-a", "10.0.2.0", "-n", "24", "-g", "10.0.2.2", "--dns-forward", "10.0.2.3"]

With the release of 5.3.0 I had tested that removing that snippet did no harm: the thing just worked.

Now I dunno if it's a regression of 5.3.1 or something with my distro (which I also reinstalled meanwhile), but now this kind of connectivity no longer works. What's worse, even the workaround doesn't seem to work anymore.

I even tried disabling the firewall on my host. I'm using SELinux:

 sestatus  
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33

Any ideas?

Thanks

Steps to reproduce the issue

run a rootless podman container sending HTTP requests to host.containers.internal:8080, while having a server listening on that host port.

Describe the results you received

no connectivity

Describe the results you expected

request is routed to localhost

podman info output

host:
  arch: amd64
  buildahVersion: 1.38.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-1.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: unknown'
  cpuUtilization:
    idlePercent: 90.08
    systemPercent: 1.69
    userPercent: 8.23
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: opensuse-tumbleweed
    version: "20250103"
  eventLogger: journald
  freeLocks: 1998
  hostname: andromeda
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.12.8-5.g4810a19-default
  linkmode: dynamic
  logDriver: journald
  memFree: 10700148736
  memTotal: 32917471232
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.13.1-1.1.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.1
    package: netavark-1.13.1-1.1.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.1
  ociRuntime:
    name: crun
    package: crun-1.19-1.1.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.19
      commit: db31c42ac46e20b5527f5339dcbf6f023fcd539c
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-20241211.09478d5-1.1.x86_64
    version: |
      pasta 20241211.09478d5-1.1
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 34359734272
  swapTotal: 34359734272
  uptime: 0h 6m 39.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /home/andrea/.config/containers/storage.conf
  containerStore:
    number: 9
    paused: 0
    running: 9
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/andrea/.local/share/containers/storage
  graphRootAllocated: 826623201280
  graphRootUsed: 49255763968
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 57
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/andrea/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.1
  Built: 1733485830
  BuiltTime: Fri Dec  6 12:50:30 2024
  GitCommit: ""
  GoVersion: go1.23.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[andreaippo]

Forgot to mention the container NetworkSettings:

"NetworkSettings": {
    "Bridge": "",
    "SandboxID": "",
    "SandboxKey": "/run/user/1000/netns/netns-1b53da85-9d88-ae87-e4ce-cafcac081d1c",
    "Ports": {
      "18090/tcp": [
        {
          "HostIp": "0.0.0.0",
          "HostPort": "18090"
        }
      ],
      "8090/tcp": [
        {
          "HostIp": "0.0.0.0",
          "HostPort": "8090"
        }
      ]
    },
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "",
    "Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "",
    "IPPrefixLen": 0,
    "IPv6Gateway": "",
    "MacAddress": "",
    "Networks": {
      "sip-infra-network": {
        "IPAMConfig": null,
        "Links": null,
        "Aliases": [
          "si-manager-service",
          "83ba816ebb3a"
        ],
        "MacAddress": "86:a3:4f:49:ea:82",
        "DriverOpts": null,
        "NetworkID": "sip-infra-network",
        "EndpointID": "",
        "Gateway": "10.89.0.1",
        "IPAddress": "10.89.0.9",
        "IPPrefixLen": 24,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "DNSNames": null
      }
    }
  }

[andreaippo]

Temporarily moved back to docker, where connectivity works fine using host.docker.internal

I think it uses slirp4netns instead, maybe that's why - no idea.

[Luap99]

Is your host application listening on localhost or 0.0.0.0?

[Luap99]

If 5.3.0 worked but 5.3.1 doesn't then this has most likely nothing to do with podman, the changes in there should have nothing to with any of this. The only relevant change in that code would be a39a749 but that only fixes a nil deref so that cannot cause such a change

[andreaippo]

Is your host application listening on localhost or 0.0.0.0?

Not sure yet, this is a spring boot app and I can't find the default value:

https://docs.spring.io/spring-boot/appendix/application-properties/index.html#application-properties.server.server.address

I guess I'll have to try myself: if I get a reply from my LAN address (e.g. 192.168.1.90:8080) then it's listening on 0.0.0.0, right? Otherwise only on localhost?

[andreaippo]

Is your host application listening on localhost or 0.0.0.0?

Not sure yet, this is a spring boot app and I can't find the default value:

https://docs.spring.io/spring-boot/appendix/application-properties/index.html#application-properties.server.server.address

I guess I'll have to try myself: if I get a reply from my LAN address (e.g. 192.168.1.90:8080) then it's listening on 0.0.0.0, right? Otherwise only on localhost?

Ok I guess I just confirmed that by default spring boot's web server (Tomcat, IIRC) binds to 0.0.0.0, because I was able to get responses either from localhost:8080 or 192.168.1.90:8080, which is the IP assigned to my wifi interface.

[Luap99]

You can get the listing address via ss -tulpn but yes if you can connect from another system it should be good.

I just tested it again here and it works fine for me. Can you check your processes for the pasta command line that was used (ps auxww | grep pasta)?

[andreaippo]

Well I really have no idea how or why, but now it works.

I have installed docker and docker-compose packages for my distro (which bring a bunch of deps, including slirp4netns), started my app with docker-compose instead of podman-compose (after changing host.container.internal to host.docker.internal), and everything worked.

Then I uninstalled everything again, leaving only podman and podman-compose (which is the situation I was in when opening this bug), reverted my app to use host.container.internal once again, and by some frigging magic it works now.

I'm honestly clueless. I'm almost tempted to rollback my OS to the previous state, although that would only apply to my root partition - so if this got somehow fixed by something left behind in my ~ by those docker/docker-compose installs, it becomes impossible to pinpoint what made this work.

But TBH I'm just happy I can move on - sorry for the time spent on this.

[andreaippo]

It happened again - I will check the pasta command used.
Also wondering if connecting/disconnecting to a VPN mid-flight might interfere with name resolution, or also DHCP lease renewal by my home router...

I'll allow myself to use this comment to take some notes :D

When it's working:

❯ ps auxww | grep pasta
andrea     24271  0.7  0.0  68624 19980 ?        Ss   09:32   0:18 /usr/bin/pasta --config-net --pid /run/user/1000/containers/networks/rootless-netns/rootless-netns-conn.pid --dns-forward 169.254.1.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/1000/containers/networks/rootless-netns/rootless-netns --map-guest-addr 169.254.1.2
andrea    104669  0.0  0.0   6648  4232 pts/3    S+   10:13   0:00 grep --color=auto pasta

Now I have to wait for things to stop working again

[Luap99]

Oh if you host ip changes that might make a difference.
@sbrivio-rh @dgibson Is the address for --map-guest-addr only set on pasta startup so that it is then expected that this will fail to map once the ip on the host changes? I guess that is something where the future netlink monitor work could help.

[sbrivio-rh]

@sbrivio-rh @dgibson Is the address for --map-guest-addr only set on pasta startup so that it is then expected that this will fail to map once the ip on the host changes?

It took me a bit, so I guess we should document --map-guest-addr better (now on my list). Yes, we get the source address for the mapping when we start, and then use it as it was, even if addresses on the host change.

I guess that is something where the future netlink monitor work could help.

Right, that's another use case we didn't think about (and probably the only reasonable "fix" for this). For context: https://pad.passt.top/p/PastaWithoutNetworkConnectivity (section 2, even if it's unrelated to this specific issue).

[andreaippo]

I don't have a reproducer at the moment, "hopefully" next week it will stop working again

Wondering if this should be reopened, or if you want to wait for my feedback

Thanks

[sbrivio-rh]

I don't have a reproducer at the moment, "hopefully" next week it will stop working again

Well, I guess you should simply keep an eye on address changes on the host. Most likely, like @Luap99 suggested, things are breaking once the host address that pasta initially picked changes.

Let's say that you start with a 192.2.0.1 address on the host. By default (without -a), when pasta starts, it sources a host address and uses it for the container, so it's also 192.2.0.1.

Podman configures pasta to change the destination address for outbound connections directed to 169.254.1.2 (inside the container) to 192.2.0.1 on the host. The source address doesn't change: it's 192.2.0.1 as well (both inside and outside).

Now that address changes to 192.2.0.2 (because the VPN connection is re-established or because your DHCP lease is renewed with a different address), but pasta still maps packets to 192.2.0.1 as destination address, which means that they can't be used by the host.

As a workaround, you can assign a fixed address to the container, as you were doing.

By the way:

I think it uses slirp4netns instead, maybe that's why - no idea.

Docker can use both, but it's still using slirp4netns by default, see https://docs.docker.com/engine/security/rootless/#networking-errors.

Wondering if this should be reopened, or if you want to wait for my feedback

If @Luap99 wants to keep this issue open as a reference for "we should change the mapping with a netlink monitor", perhaps we can reopen this.

[andreaippo]

Ok, I just reproduced this when a change in IP address occurs due to (forced) DHCP lease expiration.

To reproduce:

• check that connectivity works from a container to a host service
• go to your DHCP server or router and define a static DHCP lease for your host, with a different IP address from the current one; disconnect your host and reconnect (or force a new DHCP lease request)
• check connectivity again: it'll be broken
• go back to your DHCP server or router and revert the change so that your host gets the IP address it had at the beginning; again disconnect your host and reconnect (or force a new DHCP lease request)
• check connectivity again: it'll be restored

What would be the workaround here? Apart from a static lease, which won't always be possible (imagine a corporate environment where you don't control the router).
Is it possible to just restart some systemd service, at least on Linux? I guess a reboot should also do it, but it's much more drastic.

Thanks

[Luap99]

Yes let's reopen this one.

However I like to add this is a general problem not pasta specific. Even slirp4netns or root networking do the same thing. Podman sources one valid host ip on start and puts the as host.containers.internal. If that ip address is removed from the host it will no longer work for all network modes.

In fact the pasta --map-guest-addr would be the only network mode where we can fix this actually via the netlink monitor as the inside address 169.254.1.2 stays static so no podman changes required.
For the other network modes podman is not a daemon and as such has no way to monitor or update /etc/hosts at a later point.

As such I am actually surprised that this is a new issue for you. I guess you may have multiple ips on the host and podman picks the first one which doesn't change for you.

[sbrivio-rh]

What would be the workaround here?

#24970 (comment)

[andreaippo]

Yes let's reopen this one.

However I like to add this is a general problem not pasta specific. Even slirp4netns or root networking do the same thing. Podman sources one valid host ip on start and puts the as host.containers.internal. If that ip address is removed from the host it will no longer work for all network modes.

In fact the pasta --map-guest-addr would be the only network mode where we can fix this actually via the netlink monitor as the inside address 169.254.1.2 stays static so no podman changes required. For the other network modes podman is not a daemon and as such has no way to monitor or update /etc/hosts at a later point.

As such I am actually surprised that this is a new issue for you. I guess you may have multiple ips on the host and podman picks the first one which doesn't change for you.

Yeah it's a bit confusing for me, too.

As for docker, I guess it just happened to work because I installed it after the DHCP lease renewal, and so it was able to start clean, which is something podman couldn't cause it was already running, that's my best guess. But at this point, even on docker it's just working out of chance.

Now why I never faced this issue for weeks (i.e. since 5.3 came out) and then it suddenly started happening a few days ago, is an even bigger mystery to me

[dgibson]

@sbrivio-rh @dgibson Is the address for --map-guest-addr only set on pasta startup so that it is then expected that this will fail to map once the ip on the host changes?

It took me a bit, so I guess we should document --map-guest-addr better (now on my list). Yes, we get the source address for the mapping when we start, and then use it as it was, even if addresses on the host change.

[Sorry, replacing my original answer, because I realised I'd misread the question]

"Source address" is unclear here. Do you mean

1. The address which is translated when used by the guest - the actual argument to --map-guest-addr. Yes that's set just at start up and could not be otherwise - it's a literal address given on the command line.
2. The address to which that is translated when the flow is forwarded to the host? This is always the guest's assigned address (via DHCP or netlink or whatever), whether or not the guest actually decides to use that. Currently that is only set at startup, but if we implemented a netlink monitor that would no longer be the case; --map-guest-addrs behaviour would change accordingly.
3. The source address that appears on host-side packets created when a guest-side connection is remapped with --map-guest-addr. This doesn't differ from the source address for any host-side connection. If --outbound is used, it's set at startup. Otherwise it is picked by the kernel at the time we create the connection, and so will change if the host address changes.

@sbrivio-rh, the two code links you give don't relate to each other the way you seem to imply. The first link will parse the address we use or (1), the second is setting (3) based the information in the flow table. The flow table in turn is populated based on --outbound here. If that code isn't triggered it will be :: causing tcp_bind_outbound() to skip the bind entirely, hence leaving the choice of (3) up to the kernel.

[sbrivio-rh]

"Source address" is unclear here. Do you mean

[...]

2. The address to which that is translated when the flow is forwarded to the host?  This is always the guest's _assigned address_ (via DHCP or netlink or whatever), whether or not the guest actually decides to use that.  Currently that is only set at startup, but if we implemented a netlink monitor that would no longer be the case; `--map-guest-addr`s behaviour would change accordingly.

This (kind of). The destination address that's used on the host for outbound packets that are forwarded to the host. Once the host doesn't have that address anymore, mapping will fail.

@sbrivio-rh, the two code links you give don't relate to each other the way you seem to imply. The first link will parse the address we use or (1)

Right, I missed one step, and I can't quickly find the exact code locations right now, but in any case the problem seems to be clear at this point.

[dgibson]

"Source address" is unclear here. Do you mean
[...]

2. The address to which that is translated when the flow is forwarded to the host?  This is always the guest's _assigned address_ (via DHCP or netlink or whatever), whether or not the guest actually decides to use that.  Currently that is only set at startup, but if we implemented a netlink monitor that would no longer be the case; `--map-guest-addr`s behaviour would change accordingly.

This (kind of). The destination address that's used on the host for outbound packets that are forwarded to the host. Once the host doesn't have that address anymore, mapping will fail.

The mapping works just fine, it's just it's destination has gone away. This is essentially no different than if you accessed some random host with an untranslated address, say, 1.2.3.4. If the machine at 1.2.3.4 changes address to 5.6.7.8 then accessing it at 1.2.3.4 won't work any more.

This actually suggests to me that we might want yet another mapping option: one which always goes to an external address of the host. Unlike --map-guest-addr that doesn't allow connectivity that was actually impossible before: if the host has a different address than the guest, you can just access it with that address. However, I can see that it would be useful to assign a stable address with that meaning within the guest's network.

[dgibson]

Actually I just realised that it might be possible to implement --map-host-external or whatever we want to call it, without a netlink monitor. If on the host side we SO_BINDTODEVICE to an extrnal interface, then connect() to 0.0.0.0 the resulting connection will be given the current interface address by the kernel (at least according to ss).

As @sbrivio-rh suggests you can explicitly assign the guest's address as a workaround. Note, however, that the mechanism by which this works is a bit indirect: it causes podman to set the host.containers.internal to the real host address, rather than a translated one. To be safe you must give the guest an address which the host will never have. If you explicitly assign it the (at startup) current host address, you will have exactly the same problem as with --map-guest-addr (in fact I expect the runtime state to be precisely equivalent). If you explicitly assign an address that the host doesn't have now but might in future, it will work initially, but will break (in a slightly different way) when/if the host takes the same address that the guest is assigned.

At the passt level, another workaround would be to use --map-host-loopback instead of --map-guest-addr. That means that mapped connections will connect to the host via its loopback interface, instead of an external address, so it won't change. However, podman explicitly disables that since as a matter of policy it doesn't want the host loopback interface exposed to the container.

Which makes me think... I'm not totally sure how this is working with slirp4netns. By default slirp maps 10.0.2.2 to the host loopback - essentially equivalent to --map-host-loopback 10.0.2.2 in pasta. However as with pasta, podman explicitly disables that (--disable-host-loopback). So if it's using the host's external IP for host.containers.internal then I'd expect it to hit the same problem as with pasta: when the host changes address, that no longer points to the right place. I wonder if docker does the same: maybe the difference is not so much that it's using slirp as that it's using slirp without --disable-host-loopback.