Could not create network lnst_container_net_net1: 404 Client Error: Not Found (network not found)

[konguyen]

[root@wsfd-netdev20 lnst]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.6 (Ootpa)
[root@wsfd-netdev20 lnst]# uname -srm
Linux 4.18.0-372.9.1.el8.x86_64 x86_64

[root@wsfd-netdev20 lnst]# systemctl status podman.socket
● podman.socket - Podman API Socket
Loaded: loaded (/usr/lib/systemd/system/podman.socket; enabled; vendor preset: disabled)
Active: active (listening) since Tue 2022-05-24 10:46:42 EDT; 1 weeks 0 days ago
Docs: man:podman-system-service(1)
Listen: /run/podman/podman.sock (Stream)
Tasks: 0 (limit: 822526)
Memory: 0B
CGroup: /system.slice/podman.socket

May 24 10:46:42 wsfd-netdev20.ntdv.lab.eng.bos.redhat.com systemd[1]: Listening on Podman API Socket.

[root@wsfd-netdev20 lnst]# poetry run python3 runner.py
Loading config file '/root/.lnst/lnst-ctl.conf'
Loading config file '/root/lnst/lnst-ctl.conf'
2022-05-31 08:27:55 (localhost) - DEBUG: Connecting to Podman API
2022-05-31 08:27:55 (localhost) - DEBUG: Starting new HTTP connection (1): %2frun%2fpodman%2fpodman.sock:80
2022-05-31 08:27:55 (localhost) - DEBUG: http://%2frun%2fpodman%2fpodman.sock:80 "GET /v4.0.0/libpod/info HTTP/1.1" 200 None
2022-05-31 08:27:55 (localhost) - DEBUG: http://%2frun%2fpodman%2fpodman.sock:80 "GET /v4.0.0/libpod/images/lnst/exists HTTP/1.1" 204 0
2022-05-31 08:27:55 (localhost) - INFO: Creating container machine1
2022-05-31 08:27:55 (localhost) - DEBUG: http://%2frun%2fpodman%2fpodman.sock:80 "POST /v4.0.0/libpod/containers/create HTTP/1.1" 201 88
2022-05-31 08:27:55 (localhost) - DEBUG: http://%2frun%2fpodman%2fpodman.sock:80 "GET /v4.0.0/libpod/containers/9a858872fec0e22178e4d0cce2cc565bdda8687a8fcb509d1868410abc2c1549/json HTTP/1.1" 200 None
2022-05-31 08:27:55 (localhost) - DEBUG: Starting container sharp_pasteur
2022-05-31 08:27:55 (localhost) - DEBUG: http://%2frun%2fpodman%2fpodman.sock:80 "POST /v4.0.0/libpod/containers/9a858872fec0e22178e4d0cce2cc565bdda8687a8fcb509d1868410abc2c1549/start HTTP/1.1" 204 0
2022-05-31 08:27:55 (localhost) - DEBUG: http://%2frun%2fpodman%2fpodman.sock:80 "GET /v4.0.0/libpod/containers/9a858872fec0e22178e4d0cce2cc565bdda8687a8fcb509d1868410abc2c1549/json HTTP/1.1" 200 None
2022-05-31 08:27:55 (localhost) - DEBUG: http://%2frun%2fpodman%2fpodman.sock:80 "POST /v4.0.0/libpod/containers/9a858872fec0e22178e4d0cce2cc565bdda8687a8fcb509d1868410abc2c1549/wait?condition=running HTTP/1.1" 200 3
2022-05-31 08:27:55 (localhost) - DEBUG: Checking connection with machine 10.88.0.6
2022-05-31 08:27:55 (localhost) - DEBUG: Connecting to 10.88.0.6, retry counter: 0
2022-05-31 08:27:56 (localhost) - DEBUG: Connecting to 10.88.0.6, retry counter: 1
2022-05-31 08:27:57 (localhost) - DEBUG: Connecting to 10.88.0.6, retry counter: 2
2022-05-31 08:27:57 (localhost) - DEBUG: Connected to agent process at machine 10.88.0.6
2022-05-31 08:27:57 (localhost) - INFO: Agent process is running at 10.88.0.6
2022-05-31 08:27:57 (localhost) - DEBUG: Connecting sharp_pasteur to net1
2022-05-31 08:27:57 (localhost) - INFO: Creating network lnst_container_net_net1
2022-05-31 08:27:57 (localhost) - DEBUG: http://%2frun%2fpodman%2fpodman.sock:80 "GET /v4.0.0/libpod/networks/lnst_container_net_net1 HTTP/1.1" 404 75
Traceback (most recent call last):
File "/root/lnst/lnst/Controller/ContainerPoolManager.py", line 227, in _create_network
network = self._podman_client.networks.get(name)
File "/root/.cache/pypoetry/virtualenvs/lnst-MdZhQDa5-py3.9/lib/python3.9/site-packages/podman/domain/networks_manager.py", line 114, in get
response.raise_for_status()
File "/root/.cache/pypoetry/virtualenvs/lnst-MdZhQDa5-py3.9/lib/python3.9/site-packages/podman/api/client.py", line 64, in raise_for_status
raise not_found(cause, response=self._response, explanation=message)
podman.errors.exceptions.NotFound: 404 Client Error: Not Found (network not found)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/lnst/runner.py", line 34, in
ctl.run(recipe_instance)
File "/root/lnst/lnst/Controller/Controller.py", line 162, in run
for match in self._mapper.matches(**kwargs):
File "/root/lnst/lnst/Controller/MachineMapper.py", line 377, in matches
self._pool_manager.process_reqs(self._mreqs)
File "/root/lnst/lnst/Controller/ContainerPoolManager.py", line 290, in process_reqs
self._connect_to_networks(container, m_reqs)
File "/root/lnst/lnst/Controller/ContainerPoolManager.py", line 280, in _connect_to_networks
network = self._create_network(name)
File "/root/lnst/lnst/Controller/ContainerPoolManager.py", line 229, in _create_network
raise PoolManagerError(f"Could not create network {name}: {e}")
lnst.Controller.AgentPoolManager.PoolManagerError: Could not create network lnst_container_net_net1: 404 Client Error: Not Found (network not found)

[jtluka]

Could you please upload the runner.py, or simply paste the file content here?

[konguyen]

[konguyen@konguyen lnst]$ cat runner.py
from lnst.Controller import Controller, HostReq, DeviceReq, BaseRecipe
from lnst.Controller.ContainerPoolManager import ContainerPoolManager
from lnst.Controller.MachineMapper import ContainerMapper

class HelloWorldRecipe(BaseRecipe):
machine1 = HostReq()
machine1.nic1 = DeviceReq(label="net1")

machine2 = HostReq()
machine2.nic1 = DeviceReq(label="net1")

def test(self):
    self.matched.machine1.nic1.ip_add("192.168.1.1/24")
    self.matched.machine1.nic1.up()
    self.matched.machine2.nic1.ip_add("192.168.1.2/24")
    self.matched.machine2.nic1.up()

    self.matched.machine1.run("ping 192.168.1.2 -c 5")
    self.matched.machine2.run("ping 192.168.1.1 -c 5")

podman_uri = "unix:///run/podman/podman.sock"
image_name = "lnst"
ctl = Controller(
poolMgr=ContainerPoolManager,
mapper=ContainerMapper,
podman_uri=podman_uri,
image=image_name,
debug=1,
)

recipe_instance = HelloWorldRecipe()
ctl.run(recipe_instance)

overall_result = all([run.overall_result for run in recipe_instance.runs])

exit(not overall_result)

[jtluka]

I tried to reproduce this locally on Fedora 34 but I did not hit the issue.

I did:

git pull # to fetch the latest changes which could potentially solve the issue

poetry install
poetry install -E containers # to be able to use LNST with containers, this is a recent upstream change

poetry shell
sudo python3 ./runner.py

I will check on RHEL.

[enhaut]

@konguyen what version of podman do you use - podman --version?

[konguyen]

podman --version
podman version 4.0.2

[konguyen]

[root@wsfd-netdev20 lnst]# poetry update
Updating dependencies
Resolving dependencies... (19.9s)

Writing lock file

Package operations: 0 installs, 9 updates, 0 removals

• Updating pyroute2.core (0.6.9 -> 0.6.12)
• Updating pyroute2.ethtool (0.6.9 -> 0.6.12)
• Updating pyroute2.ipdb (0.6.9 -> 0.6.12)
• Updating pyroute2.ipset (0.6.9 -> 0.6.12)
• Updating pyroute2.ndb (0.6.9 -> 0.6.12)
• Updating pyroute2.nftables (0.6.9 -> 0.6.12)
• Updating pyroute2.nslink (0.6.9 -> 0.6.12)
• Updating lxml (4.8.0 -> 4.9.0)
• Updating pyroute2 (0.6.9 -> 0.6.12)
[root@wsfd-netdev20 lnst]# poetry install
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: lnst (16.1.0)
[root@wsfd-netdev20 lnst]# poetry install -E containers
Installing dependencies from lock file

Package operations: 0 installs, 2 updates, 0 removals

• Updating pyxdg (0.27 -> 0.28)
• Updating requests (2.27.1 -> 2.28.0)

Installing the current project: lnst (16.1.0)
[root@wsfd-netdev20 lnst]# poetry shell
Spawning shell within /root/.cache/pypoetry/virtualenvs/lnst-MdZhQDa5-py3.9
[root@wsfd-netdev20 lnst]# . /root/.cache/pypoetry/virtualenvs/lnst-MdZhQDa5-py3.9/bin/activate
(lnst-MdZhQDa5-py3.9) [root@wsfd-netdev20 lnst]# sudo python3 ./runner.py
Traceback (most recent call last):
File "./runner.py", line 1, in
from lnst.Controller import Controller, HostReq, DeviceReq, BaseRecipe
File "/root/lnst/lnst/Controller/init.py", line 8, in
from lnst.Controller.Controller import Controller
File "/root/lnst/lnst/Controller/Controller.py", line 17, in
from lnst.Common.NetUtils import MacPool
File "/root/lnst/lnst/Common/NetUtils.py", line 16, in
from pyroute2 import IPRoute
ModuleNotFoundError: No module named 'pyroute2'

[enhaut]

Thanks @konguyen, i found an issue.
Podman has changed it's default networking backed to Netavark - https://www.redhat.com/sysadmin/podman-new-network-stack . We use CNI for networking so once LNST create CNI network manually, podman API tries to find it as a netavark network and it fails - throws 404.

Here is how to configure podman4.0+ to use cni:

1. Uninstall podman - dnf remove podman
2. remove all files from /etc/cni/net.d/
3. Install it again dnf install podman
4. Set network backend to cni -
   add

[network]
network_backend="cni"

to containers.conf - https://github.com/containers/common/blob/main/docs/containers.conf.5.md#description
5. install cni plugins - dnf install containernetworking-plugins
6. start podman api - systemctl enable podman.socket
7. now you should be able to run the containers, just tested at fedora 36

The log from RH infra seems OK, you just need to run it using just python instead of python3.

---

Just note, please paste large blocks of tracebacks/code to ``` quotes for better formatting.

[olichtne]

so this sounds like we should maybe start thinking about migrating to the netavark/aardvark instead of CNI

AND/OR we should include or link to the configuration steps you described in the docs?

[enhaut]

I just tested netavark real quick and it seems it also does not support multiple nics of containers connected to the same network. Maybe we can instead migrate to technology that supports that as you mentioned in #225 .
Until then i'll add a configuration steps to docs.

[enhaut]

I've added CNI configuration steps to #248 , so i think, we can close this issue

[konguyen]

Samuel - I just tried your steps but still hit the same issue. Do I need to copy the containers.conf into /etc/cni/net.d/?