Poetry module not found by systemd services

[ChrisThibodeaux]

About accounts on capesandbox.com

• Issues isn't the way to ask for account activation. Ping capesandbox in Twitter with your username

This is open source and you are getting free support so be friendly!

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest version
• I did read the README!
• I checked the documentation and found no answer
• I checked to make sure that this issue has not already been filed
• I'm reporting the issue to the correct repository (for multi-repository projects)
• I have read and checked all configs (with all optional parts)

Expected Behavior

After full install, CAPE's systemd services should be active.

Current Behavior

Services all fail to activate.

Failure Information (for bugs)

Cape, cape-web, cape-rooter, and cape-processor services fail to activate. Errors indicate poetry not being found by python correctly.

Steps to Reproduce

1. Edit and run installation scripts.
2. Install dependencies with poetry install
3. Restart cape, cape-web, etc. services.
4. Check the systemctl status and/or journalctl -u <service> -f. Will display failure info.

Context

Installed on a bare metal server with Ubuntu 22.04.

Recent changes to the method of installing poetry in cape2.sh (curl -sSL https://install.python-poetry.org | POETRY_HOME=/etc/poetry python3 -) move where it is installed to /etc/poetry. No issue with installing that way or in that location because that is exactly how poetry's docs say it can be done.

I believe this new install location is not a location that python will check when attempting to load a module with python -m. The previous use of apt install python-poetry was likely installing at /usr/local/lib/python3.10/dist-packages or /usr/lib/python3/dist-packages. Directly running sudo -u cape poetry run python3 cuckoo.py from /opt/CAPEv2 works properly. Nearly positive this issue is isolated to the systemd service files.

PR with proposed fix here: #2407
I am not entirely sure about the ramifications of running the ExecStarts with /etc/poetry/bin/poetry run python ... instead of /usr/bin/python3 poetry run python ..., but the changes in this PR fix the problem for me.

Leaving it as a draft until anyone else can confirm that

1. A fresh install fails for them in the same way
2. Applying the Fix systemd services being unable to find the poetry module. #2407 changes fixes that

Failure Logs

From the journalctl for cape.service.

Nov 20 10:18:14 hairy-aviators systemd[1]: Started CAPE.
Nov 20 10:18:14 hairy-aviators python3[1272]: /usr/bin/python3: No module named poetry
Nov 20 10:18:14 hairy-aviators systemd[1]: cape.service: Main process exited, code=exited, status=1/FAILURE
Nov 20 10:18:14 hairy-aviators systemd[1]: cape.service: Failed with result 'exit-code'.
Nov 20 10:23:14 hairy-aviators systemd[1]: cape.service: Scheduled restart job, restart counter is at 1.
Nov 20 10:23:14 hairy-aviators systemd[1]: Stopped CAPE.

Note that poetry is available in the PATH variable.

root@hairy-aviators:/opt/CAPEv2# poetry
Poetry (version 1.8.4)
...

[poland4000]

I'm getting the same exact issue and the suggested PR lets me progress further. Still stuck afterwards due to the cape2.sh installer failing to find poetry anyways despite the fixed issue 2400.

[ChrisThibodeaux]

@poland4000 Can you share the portion of cape.log with the errors? My runs of sudo ./cape.sh base cape | tee cape.log don't seem to fail. This is the only part of the log I see anything about Poetry:

Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/etc/poetry/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.8.4)
Installing Poetry (1.8.4): Creating environment
Installing Poetry (1.8.4): Installing Poetry
Installing Poetry (1.8.4): Creating script
Installing Poetry (1.8.4): Done

Poetry (1.8.4) is installed now. Great!

To get started you need Poetry's bin directory (/etc/poetry/bin) in your `PATH`
environment variable.

Add `export PATH="/etc/poetry/bin:$PATH"` to your shell configuration file.

Alternatively, you can call Poetry explicitly with `/etc/poetry/bin/poetry`.

You can test that everything is set up by executing:

`poetry --version`

[poland4000]

I'd like to provide a log though I already 'fixed' it by including the poetry path into secure_path in visudo. Without this It would install just as in your logs, poetry installation would be successful however every step involving poetry e.g. "poetry --directory /opt/CAPEv2/ run pip install yara-x" would throw poetry command not found. I used "sudo ./cape2.sh all cape | tee cape.log" to install.

[ChrisThibodeaux]

So, with the base install that I ran, I did not hit those errors. I have a feeling poetry --directory /opt/CAPEv2/ run ... fails, but sudo -u ${USER} poetry --directory /opt/CAPEv2 run ... definitely does not. There is no place where the direct poetry --directory ... style is used when base arg is used.

May simply be that the spots where the first command is used need to be updated to prepend sudo -u ${USER} to them.

[czechmate247]

So I got around this by modifying the ExecStart=/usr/bin/python3 -m poetry ... part of each cape.*.service file with ExecStart=/etc/poetry/bin/poetry. The problem is that the cape2.sh script doesn't install poetry as a module to the base system's pip repo so the systemd service files ultimately can't find it. I haven't investigated which part of the cape2.sh install script creates the service files but it seems like if my changes above were put into the service file templates, all will work as intended. I don't know if this will have any unintended side effects but it is working for me so far.

[doomedraven]

Hey thanks for heads, I will fix systemd tomorrow, files are in systemd folder inside of CAPEv2, CAPEv2.sh just copies them El vie, 22 nov 2024, 23:30, czechmate247 ***@***.***> escribió:

…

So I got around this by modifying the ExecStart=/usr/bin/python3 -m poetry ... part of each cape.*.service file with ExecStart=/etc/poetry/bin/poetry. The problem is that the cape2.sh script doesn't install poetry as a module to the base system's pip repo so the systemd service files ultimately can't find it. I haven't investigated which part of the cape2.sh install script creates the service files but it seems like if my changes above were put into the service file templates, all will work as intended. I don't know if this will have any unintended side effects but it is working for me so far. — Reply to this email directly, view it on GitHub <#2408 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOFH36RS4HAQWDRIHQQQ7D2B6V7XAVCNFSM6AAAAABSIIFO4GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJUHE3TQNJYGY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[doomedraven]

fixed here a8c6b37

[ChrisThibodeaux]

@doomedraven From the fix commit in /systemd/cape-rooter.service:

ExecStartPre=/etc/poetry/bin/poetry -m poetry config cache-dir /opt/CAPEv2/.cache/pypoetry
ExecStart=/etc/poetry/bin/poetry -m poetry run python rooter.py -g cape

Should these two lines be this instead?

ExecStartPre=/etc/poetry/bin/poetry config cache-dir /opt/CAPEv2/.cache/pypoetry
ExecStart=/etc/poetry/bin/poetry run python rooter.py -g cape

[doomedraven]

ah yes good catch, thanks, fixing

[ChrisThibodeaux]

@doomedraven As always, thank you for all your work! I'll close out this issue and the PR I put up.

There may remain the problem others are reporting here with the cape2.sh script failing to find Poetry. I don't think that necessarily should be solved in this ticket, though. When I have more time, I will take a look there as well.

[doomedraven]

thanks for the headups, the poetry now is instaled to /etc/poetry so shouldn't be a problem anymore, let me know if you find any other problem