Documentation relating to Directory Structure & File Systems

[pulpbot]

Author: wibbit (wibbit)

Redmine Issue: 8297, https://pulp.plan.io/issues/8297

---

Good morning all.

TLDR: Should the documentation be updated to reference DEPLOY_ROOT, and recommend that it is on a single partition, further ammendments to MEDIA_ROOT, WORKING_DIRECTORY, CHUNKED_UPLOAD_DIR, STATIC_ROOT or FILE_UPLOAD_TEMP_DIR are limited/restrictred to where it's actually pertinent and detailed information as to why you might choose to make that change with associated risks and caveats supplied where appropriate for that deployment style.

And the rest...

Re-reading the installation instructions and settings for Pulp3, I think there is a hole in relation to recommended partition layout, and associated settings.

We appear to have 3 documented variables (suggesting, that these are things we want to consider setting)

MEDIA_ROOT '/var/lib/pulp/media'
WORKING_DIRECTORY PosixPath('/var/lib/pulp/tmp')
CHUNKED_UPLOAD_DIR 'upload'

Being the curious fellow that I am, looking through dynaconf I also find the following variables.

DEPLOY_ROOT PosixPath('/var/lib/pulp')
STATIC_ROOT PosixPath('/var/lib/pulp/assets')
FILE_UPLOAD_TEMP_DIR PosixPath('/var/lib/pulp/tmp')

This appears to indicate the following layout.

/var/lib/pulp
/var/lib/pulp/assets
/var/lib/pulp/media/
/var/lib/pulp/media/??? (artifacts? 1TB on my current test install)
/var/lib/pulp/media/upload
/var/lib/pulp/tmp

The below note recommends MEDIA_ROOT and WORKING_DIRECTORY share the same filesystem.

https://docs.pulpproject.org/pulpcore/settings.html#working-directory

The only other "documentation" that I can find is https://pulp.plan.io/issues/7178

This suggests that there is an expectation that at a minimum DEPLOY_ROOT is a single partition, sized appropriately to store all of pulp's content.

I'm purprosefully ignoring the question of whether CHUNKED_UPLOAD, WORKING_DIRECTORY or FILE_UPLOAD_TEMP_DIR would not be better suited to /var/spool or /var/cache

[pulpbot]

From: ekohl (ekohl)
Date: 2021-02-23T14:52:01Z

---

For myself I started a reference in our Puppet module: https://github.com/theforeman/puppet-pulpcore#installation-layout. Since Pulp 3.10 this is also the default layout. Perhaps that documentation should be copied upstream.

In #799 (comment) I've argued that DEPLOY_ROOT should not be documented right now. That's because it's an internal variable and dynaconf doesn't allow you to override it today. If you do, MEDIA_ROOT will not be updated. Documenting it could give a false impression that it can be changed.

STATIC_ROOT is something that does not need to live on the same filesystem. It is essentially a cache of all assets shipped by Pulp and its plugins. That should be documented but generally not something users change.

FILE_UPLOAD_TEMP_DIR is a built in Django setting and due to how Pulp works, it is indeed recommended to have on the same filesystem as MEDIA_ROOT. See https://docs.djangoproject.com/en/2.2/ref/settings/#file-upload-temp-dir for more information. In #7178 I implemented a check that should verify the directories are on the same filesystem.

With that check, running pulpcore-manager check --deploy should warn you. The problem is that there's a lot of warnings. In theforeman/puppet-pulpcore#155 (comment) I started to analyze the ones I saw in our module. IMHO these should be silenced if they're irrelevant or solved if they are relevant. All of that should be in the default config rather than in some installer. Once that's done, running checks should become something any installer does and also a step in troubleshooting issues.

As a final note, if you do use SELinux then you should also take care of the right labels. By default https://github.com/pulp/pulpcore-selinux/blob/master/pulpcore.fc takes care of this.

[pulpbot]

From: @mdellweg (mdellweg)
Date: 2021-04-27T13:34:31Z

---

There's a list at the end of this docs page that might be needed updating here:

https://github.com/pulp/pulpcore/blob/master/docs/components.rst

[stale]

This issue has been marked 'stale' due to lack of recent activity. If there is no further activity, the issue will be closed in another 30 days. Thank you for your contribution!