[PR #6064/0a5ac4aa backport][3.49] [SAT-29018] Fix/corrupted RA blocks content streaming

[pedro-psb]

The #6161 was not very good because it still added the field to the database.

This approaches adds a unique field to the branch which won't conflict with any future field.
It must be cleaned up later with an idempotent remove migration.

Still testing if it runs without a problem.

[pedro-psb]

My attempt to test the upgrade process:

1. Start on 3.49 with a clean install
2. checking out to main <-- errors [0]

I assume I'm missing some steps (update deps, ...?). Will continue on monday.

[0]:

[pulp]  | Traceback (most recent call last):
[pulp]  |   File "/usr/local/lib/python3.9/site-packages/gunicorn/arbiter.py", line 608, in spawn_worker
[pulp]  |     worker.init_process()
[pulp]  |   File "/src/pulpcore/pulpcore/app/entrypoint.py", line 47, in init_process
[pulp]  |     django.setup()
[pulp]  |   File "/usr/local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
[pulp]  |     apps.populate(settings.INSTALLED_APPS)
[pulp]  |   File "/usr/local/lib/python3.9/site-packages/django/apps/registry.py", line 124, in populate
[pulp]  |     app_config.ready()
[pulp]  |   File "/src/pulpcore/pulpcore/app/apps.py", line 258, in ready
[pulp]  |     super().ready()
[pulp]  |   File "/src/pulpcore/pulpcore/app/apps.py", line 134, in ready
[pulp]  |     self.import_viewsets()
[pulp]  |   File "/src/pulpcore/pulpcore/app/apps.py", line 171, in import_viewsets
[pulp]  |     from pulpcore.app.viewsets import NamedModelViewSet
[pulp]  |   File "/src/pulpcore/pulpcore/app/viewsets/__init__.py", line 1, in <module>
[pulp]  |     from .base import (
[pulp]  |   File "/src/pulpcore/pulpcore/app/viewsets/base.py", line 16, in <module>
[pulp]  |     from pulpcore.openapi import PulpAutoSchema
[pulp]  |   File "/src/pulpcore/pulpcore/openapi/__init__.py", line 13, in <module>
[pulp]  |     from drf_spectacular.plumbing import (
[pulp]  | ImportError: cannot import name 'process_webhooks' from 'drf_spectacular.plumbing' (/usr/local/lib/python3.9/site-packages/drf_spectacular/plumbing.py)

[pedro-psb]

The upgrade did work. Here were the steps:

1. Compose up oci_env with this branch code (based on 3.49 and with the failed_at_backport49 field)
2. Checkout to pulpcore main
3. run pip install src/pulpcore
4. run pulpcore-manager migrate

The results

[root@4a043b9b8239 /]# pulpcore-manager migrate
Operations to perform:
  Apply all migrations: auth, certguard, contenttypes, core, file, sessions
Running migrations:
  Applying core.0118_task_core_task_unblock_2276a4_idx_and_more... OK
  Applying core.0119_grouprole_core_groupr_object__250e22_idx_and_more... OK
  Applying core.0120_get_url_removal... OK
  Applying core.0121_add_profile_artifacts_table... OK
  Applying core.0122_record_last_replication_timestamp... OK
  Applying core.0123_upstreampulp_q_select... OK
  Applying core.0124_task_deferred_task_immediate... OK
  Applying core.0125_openpgpdistribution_openpgpkeyring_openpgppublickey_and_more... OK
  Applying core.0126_remoteartifact_failed_at... OK
Access policy for artifacts created.
Access policy for domains updated.
Access policy for repositories/core/openpgp_keyring created.
Access policy for task-groups created.
Access policy for tasks updated.
Access policy for login created.
[root@4a043b9b8239 /]# su pulp
bash-5.1$ psql
psql (13.16)
Type "help" for help.

pulp=> \d core_remoteartifact;
                        Table "public.core_remoteartifact"
        Column        |           Type           | Collation | Nullable | Default
----------------------+--------------------------+-----------+----------+---------
 pulp_id              | uuid                     |           | not null |
 pulp_created         | timestamp with time zone |           | not null |
 pulp_last_updated    | timestamp with time zone |           |          |
 url                  | text                     |           | not null |
 size                 | bigint                   |           |          |
 md5                  | character varying(32)    |           |          |
 sha1                 | character varying(40)    |           |          |
 sha224               | character varying(56)    |           |          |
 sha256               | character varying(64)    |           |          |
 sha384               | character varying(96)    |           |          |
 sha512               | character varying(128)   |           |          |
 content_artifact_id  | uuid                     |           | not null |
 remote_id            | uuid                     |           | not null |
 pulp_domain_id       | uuid                     |           | not null |
 failed_at_backport49 | timestamp with time zone |           |          |
 failed_at            | timestamp with time zone |           |          |

[mdellweg]

Now I'm wondering whether it'd be better to add the timedelta to setting the "failed_at" ("pulp_last_udated" in this case) and only check against now here.

[mdellweg]

Nice!

[pedro-psb]

Thanks for all the help to get this working!