[Ingest] Batch processing logic for HOLD condition not correct

[mreyescdl]

Batch level HOLD is not acting correctly and requires additional effort once HOLD is released.

BatchConsumer daemon detects HOLD condition, but does not put Batch in "Held" state but leaves it in a processing state.
Once HOLD is released, the following manual ZK queue actions are required to release.

        Set STATE to pending (or Processing if in Reporting state)
        Remove batches/<bid>/states
        Remove batches/<bid>/lock

For example,

{"message":"","last_modified":"2025-01-07 15:34:17 -0800","status":"Processing"}
{"message":"","last_modified":"2025-01-07 15:34:17 -0800","status":"Pending"}

{"message":"","last_modified":"2025-01-08 11:55:27 -0800","status":"Reporting"}
{"message":"","last_modified":"2025-01-08 11:55:27 -0800","status":"Processing"}

Upon detection, Batch should be put in "Held" state.
Then, after release, the ADMIN UI will be needed to go from Held to Pending

[mreyescdl]

After discussing with Terry, we will do the following.

Existing behavior( which does not work well for patching cycle)

• BatchConsumer daemon acquires PENDING batches (changing state to PROCESSSING)
• Determines a HOLD is in place, so it goes into a waiting loop until HOLD is released.

Desired behavior, which works well for patching cycle

• BatchConsumer daemon will first determine if HOLD is in place.
• If yes, then no acquiring of PENDING jobs is performed. Loop until HOLD released.

[mreyescdl]

Looking closer at the code, we are doing the right thing. If Ingest queue is held, we go into a waiting pattern which includes NOT acquiring PENDING jobs.
I'm unsure why after patching that jobs were in a PROCESSING state and unable to continue.
I tested a HELD queue with a Object manifest and Batch manifest and it works as expected.

Lets keep this ticket open until the next patching cycle. Before queue is released we will determine which batches arrived while HELD and examine how they are processed when released.