Bug fix in job completion checks

[TaekyungHeo]

Summary

This is a bug fix for https://redmine.mellanox.com/issues/4248881. @Bohatchuk reported that Chakra replay fails to generate reports when pre-test hooks are enabled. This issue arises from how job completion is checked. Currently, the system checks for specific keywords such as "COMPLETED" in the output of sacct. However, when multiple tasks exist within a single job, the output may include a mix of "RUNNING" and "COMPLETED," as shown below:

RUNNING
COMPLETED

This scenario should be interpreted as the job still running. Otherwise, jobs may be prematurely marked as complete.

Test Plan

1. CI passes
2. Ran on a server (EOS)

$ python ./cloudaix.py --log-level DEBUG --log-file run_chakra_replay_3.log run --system-config 
conf/common/system/eos.toml --tests-dir conf/common/test --test-scenario  conf/devops/verification/test_scenario/chakra_replay.toml  
...                                                                 
[INFO] Test Scenario: chakra_replay                                                                                                                                    
                                                                                                                                                                       
Section Name: Tests.1                                                                                                                                                  
  Test Name: chakra_replay                                                                                                                                             
  Description: chakra_replay                                                                                                                                           
  No dependencies                                                                                                                                                      
[INFO] Initializing Runner [RUN] mode                                                                                                                                  
[INFO] Creating SlurmRunner                                                                                                                                            
[DEBUG] SlurmRunner initialized                                                                                                                                        
[INFO] Starting test scenario execution.                                                                                                                               
[INFO] Starting test: Tests.1                                                                                                                                          
[INFO] Running test: Tests.1                                                                                                                                           
[DEBUG] Executing command for test Tests.1: sbatch results/chakra_replay_2025-01-14_16-35-35/Tests.1/0/cloudai_sbatch_script.sh                                        
[INFO] Submitted slurm job: 1828634   

After job completion

$ ls $RESULTS
chakra_replay_report.html  cloudai_sbatch_script.sh  stderr.txt  stdout.txt

pre_test:
nccl_test_all_gather

Before pre-test hook completion

   RUNNING
   RUNNING
   RUNNING
   RUNNING
   RUNNING

After pre-test hook completion

   RUNNING
   RUNNING
   RUNNING
 COMPLETED
   RUNNING