Safety valve to seperate the DSE execution with benchmarking execution #334
Conversation
| def update_nested_attr(obj, attr_path, value): | ||
| """Update a nested attribute of an object.""" | ||
| attrs = attr_path.split(".") | ||
| # hot fix. Will be removed after the issue is fixed in the codebase |
There was a problem hiding this comment.
Which issue are you referring to in this line?
There was a problem hiding this comment.
This was the discussion regarding why I wanted name field to the TestDefinition. The way we have even in several generation strategies today is hardcoded to Grok to update various flags. For now it's in handlers. But will be moved inside the environment. We can just query the name field for each test definition wherever there is nested structure.
|
@srivatsankrishnan Please fix the CI issues when you get a chance. |
| iteration_path = self.output_path / f"iteration_{tr.dse_iteration}" | ||
| iteration_path.mkdir(parents=True, exist_ok=True) | ||
| test_output_path = iteration_path / f"{datetime.now().strftime('%Y-%m-%d_%H-%M-%S')}" / tr.name | ||
| test_output_path.mkdir(parents=True, exist_ok=True) |
There was a problem hiding this comment.
I do not agree with the proposed directory structure. We need to discuss this with all core members: Srinivas, Srivatsan, and Andrei.
My proposal
- For non-DSE jobs -> Keep the current directory structure.
results
|--test_scenario.name
|--Test.1
|---0
|--sbatch
|--run.sh
- For DSE jobs -> Follow Srivatsan's proposal, but switch the levels between iteration and test scenario name.
results
|--test_scenario.name
|--iteration_1
|--Test.1
|---0
|--sbatch
|--run.sh
|--iteration_2
|--Test.1
|---0
|--sbatch
|--run.sh
The CI errors comes from the acceptance unit tests depends upon the final directory structure we agree on. |
Summary
This PR separates the DSE execution with benchmarking execution by identifying the job type (
DSEor Non-DSE (i.e., benchmarking) by dynamically parsing the TestRun object. If it's a DSE job, the agent and environment integration happens along with iterators. If it's a regular job, then it will not instantiate the agent or the environment.Input
Output
Test Plan
CI/CD
Additional Notes