Llama

[jeffnvidia]

Summary

Based on PR 20
enable to use Llama tests
create llama toml file and adapt NeMo gen command

There is a FIXME inside this PR (in the Llama.toml file):

FIXME : ~training.model.position_embedding_type was added in the extra_cmd_args in order to fix a bug from NeMo repository (https://github.com/NVIDIA/NeMo).
the commit that should fix this issue in NeMo is : 5b296e8af832c67d361fdfb80a165db3affaf76a.
Once the new release of NeMoLauncher includes this commit (check by downloading the corresponding container and look inside /opt for this commit), ~training.model.position_embedding_type should be removed from the extra args

Test Plan

Test by @amaslenn
CI
Test by @jeffnvidia
2.1 Slurm command generation
$ python ./cloudaix.py --mode run --system_config_path conf/v0.6/general/system/... --test_scenario_path conf/v0.6/general/test_scenario/llama/llama.toml

Additional Notes

Llama 16 nodes alone (3, 5, 8) : /auto/mtrsysgwork/jmahou/git/asap_cloudai/results/2024-05-01_12-17-26

cmd : $ python ./cloudaix.py --mode run --system_config_path conf/v0.6/general/system/... --test_scenario_path conf/v0.6/general/test_scenario/llama/llama.toml
Llama 16 nodes (3, 5, 8) + bisection noise 72 nodes : /auto/mtrsysgwork/jmahou/git/asap_cloudai/results/2024-05-01_13-42-47

cmd : $ python ./cloudaix.py --mode run --system_config_path conf/v0.6/general/system/... --test_scenario_path conf/v0.6/general/test_scenario/llama/llama_with_noise.toml