We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
| | GPT-NeoX | NVIDIA Megatron | DeepSpeed Megatron |---------------------|-----------|----------------| | model parallel | ? | ? | | data parallel | y | ? | | pipeline parallel | y | ? | | other optimizations | ZeRO | ? | | benchmarks | | |