Code

fix diloco 2#13

Code

Merged

samsja merged 9 commits intomainPrimeIntellect-ai/prime:mainfrom fix-diloco-2PrimeIntellect-ai/prime:fix-diloco-2Copy head branch name to clipboardSep 27, 2024

+199-106

Commits

Commits on Sep 26, 2024

• only sync loss between local pg

  samsja

  committed
  edf0b02

  Copy full SHA for edf0b02

• refactor tests dist

  samsja

  committed
  3be4213

  Copy full SHA for 3be4213

• update docstring

  samsja

  committed
  df1616c

  Copy full SHA for df1616c

• refactor: use only global pg in diloco + first failing tests

  samsja

  committed
  294c36d

  Copy full SHA for 294c36d

• fix config to use shard grad op

  samsja

  committed
  ba392de

  Copy full SHA for ba392de

Commits on Sep 27, 2024

• do copy instead of assigment

  samsja

  committed
  a143cab

  Copy full SHA for a143cab

• init adamw inner before diloco

  samsja

  committed
  270bd95

  Copy full SHA for 270bd95

• make weight init same across diloco rank

  samsja

  committed
  b43bde7

  Copy full SHA for b43bde7

• remove the to in the copy because jack said so and the test pass

  samsja

  committed
  9d801d3

  Copy full SHA for 9d801d3