Code

405b more#552

Code

Closed

fduwjj wants to merge 250 commits intomainpytorch/torchtitan:mainfrom 405b_morepytorch/torchtitan:405b_moreCopy head branch name to clipboard

+45,968-336

Commits

Commits on Feb 13, 2024

• check in tokenizer.model for ease of dev setup (#59)

  Show description for 054f088
  wanchaol

  authored
  054f088

  Copy full SHA for 054f088

Commits on Feb 14, 2024

• Add truncated llama style model init via reset parameters() (#54)

  Show description for bfe2b58
  lessw2020

  authored
  bfe2b58

  Copy full SHA for bfe2b58

Commits on Feb 15, 2024

• add model num params display, gpu memory metrics (#56)

  Show description for 60f021a
  lessw2020

  authored
  60f021a

  Copy full SHA for 60f021a

• add TensorBoard logging with loss and wps

  Show description for ad69e62
  tianyu-l

  committed
  ad69e62

  Copy full SHA for ad69e62

Commits on Feb 17, 2024

• add memory metrics to TensorBoard

  Show description for a4663b1
  tianyu-l

  committed
  a4663b1

  Copy full SHA for a4663b1

Commits on Feb 21, 2024

• modify data split to use HF api

  Show description for 2daf53f
  tianyu-l

  committed
  2daf53f

  Copy full SHA for 2daf53f

Commits on Feb 22, 2024

• add multinode support via slurm trainer, large scale race condition fix (#63)

  Show description for 50d69f6
  lessw2020

  authored
  50d69f6

  Copy full SHA for 50d69f6

• add configurable unique layer init, clean up lr and loss display (#64)

  Show description for 8ad4dcb
  lessw2020

  authored
  8ad4dcb

  Copy full SHA for 8ad4dcb

Commits on Feb 23, 2024

• add bunch of cleanups and design principle section (#71)

  wanchaol

  authored
  8097c26

  Copy full SHA for 8097c26

• delete the linter to see if re-adding it helps (#80)

  wconstab

  authored
  28f431f

  Copy full SHA for 28f431f

Commits on Feb 24, 2024

• Unified config manager for toml and command line (#76)

  Show description for ebbb1cb
  

  gnadathur

  and

  gnadathur

  authored
  ebbb1cb

  Copy full SHA for ebbb1cb

• Whc/add linter (#81)

  Show description for bccad90
  wconstab

  authored
  bccad90

  Copy full SHA for bccad90

• Add 4GPU unit test (#82)

  Show description for ab75dbd
  wconstab

  authored
  ab75dbd

  Copy full SHA for ab75dbd

• update readme (#74)

  Show description for 468ce8f
  wanchaol

  authored
  468ce8f

  Copy full SHA for 468ce8f

• move config folder to root and adjust options (#83)

  Show description for 3fce6bb
  wanchaol

  authored
  3fce6bb

  Copy full SHA for 3fce6bb

Commits on Feb 26, 2024

• add iter time tracking via cuda events, add data loading times, add columnar display to show both, show avg iter & data loading times at end of training (#87)

  Show description for 3b48039
  lessw2020

  authored
  3b48039

  Copy full SHA for 3b48039

• Fill missing options in toml file wih argparse defaults (#91)

  Show description for df77f4e
  

  gnadathur

  and

  gnadathur

  authored
  df77f4e

  Copy full SHA for df77f4e

Commits on Feb 27, 2024

• support infinite loop over alpaca dataset

  Show description for 325951f
  tianyu-l

  committed
  325951f

  Copy full SHA for 325951f

• Add color to console output if local logging, auto avoid color logging on slurm (#93)

  Show description for b12b6dd
  lessw2020

  authored
  b12b6dd

  Copy full SHA for b12b6dd

• update GPU metrics logging to GiB (gibibytes) (#95)

  Show description for 254279f
  lessw2020

  authored
  254279f

  Copy full SHA for 254279f

• improve TensorBoard instructions in README

  Show description for 4c03475
  tianyu-l

  committed
  4c03475

  Copy full SHA for 4c03475

Commits on Feb 28, 2024

• Enable libUV for torchtrain (#98)

  Show description for 7ea0679
  

  gnadathur

  and

  gnadathur

  authored
  7ea0679

  Copy full SHA for 7ea0679

Commits on Feb 29, 2024

• use warmup steps for lr scheduler, ban steps == -1 (#99)

  Show description for e60c573
  wanchaol

  authored
  e60c573

  Copy full SHA for e60c573

• Add llama 7B config (#100)

  Show description for 900b215
  wanchaol

  authored
  900b215

  Copy full SHA for 900b215

• add selective activation checkpointing

  Show description for 6e87471
  tianyu-l

  committed
  6e87471

  Copy full SHA for 6e87471

Commits on Mar 1, 2024

• Add job description field in toml (#101)

  Show description for 1b343f2
  

  gnadathur

  and

  gnadathur

  authored
  1b343f2

  Copy full SHA for 1b343f2

Commits on Mar 2, 2024

• fix 2D parallel crash caused by all-reduce on 2D world_mesh

  Show description for 42f8907
  tianyu-l

  committed
  42f8907

  Copy full SHA for 42f8907

Commits on Mar 5, 2024

• Load missing keys default from argparse (#111)

  Show description for 4042b05
  

  gnadathur

  and

  gnadathur

  authored
  4042b05

  Copy full SHA for 4042b05

• Add meta_init, enable it as default init process (#84)

  Show description for 6529af1
  lessw2020

  authored
  6529af1

  Copy full SHA for 6529af1

• Fix feedback from PR 111 (#113)

  Show description for 5f0eaea
  

  gnadathur

  and

  gnadathur

  authored
  5f0eaea

  Copy full SHA for 5f0eaea

Commits on Mar 6, 2024

• fix SP minor issues

  Show description for 1ce8188
  tianyu-l

  committed
  1ce8188

  Copy full SHA for 1ce8188

• enable loss parallel in SP

  Show description for bb5c4c6
  tianyu-l

  committed
  bb5c4c6

  Copy full SHA for bb5c4c6

• Float8_experimental option for training (#102)

  drisspg

  authored
  f31adb0

  Copy full SHA for f31adb0

Commits on Mar 7, 2024

• add miniPile dataset for pretraining, 1M entries (solves the 'out of data' at 40 iters issue) (#88)

  Show description for 6927e45
  lessw2020

  authored
  6927e45

  Copy full SHA for 6927e45

• add data loading option to load from local file system

  Show description for d902a47
  tianyu-l

  committed
  d902a47

  Copy full SHA for d902a47

Commits on Mar 9, 2024

• add llama 13B configs

  Show description for 422910b
  wanchaol

  committed
  422910b

  Copy full SHA for 422910b

• add llama 70B toml

  Show description for af221ce
  wanchaol

  committed
  af221ce

  Copy full SHA for af221ce

• set betas and weight decay for optimizers

  Show description for 5e36c74
  wanchaol

  committed
  5e36c74

  Copy full SHA for 5e36c74

• Add c4 dataset (177M, streaming), update multi-node support for latest job configs (#124)

  Show description for 08b332c
  lessw2020

  authored
  08b332c

  Copy full SHA for 08b332c

Commits on Mar 12, 2024

• Add openwebtext dataset for larger scale training without shuffling (#130)

  Show description for 1d11cf5
  lessw2020

  authored
  1d11cf5

  Copy full SHA for 1d11cf5

• [TorchTrain][Checkpoint] Fix TrainState state_dict to unblock loading (#131)

  Show description for 2722865
  wz337

  authored
  2722865

  Copy full SHA for 2722865

Commits on Mar 13, 2024

• improve logging

  Show description for 2369861
  tianyu-l

  committed
  2369861

  Copy full SHA for 2369861

• use SequenceParallel style in tp/sp (#133)

  Show description for 3262a8b
  wanchaol

  authored
  3262a8b

  Copy full SHA for 3262a8b

Commits on Mar 14, 2024

• support TP-only parallelism

  Show description for d9253ee
  tianyu-l

  committed
  d9253ee

  Copy full SHA for d9253ee

• disable verbose print from profiling

  Show description for b42ce91
  tianyu-l

  committed
  b42ce91

  Copy full SHA for b42ce91

• add Selective layer activation checkpointing, single control for turning AC on or off. (#125)

  Show description for 3ac610b
  lessw2020

  authored
  3ac610b

  Copy full SHA for 3ac610b

• remove per iter syncronize

  Show description for af56ae0
  tianyu-l

  committed
  af56ae0

  Copy full SHA for af56ae0

Commits on Mar 15, 2024

• Shorten nccl comm timeout and enable flight recorder dumping (#103)

  Show description for 073909b
  wconstab

  authored
  073909b

  Copy full SHA for 073909b

• fix up gpu memory monitoring and logging

  Show description for e3204c6
  tianyu-l

  committed
  e3204c6

  Copy full SHA for e3204c6

• Separate timeout during init and training (#149)

  Show description for a257bc3
  wconstab

  authored
  a257bc3

  Copy full SHA for a257bc3

Commits on Mar 20, 2024

• Update activation check with updates to config manager (#152)

  drisspg

  authored
  1d6100c

  Copy full SHA for 1d6100c

• Refactor to clean up parallelisms/__init__.py

  Show description for ae9a966
  wconstab

  committed
  ae9a966

  Copy full SHA for ae9a966

• enable gc control scheduling to help avoid stragglers (#148)

  Show description for 47bb509
  lessw2020

  authored
  47bb509

  Copy full SHA for 47bb509

• Add float8 specific parallel strategies (#153)

  drisspg

  authored
  fcca670

  Copy full SHA for fcca670

• add MFU to metrics

  Show description for 5d28009
  tianyu-l

  committed
  5d28009

  Copy full SHA for 5d28009

Commits on Mar 21, 2024

• disable buffer reuse for compile for now (#156)

  Show description for 35d881e
  wanchaol

  authored
  35d881e

  Copy full SHA for 35d881e

Commits on Mar 22, 2024

• refactor config manager and support cmd overrides (#157)

  Show description for f080027
  wanchaol

  authored
  f080027

  Copy full SHA for f080027

Commits on Mar 24, 2024

• Add support for generating debug traces on failure

  chauhang

  authored
  34732f5

  Copy full SHA for 34732f5

Commits on Mar 25, 2024

• rename sequence_parallel to tensor_parallel (#162)

  Show description for e008027
  wanchaol

  authored
  e008027

  Copy full SHA for e008027

Commits on Mar 27, 2024

• add basic AC configs for 13B and 70B (#169)

  Show description for 44808f9
  wanchaol

  authored
  44808f9

  Copy full SHA for 44808f9

• [TorchTrain][Checkpoint] Update train state to include global_avg_losses and global_max_losses (#167)

  Show description for bb61af0
  wz337

  authored
  bb61af0

  Copy full SHA for bb61af0

• Basic integration test infra (#170)

  Show description for 6500bc6
  

  gnadathur

  and

  gnadathur

  authored
  6500bc6

  Copy full SHA for 6500bc6

• Add 2D integration test (FSDP + TP) (#171)

  Show description for 479694f
  

  gnadathur

  and

  gnadathur

  authored
  479694f

  Copy full SHA for 479694f

Commits on Mar 28, 2024

• Used per-parameter FSDP (#165)

  Show description for 02923f0
  awgu

  authored
  02923f0

  Copy full SHA for 02923f0

• plot losses in loaded TrainState to TensorBoard

  Show description for 615f9c1
  tianyu-l

  committed
  615f9c1

  Copy full SHA for 615f9c1

Commits on Mar 29, 2024

• Removed setting global flag for swap_tensors since not needed anymore

  Show description for b1349da
  awgu

  committed
  b1349da

  Copy full SHA for b1349da

Commits on Apr 2, 2024

• Add integration test with compile enabled (#183)

  Show description for 65f0297
  

  gnadathur

  and

  gnadathur

  authored
  65f0297

  Copy full SHA for 65f0297

Commits on Apr 3, 2024

• remove folding and unfolding of sequence dim in model.py

  Show description for e1e17c9
  tianyu-l

  committed
  e1e17c9

  Copy full SHA for e1e17c9

Commits on Apr 4, 2024

• bump comm.train_timeout_seconds (#189)

  Show description for b9a4548
  wanchaol

  authored
  b9a4548

  Copy full SHA for b9a4548

Commits on Apr 5, 2024

• fix checkpoint parser

  Show description for 3686897
  wz337

  committed
  3686897

  Copy full SHA for 3686897

• support sequence of tests and add checkpoint test

  Show description for 7872248
  wz337

  committed
  7872248

  Copy full SHA for 7872248

• Make freqs_cis a persistent buffer for pp init

  Show description for 5ac3aa6
  wconstab

  committed
  5ac3aa6

  Copy full SHA for 5ac3aa6

• Delete grad scaler, which is unsupported/unused

  Show description for 5379282
  wconstab

  committed
  5379282

  Copy full SHA for 5379282

• Factor out loss_fn to share code with pipeline par

  Show description for d8e64cc
  wconstab

  committed
  d8e64cc

  Copy full SHA for d8e64cc

• [TorchTrain] Minor fix for #197 (#204)

  Show description for 0397fef
  wz337

  authored
  0397fef

  Copy full SHA for 0397fef

• Add FusedRMSNorm (Triton kernel, +15% eager), Add NPLayerNorm, Enable config selectable Norm Type (#181)

  Show description for cd1e5e8
  lessw2020

  authored
  cd1e5e8

  Copy full SHA for cd1e5e8

• remove .item() per iter

  Show description for f795361
  tianyu-l

  committed
  f795361

  Copy full SHA for f795361

Commits on Apr 10, 2024

• Removed cache_k and cache_v comments

  Show description for 946780a
  awgu

  committed
  946780a

  Copy full SHA for 946780a

• Some more cleanups

  Show description for 18adb2f
  awgu

  committed
  18adb2f

  Copy full SHA for 18adb2f

• avoid record streams and make color printing a config

  Show description for ef4c5d2
  tianyu-l

  committed
  ef4c5d2

  Copy full SHA for ef4c5d2

• fix SAC to use the correct reduce_scatter op (#215)

  Show description for 6629659
  wanchaol

  authored
  6629659

  Copy full SHA for 6629659

• Test runner raises exception on failures (#216)

  Show description for ddf916e
  gnadathur

  authored
  ddf916e

  Copy full SHA for ddf916e

• Revert "Separate TransformerEmbedding layer (#33)"

  Show description for ecdbacc
  wconstab

  committed
  ecdbacc

  Copy full SHA for ecdbacc

• Fix 2DParallel test (#219)

  Show description for 656be68
  gnadathur

  authored
  656be68

  Copy full SHA for 656be68

• Added initial FSDP readme

  Show description for 97fe9a4
  awgu

  committed
  97fe9a4

  Copy full SHA for 97fe9a4

Commits on Apr 11, 2024

• [TorchTrain][Checkpoint] Add model_weights_only option to train_config (#220)

  Show description for ce05f65
  wz337

  authored
  ce05f65

  Copy full SHA for ce05f65

• Rename to torchtitan (#221)

  Show description for 00293cb
  wanchaol

  authored
  00293cb

  Copy full SHA for 00293cb

Commits on Apr 12, 2024

• [TorchTitan] Add destory process group at the end of training (#223)

  wz337

  authored
  c08f617

  Copy full SHA for c08f617

• Add 1 sec delay to rank 0 cleanup (#224)

  Show description for 7712f72
  gnadathur

  authored
  7712f72

  Copy full SHA for 7712f72

• [Torchtrain][Checkpoint] Add support to allow dtype conversion (#222)

  Show description for 71621a2
  wz337

  authored
  71621a2

  Copy full SHA for 71621a2

• [TorchTitan] Remove checkpoint folder at the end in test_runner.py (#225)

  wz337

  authored
  5aa0aec

  Copy full SHA for 5aa0aec

Commits on Apr 15, 2024

• codebase cleanup

  Show description for cb24eb5
  tianyu-l

  committed
  cb24eb5

  Copy full SHA for cb24eb5

Commits on Apr 16, 2024

• Update README to reflect positioning (#229)

  Show description for 3cfdbf2
  wanchaol

  authored
  3cfdbf2

  Copy full SHA for 3cfdbf2

• First release readme (#227)

  Show description for db04c7e
  lessw2020

  authored
  db04c7e

  Copy full SHA for db04c7e

• Update licenses and headers (#231)

  wanchaol

  authored
  f504816

  Copy full SHA for f504816

• use permalink for logo image (#232)

  Show description for 41fb267
  lessw2020

  authored
  41fb267

  Copy full SHA for 41fb267

• [TorchTitan][Checkpoint] Move checkpoint folder under dump_folder and a few config updates (#230)

  Show description for 82c2518
  wz337

  authored
  82c2518

  Copy full SHA for 82c2518

• use combo of html and local file src for logo (#234)

  Show description for d42a7d1
  lessw2020

  authored
  d42a7d1

  Copy full SHA for d42a7d1

• add performance -- infra metrics and loss curves (#237) (#238)

  Show description for 80103a9
  

  lessw2020

  and

  tianyu-l

  authored
  80103a9

  Copy full SHA for 80103a9

• add license section in readme (#239)

  wanchaol

  authored
  09e7bec

  Copy full SHA for 09e7bec

• [TorchTitan][Checkpoint] Add a step-by-step instruction for checkpoint conversion (#235)

  wz337

  authored
  22aa488

  Copy full SHA for 22aa488

• more license headers (#240)

  wanchaol

  authored
  81138d6

  Copy full SHA for 81138d6

• Update README (#242)

  wanchaol

  authored
  04f5b82

  Copy full SHA for 04f5b82

• Add torchtune checkpoint link, modify product position statement location (#241)

  Show description for 4f6ed9a
  lessw2020

  authored
  4f6ed9a

  Copy full SHA for 4f6ed9a

• Add pyproject and upgrade version (#236)

  wanchaol

  authored
  cd55a38

  Copy full SHA for cd55a38

• minor doc updates - remove asynch checkpt ref, grammar on prod position, update checkpointing from 5 to 500 (#243)

  Show description for 78b843b
  lessw2020

  authored
  78b843b

  Copy full SHA for 78b843b

• Fix multi-line string usage (#244)

  Show description for ce0fff0
  gnadathur

  authored
  ce0fff0

  Copy full SHA for ce0fff0

• polish toml files

  Show description for 7b353c8
  tianyu-l

  committed
  7b353c8

  Copy full SHA for 7b353c8

• [torchtitan][checkpoint][doc] Minor fix checkpoint doc (#246)

  wz337

  authored
  bc7fec5

  Copy full SHA for bc7fec5

Commits on Apr 17, 2024

• fix default max_seq_len for freq_cis init (#248)

  Show description for a682505
  wanchaol

  authored
  a682505

  Copy full SHA for a682505

• set max_seq_len before training to make it align with input data (#249)

  Show description for 1ea4dee
  wanchaol

  authored
  1ea4dee

  Copy full SHA for 1ea4dee

• fix pypi docs

  Show description for 55c8e48
  tianyu-l

  committed
  55c8e48

  Copy full SHA for 55c8e48

Commits on Apr 18, 2024

• update dataset to use c4

  Show description for fd9b498
  tianyu-l

  committed
  fd9b498

  Copy full SHA for fd9b498

• Add c4_mini, a local 45K dataset (subset of c4) (#253)

  Show description for 978c5c6
  lessw2020

  authored
  978c5c6

  Copy full SHA for 978c5c6

• remove logo, update pre-release date to 4/18 (#254)

  Show description for 4020e92
  lessw2020

  authored
  4020e92

  Copy full SHA for 4020e92

• add intro video (#233)

  Show description for 51a6f6f
  lessw2020

  authored
  51a6f6f

  Copy full SHA for 51a6f6f

• add performance file to show convergence with 64 a100s (#255)

  Show description for 6aafe3c
  lessw2020

  authored
  6aafe3c

  Copy full SHA for 6aafe3c

Commits on Apr 20, 2024

• Support Llama3 8b/70b (#256)

  Show description for 35470ca
  wanchaol

  authored
  35470ca

  Copy full SHA for 35470ca

Commits on Apr 22, 2024

• polish llama 3 setup

  Show description for 960e70f
  tianyu-l

  committed
  960e70f

  Copy full SHA for 960e70f

Commits on Apr 23, 2024

• reenable integration tests with a test tokenizer (#259)

  Show description for e1c116a
  wanchaol

  authored
  e1c116a

  Copy full SHA for e1c116a

Commits on Apr 24, 2024

• warn supported dataset checks instead of throw (#260)

  wanchaol

  authored
  be432e1

  Copy full SHA for be432e1

• De-dup repeated freqs_cis computation code

  Show description for 192ed48
  awgu

  committed
  192ed48

  Copy full SHA for 192ed48

• update readme.md and performance.md

  Show description for f38766e
  tianyu-l

  committed
  f38766e

  Copy full SHA for f38766e

• followup changes to allow unsupported datasets

  Show description for 0eacbae
  tianyu-l

  committed
  0eacbae

  Copy full SHA for 0eacbae

• fix ac 'checkpointing' spelling, minor spacing tweaks (#265)

  Show description for 217cc94
  lessw2020

  authored
  217cc94

  Copy full SHA for 217cc94

Commits on Apr 25, 2024

• Update legal terms (#269)

  Show description for e3b47ea
  lessw2020

  authored
  e3b47ea

  Copy full SHA for e3b47ea

• apply less heavy profiling

  Show description for eed7495
  tianyu-l

  committed
  eed7495

  Copy full SHA for eed7495

• Showcase where the product positioning lies more clearly (#272)

  Show description for 3393c2a
  soumith

  authored
  3393c2a

  Copy full SHA for 3393c2a

• Doc Fixes (#273)

  Show description for 568dad6
  msaroufim

  authored
  568dad6

  Copy full SHA for 568dad6

Commits on Apr 26, 2024

• fix lr scheduling by checkpointing scheduler

  Show description for 3e13e24
  tianyu-l

  committed
  3e13e24

  Copy full SHA for 3e13e24

• insert barrier to profiler to resolve collectives timeout

  Show description for f03c128
  tianyu-l

  committed
  f03c128

  Copy full SHA for f03c128

• some misc changes (#278)

  Show description for 42549a9
  wanchaol

  authored
  42549a9

  Copy full SHA for 42549a9

• inherit stateful protocol where appropriate

  Show description for 0d09a32
  tianyu-l

  committed
  0d09a32

  Copy full SHA for 0d09a32

Commits on Apr 29, 2024

• Fixed docs on HSDP sharding/replication dims

  Show description for 06da6c2
  awgu

  committed
  06da6c2

  Copy full SHA for 06da6c2

• Add more Float8 description (#284)

  Show description for a843abf
  drisspg

  authored
  a843abf

  Copy full SHA for a843abf

• Remove unneeded torchvision/audio deps

  Show description for d442743
  wconstab

  committed
  d442743

  Copy full SHA for d442743

Commits on Apr 30, 2024

• fix 3d mesh order (#288)

  Show description for e7f2d28
  wanchaol

  authored
  e7f2d28

  Copy full SHA for e7f2d28

• unify data loading from HF and from disk

  Show description for 4e5ffaf
  tianyu-l

  committed
  4e5ffaf

  Copy full SHA for 4e5ffaf

Commits on May 1, 2024

• Add periodic integration test with signal (#289)

  Show description for 58b1169
  gnadathur

  authored
  58b1169

  Copy full SHA for 58b1169

Commits on May 2, 2024

• exclude embedding in MFU computation

  Show description for 4d8c245
  tianyu-l

  committed
  4d8c245

  Copy full SHA for 4d8c245

• Add support for seed checkpoint creation for meta-init flow

  Show description for 17cda29
  wconstab

  committed
  17cda29

  Copy full SHA for 17cda29

• remove unnecessary install of torchtitan

  Show description for 1a6caf2
  tianyu-l

  committed
  1a6caf2

  Copy full SHA for 1a6caf2

• Remove unnecessary .to() inside model forward

  Show description for 787a571
  wconstab

  committed
  787a571

  Copy full SHA for 787a571

Commits on May 3, 2024

• Fix the incorrect step log for profiler after resuming from a checkpoint (#293)

  Show description for 695bd01
  fegin

  authored
  695bd01

  Copy full SHA for 695bd01

• turn off dynamic shape for torch.compile (#297)

  Show description for 143b586
  wanchaol

  authored
  143b586

  Copy full SHA for 143b586

• Renamed bsz to bs for consistency; removed dead code

  Show description for f72a2a0
  awgu

  committed
  f72a2a0

  Copy full SHA for f72a2a0

Commits on May 7, 2024

• Implement async_checkpoint

  Show description for 3295448
  fegin

  committed
  3295448

  Copy full SHA for 3295448

Commits on May 8, 2024

• simplify embedding + first transformer block TP (#314)

  Show description for f5a3ad7
  wanchaol

  authored
  f5a3ad7

  Copy full SHA for f5a3ad7

Commits on May 10, 2024

• Only include checkpoints that have .metadata written (#315)

  Show description for e64c6ca
  liangluofb

  authored
  e64c6ca

  Copy full SHA for e64c6ca

Commits on May 13, 2024

• Refactor freqs_cis slice to be safer for PP

  Show description for 3444c4c
  wconstab

  committed
  3444c4c

  Copy full SHA for 3444c4c

• Make Transformer tolerate missing layers for PP

  Show description for 7f92f45
  wconstab

  committed
  7f92f45

  Copy full SHA for 7f92f45

Commits on May 15, 2024

• Use torch generic workflow for CI

  Show description for 0973bab
  wconstab

  committed
  0973bab

  Copy full SHA for 0973bab

• [checkpointing] import async checkpoint with pinned memory only when needed

  Show description for 5ba0a4b
  tianyu-l

  committed
  5ba0a4b

  Copy full SHA for 5ba0a4b

Commits on May 16, 2024

• Add a workflow to build torchtitan-ubuntu-20.04-clang12 Docker image for CI (#338)

  Show description for 847189d
  huydhn

  authored
  847189d

  Copy full SHA for 847189d

Commits on May 17, 2024

• Make pip install torch quiet

  Show description for a2ace60
  wconstab

  committed
  a2ace60

  Copy full SHA for a2ace60

• Make test_runner.py warn on non-empty output dir

  Show description for f2c3a11
  wconstab

  committed
  f2c3a11

  Copy full SHA for f2c3a11

Commits on May 21, 2024

• Expose mixed_precision dtype arguments

  Show description for 3bd14ec
  wconstab

  committed
  3bd14ec

  Copy full SHA for 3bd14ec

• Use stateful dataloader to checkpoint data iteration order and token buffer (#279)

  Show description for 99a73dd
  gokulavasan

  authored
  99a73dd

  Copy full SHA for 99a73dd

• Add Pipeline Parallel (and 2D PP+FSDP) support

  Show description for e7c31be
  wconstab

  committed
  e7c31be

  Copy full SHA for e7c31be

Commits on May 22, 2024

• fix i periodic integration test and add helper message on torchdata import failure

  Show description for c5a9718
  tianyu-l

  committed
  c5a9718

  Copy full SHA for c5a9718

• torch.compile each TransformerBlock instead of the whole model (#268)

  Show description for 60810a9
  wanchaol

  authored
  60810a9

  Copy full SHA for 60810a9

• Make test_runner use separate logger with default INFO

  Show description for 910662c
  wconstab

  committed
  910662c

  Copy full SHA for 910662c

• Fix llama_13b.toml -> llama2_13b.toml in multinode_trainer.slurm (#350)

  Show description for 6807909
  pbelevich

  authored
  6807909

  Copy full SHA for 6807909

• Fix bug in PP output layer shape

  Show description for 638ec48
  wconstab

  committed
  638ec48

  Copy full SHA for 638ec48

Commits on May 23, 2024

• Update pipelining import after change on pytorch

  Show description for c87e8bc
  wconstab

  committed
  c87e8bc

  Copy full SHA for c87e8bc

Commits on May 24, 2024

• update .gitignore to screen out slew of new temp files (#359)

  Show description for 02ae169
  lessw2020

  authored
  02ae169

  Copy full SHA for 02ae169

• Add test for PP tracer frontend

  Show description for 1ceaa4e
  wconstab

  committed
  1ceaa4e

  Copy full SHA for 1ceaa4e

Commits on May 29, 2024

• only produce tensorboard logs on rank 0 by default

  Show description for 6a8455e
  tianyu-l

  committed
  6a8455e

  Copy full SHA for 6a8455e

• replace old torch dependency in requirements.txt

  Show description for 5831e81
  tianyu-l

  committed
  5831e81

  Copy full SHA for 5831e81

Commits on May 30, 2024

• Add --test option to specify test to run (#368)

  Show description for 3343d1d
  kwen2501

  authored
  3343d1d

  Copy full SHA for 3343d1d

• use integration test as the badge shown on the homepage

  Show description for 07fa503
  tianyu-l

  committed
  07fa503

  Copy full SHA for 07fa503

Commits on May 31, 2024

• keep only latest k checkpoints (#366)

  Show description for 54b5fa2
  liangluofb

  authored
  54b5fa2

  Copy full SHA for 54b5fa2

Commits on Jun 3, 2024

• Make seed checkpoint creation work on CPU

  Show description for a0f82d5
  wconstab

  committed
  a0f82d5

  Copy full SHA for a0f82d5

• Fix start/stop layer parsing

  Show description for 8badf7e
  wconstab

  committed
  8badf7e

  Copy full SHA for 8badf7e

• Use general way to access and update submodules

  Show description for 3050098
  kwen2501

  committed
  3050098

  Copy full SHA for 3050098

Commits on Jun 4, 2024

• Make metrics logging work for pipeline parallelism

  Show description for 0594c04
  wconstab

  committed
  0594c04

  Copy full SHA for 0594c04

Commits on Jun 5, 2024

• [RFC] Allow ModelWrapper and OptimizerWrapper to accept multiple models

  Show description for c89aa40
  

  fegin

  authored and

  wconstab

  committed
  c89aa40

  Copy full SHA for c89aa40

• Add 3D support

  Show description for 3bc7678
  wconstab

  committed
  3bc7678

  Copy full SHA for 3bc7678

Commits on Jun 6, 2024

• [torchtitan][optim] Add fused as an option in train config (#355)

  Show description for 7cf41bb
  wz337

  authored
  7cf41bb

  Copy full SHA for 7cf41bb

• [torchtitan] Fix test runner fused optim tests (#384)

  wz337

  authored
  baa678c

  Copy full SHA for baa678c

Commits on Jun 7, 2024

• Abstract out out optimizer params and update foreach calling convention (#386)

  Show description for 104bd6c
  drisspg

  authored
  104bd6c

  Copy full SHA for 104bd6c

Commits on Jun 9, 2024

• DeviceMesh BC fix (#387)

  Show description for ccf1ed8
  wanchaol

  authored
  ccf1ed8

  Copy full SHA for ccf1ed8

• BC fix for ManualPipelineStage import (#388)

  wanchaol

  authored
  6de5d31

  Copy full SHA for 6de5d31

Commits on Jun 10, 2024

• fix missing tb logs

  Show description for 9700e0f
  tianyu-l

  committed
  9700e0f

  Copy full SHA for 9700e0f

• add the 8-gpu test badge and use correct links for the integration test badges

  Show description for 3bb7bf3
  tianyu-l

  committed
  3bb7bf3

  Copy full SHA for 3bb7bf3

Commits on Jun 11, 2024

• Fix 1D PP tracer test

  Show description for e858ab4
  

  kwen2501

  authored and

  wconstab

  committed
  e858ab4

  Copy full SHA for e858ab4

Commits on Jun 12, 2024

• del logits=(bs, seq_len, vocab_size) to save 3.9G memory (#391)

  Show description for c5d5c1f
  weifengpy

  authored
  c5d5c1f

  Copy full SHA for c5d5c1f

• Update contributing.md (#385)

  Show description for e991ae4
  H-Huang

  authored
  e991ae4

  Copy full SHA for e991ae4

• update all toml files to use experimental section (#392)

  wanchaol

  authored
  763b810

  Copy full SHA for 763b810

Commits on Jun 13, 2024

• enable TP fp8 allgather with PrepareFloat8ModuleInput (#393)

  Show description for e17e3b8
  wanchaol

  authored
  e17e3b8

  Copy full SHA for e17e3b8

• Update unit_test_cpu.yaml with cpu nightly (#396)

  wanchaol

  authored
  a4cd9ba

  Copy full SHA for a4cd9ba

• Fix SAC BC breaking and renaming to ac_freq (#397)

  Show description for 33f301d
  wanchaol

  authored
  33f301d

  Copy full SHA for 33f301d

• SAC API follow ups to restore old behavior (#401)

  Show description for 093ba15
  wanchaol

  authored
  093ba15

  Copy full SHA for 093ba15

Commits on Jun 14, 2024

• enable TritonFusedRMSNorm with local_map annotation (#404)

  Show description for d761994
  XilunWu

  authored
  d761994

  Copy full SHA for d761994

• Cosmetic changes to train.py

  Show description for 3e946a1
  kwen2501

  committed
  3e946a1

  Copy full SHA for 3e946a1

• Break down parallelize_llama for inference cases

  Show description for d5b7525
  kwen2501

  committed
  d5b7525

  Copy full SHA for d5b7525

Commits on Jun 17, 2024

• Change debugmodel to have 8 layers

  Show description for 4a1095a
  wconstab

  committed
  4a1095a

  Copy full SHA for 4a1095a

Commits on Jun 18, 2024

• Prepare train.py for model chunks for pipelining

  Show description for 443ead2
  wconstab

  committed
  443ead2

  Copy full SHA for 443ead2

Commits on Jun 19, 2024

• dump memory snapshot to analyze OOMs (#395)

  Show description for 8adbfa3
  weifengpy

  authored
  8adbfa3

  Copy full SHA for 8adbfa3

Commits on Jun 20, 2024

• whole_model for fp8 (#414)

  Show description for c88d0bc
  weifengpy

  authored
  c88d0bc

  Copy full SHA for c88d0bc

Commits on Jun 21, 2024

• Add train loop support for looped PP schedules

  Show description for ac83f9c
  wconstab

  committed
  ac83f9c

  Copy full SHA for ac83f9c

Commits on Jun 25, 2024

• Set record_shapes=True for profiler

  Show description for abb9e15
  awgu

  committed
  abb9e15

  Copy full SHA for abb9e15

• Improved repeat_kv eager perf

  Show description for 4ba94bd
  awgu

  committed
  4ba94bd

  Copy full SHA for 4ba94bd

• Adding FSDP Memory Tracking and Estimation

  Show description for 236d2ff
  sanketpurandare

  committed
  236d2ff

  Copy full SHA for 236d2ff

• Adding integration test for FSDP Memory Tracking and Estimation

  Show description for cb73810
  sanketpurandare

  committed
  cb73810

  Copy full SHA for cb73810

Commits on Jun 26, 2024

• by default disable heavy memory profiling

  Show description for f3fecb2
  tianyu-l

  committed
  f3fecb2

  Copy full SHA for f3fecb2

Commits on Jun 27, 2024

• Add the option to turn on async-TP

  Show description for ea8c5c8
  yifuwang

  committed
  ea8c5c8

  Copy full SHA for ea8c5c8

Commits on Jul 1, 2024

• Modifying memory estimation options and minor changes

  Show description for b0ed7f0
  sanketpurandare

  committed
  b0ed7f0

  Copy full SHA for b0ed7f0

Commits on Jul 8, 2024

• add comment pointing to Sequence Parallel optimization example

  Show description for 21dd980
  tianyu-l

  committed
  21dd980

  Copy full SHA for 21dd980

• switch float8 logic from Float8DynamicLinear to Float8Linear (#436)

  Show description for 3fca883
  vkuzo

  authored
  3fca883

  Copy full SHA for 3fca883

Commits on Jul 10, 2024

• Removed _experimental_support_context_fn_in_torch_utils_checkpoint

  Show description for 2f5285b
  awgu

  committed
  2f5285b

  Copy full SHA for 2f5285b

• Reordered TP parallel plan to follow execution order

  Show description for f0ca3e8
  awgu

  committed
  f0ca3e8

  Copy full SHA for f0ca3e8

• Made some stylistic changes to apply_dp

  Show description for c7a6a3e
  awgu

  committed
  c7a6a3e

  Copy full SHA for c7a6a3e

• Refactored activation checkpointing

  Show description for bc3ec02
  awgu

  committed
  bc3ec02

  Copy full SHA for bc3ec02

• compiled RMSNorm

  Show description for 7afe902
  tianyu-l

  committed
  7afe902

  Copy full SHA for 7afe902

Commits on Jul 11, 2024

• Renamed parallel styles for transformer block weights

  Show description for 0929280
  awgu

  committed
  0929280

  Copy full SHA for 0929280

• Added type annotations and more stylistic changes

  Show description for 040ea1d
  awgu

  committed
  040ea1d

  Copy full SHA for 040ea1d

Commits on Jul 15, 2024

• [Cleanup] Remove libuv from run_llama_train.sh

  Show description for e591f62
  wconstab

  committed
  e591f62

  Copy full SHA for e591f62

• [Cleanup] Organize run_llama_train.sh options

  Show description for 6f4d1d1
  wconstab

  committed
  6f4d1d1

  Copy full SHA for 6f4d1d1

• [Cleanup] Split run_llama_train.sh and run_memory_estimation.sh

  Show description for db609d5
  wconstab

  committed
  db609d5

  Copy full SHA for db609d5

• [Cleanup] Remove unused TRAINER_DIR

  Show description for 07ab2f9
  wconstab

  committed
  07ab2f9

  Copy full SHA for 07ab2f9

• Add educational code pointers to top level README

  Show description for 0bb6980
  wconstab

  committed
  0bb6980

  Copy full SHA for 0bb6980

Commits on Jul 16, 2024

• enable FSDP2 + fp8 all-gather and fix TP fp8 all-gather (#413)

  Show description for f025335
  weifengpy

  authored
  f025335

  Copy full SHA for f025335

Commits on Jul 17, 2024

• import float8_experimental only when fp8 is enabled and install it in CI (#464)

  Show description for 298494b
  weifengpy

  authored
  298494b

  Copy full SHA for 298494b

• skip fp8 CI on non-H100 GPUs (#465)

  Show description for 183390e
  weifengpy

  authored
  183390e

  Copy full SHA for 183390e

• clean up float8 configs in torchtitan (#466)

  Show description for 4a2de42
  vkuzo

  authored
  4a2de42

  Copy full SHA for 4a2de42

Commits on Jul 18, 2024

• Add support of DDP and experimental CompiledAutograd

  Show description for 2937167
  fegin

  committed
  2937167

  Copy full SHA for 2937167

Commits on Jul 19, 2024

• add torch.compile + FSDP2 float8 all-gather in CI (#468)

  Show description for 9b37408
  weifengpy

  authored
  9b37408

  Copy full SHA for 9b37408

• [float8] keep model.output as nn.Linear (high precision, not fp8) (#469)

  Show description for b502cdc
  weifengpy

  authored
  b502cdc

  Copy full SHA for b502cdc

Commits on Jul 20, 2024

• remove CI for FSDP2 + fp8 all-gather (#470)

  Show description for d76b77f
  weifengpy

  authored
  d76b77f

  Copy full SHA for d76b77f

Commits on Jul 21, 2024

• dynamically update torch.compile cache config to ensure async tp support, enhance async tp UX (#471)

  Show description for 0f70507
  lessw2020

  authored
  0f70507

  Copy full SHA for 0f70507

Commits on Jul 26, 2024

• Fix 8gpu PP failure due to 2D DCP disablement

  Show description for 00a3c21
  wconstab

  committed
  00a3c21

  Copy full SHA for 00a3c21

• update float8 integration after UX changes (#484)

  Show description for 5124c14
  vkuzo

  authored
  5124c14

  Copy full SHA for 5124c14

• Re-enable FSDP2 Mem Tracker integration tests

  Show description for 43584e0
  

  Sanket Jayant Purandare

  committed
  43584e0

  Copy full SHA for 43584e0

Commits on Jul 29, 2024

• Used partial instead of global vars for LR scheduling

  Show description for 668f6cd
  awgu

  committed
  668f6cd

  Copy full SHA for 668f6cd

Commits on Jul 30, 2024

• [EZ] Add logs for some basic training params so that we can verify in… (#491)

  Show description for f13fe3f
  fduwjj

  authored
  f13fe3f

  Copy full SHA for f13fe3f

• make float8 scaling type configurable (#489)

  Show description for b012237
  vkuzo

  authored
  b012237

  Copy full SHA for b012237

• [PP] add flexible interleaved 1f1b schedule #490 (#493)

  Show description for 9cf4b2f
  H-Huang

  authored
  9cf4b2f

  Copy full SHA for 9cf4b2f

• move float8 callsites to torchao.float8 (#492)

  Show description for b069f70
  vkuzo

  authored
  b069f70

  Copy full SHA for b069f70

Commits on Aug 1, 2024

• [BE][1/n] simplify train.py

  Show description for 3ddce59
  tianyu-l

  committed
  3ddce59

  Copy full SHA for 3ddce59

• [BE][2/n] use proper method signatures in parallelize_llama

  Show description for 7119d0c
  tianyu-l

  committed
  7119d0c

  Copy full SHA for 7119d0c

• [BE][3/n] wrap fp8 logic using Float8Handler

  Show description for 04d219a
  tianyu-l

  committed
  04d219a

  Copy full SHA for 04d219a

• Bring LLaMa 3.1 405B to TorchTitan family (#481)

  Show description for e457deb
  fduwjj

  authored
  e457deb

  Copy full SHA for e457deb

Commits on Aug 2, 2024

• [TP] Infer local n_heads instead of ad-hoc model changes

  Show description for 72a1614
  kwen2501

  committed
  72a1614

  Copy full SHA for 72a1614

Commits on Aug 3, 2024

• some compile-related updates

  Show description for de9fd2b
  tianyu-l

  committed
  de9fd2b

  Copy full SHA for de9fd2b

Commits on Aug 5, 2024

• [EZ][405B] Use scientific notation for 405B model lr

  fduwjj

  committed
  b3f2f58

  Copy full SHA for b3f2f58

Commits on Aug 21, 2024

• [405B] Add performance data for 405B model

  fduwjj

  committed
  904913f

  Copy full SHA for 904913f

• Merge branch 'main' into 405b_more

  fduwjj

  committed
  7928650

  Copy full SHA for 7928650

• Handle rebase

  fduwjj

  committed
  57d4725

  Copy full SHA for 57d4725

• Further handle merge

  fduwjj

  committed
  cbcd3f2

  Copy full SHA for cbcd3f2