Model loading #31

tigranfah · 2024-10-18T14:11:06Z

add llama configs for training
add the possibility of downsampling some molecular properties during the training
add special tokens for the property tags, smiles and safe
add CONFORMER tag to the tokenizer
verify logits matching with huggingface when download and exporting llama

…save the checkpoint into the same place

…and some molecular properties

…ty tags, implement safe data processing

MenuaB · 2024-10-19T07:15:47Z

LGTM

tigranfah added 17 commits October 2, 2024 15:15

prepare llama3.2-1b for pretraining

bc00070

temporeraly remove many of the properties for llama 3.2 training

7778ea3

add llama3.2 export and share the aim hash to all ranks to have them …

9f5099d

…save the checkpoint into the same place

Merge branch 'hparams' into model_loading

a88f514

tune the config of llama3.2 for pretraining wiht safe representation …

d27cdee

…and some molecular properties

add loading from specific step

40ae625

small fices

2139ac6

change the properties used

3b171f7

build llama 3.2 tokenizer with special tokens for molecule and proper…

8ac0a17

…ty tags, implement safe data processing

rename llama 3.2 tokenizer

bae05b5

resize the llama 3.2 token embedding size

5b23fa3

add data processing with downsampling

ed0c9f2

add new learning rates, log the learning rate and track it on aim

3d4946c

change warmup steps to 1000 in llama 3 1b for lr tunning

e8fd8a3

prepare for llama 3.2 1b training, lr=2.5-e4

5dc2dbb

Add CONFORMER tag

1a03114

add logits matching verified when downloading and exporting to hf

7b97d4d

tigranfah requested a review from MenuaB October 18, 2024 14:11

MenuaB approved these changes Oct 19, 2024

View reviewed changes

MenuaB merged commit 8e38015 into main Oct 19, 2024
0 of 4 checks passed

Provide feedback