Add option to support garbage collection after torch compilation #2559
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request was exported from Phabricator. Differential Revision: D67062158 |
qiurc
added a commit
to qiurc/benchmark
that referenced
this pull request
Dec 14, 2024
…orch#2559) Summary: X-link: pytorch/pytorch#142821 This diff is an extension of ezyang's PR https://fburl.com/6uvvzb4f. In ezyang's PR above, it adds gc after torch compilation finished. The gc operation is guarded by jk: pytorch/compiler:enable_run_gc_after_compile The gc op time cost will be logged into dynamo_compile scuba table. This diff extends the PR to: - Use garbage collection on Generation 1 instead of generation 2 (default), which greatly reduced the gc latency overhead from 160 sec per rank to 10 sec per rank. - Additionally introduce an environment variance which has the higher priority than the JK to control whether we do gc or not after the torch compilation. (default value set to gc enabled). This environment variance will be used for AB testing of training jobs to compare the pt2 compilation time and memory cost. Reviewed By: ezyang, yuxihu Differential Revision: D67062158
qiurc
force-pushed
the
export-D67062158
branch
from
b5754ce to
95f500a
Compare
|
This pull request was exported from Phabricator. Differential Revision: D67062158 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D67062158 |
qiurc
force-pushed
the
export-D67062158
branch
from
95f500a to
68829a6
Compare
|
This pull request was exported from Phabricator. Differential Revision: D67062158 |
qiurc
force-pushed
the
export-D67062158
branch
2 times, most recently
from
d082407 to
29f6ce7
Compare
|
This pull request was exported from Phabricator. Differential Revision: D67062158 |
Summary: See test plan in internal diff [D66679369](https://our.internmc.facebook.com/intern/diff/D66679369) X-link: pytorch/pytorch#141919 Approved by: https://github.com/jamesjwu, https://github.com/ezyang Differential Revision: D67218561 Pulled By: masnesral
|
This pull request was exported from Phabricator. Differential Revision: D67062158 |
qiurc
force-pushed
the
export-D67062158
branch
from
29f6ce7 to
0489041
Compare
qiurc
had a problem deploying
to
docker-s3-upload
GitHub Actions
Failure
qiurc
temporarily deployed
to
docker-s3-upload
GitHub Actions
Inactive
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
X-link: pytorch/pytorch#142821
This diff is an extension of ezyang's PR https://fburl.com/6uvvzb4f.
In ezyang's PR above, it adds gc after torch compilation finished.
The gc operation is guarded by jk: pytorch/compiler:enable_run_gc_after_compile
The gc op time cost will be logged into dynamo_compile scuba table.
This diff extends the PR to:
Reviewed By: ezyang
Differential Revision: D67062158