Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Free up GPU memory after unregistering model in meta reference inference #768

Open
SLR722 opened this issue Jan 15, 2025 · 0 comments
Open
Assignees
Labels
enhancement New feature or request

Comments

@SLR722
Copy link
Contributor

SLR722 commented Jan 15, 2025

🚀 Describe the new functionality needed

as title

💡 Why is this needed? What if we don't build it?

This can free up the memory if we want to switch the models for inference. Also semantically, it doesn't make sense to continue occupying the memory after the model is unregistered

A real application is that if we want to inference the base model -> do post training -> inference the finetuned model. If we can not free up the memory occupied by the base model with unregister command, it's easy to be OOMed.

Other thoughts

No response

@SLR722 SLR722 changed the title Free-up GPU memory after unregistering model in meta reference inference Free up GPU memory after unregistering model in meta reference inference Jan 15, 2025
@SLR722 SLR722 self-assigned this Jan 15, 2025
@SLR722 SLR722 added the enhancement New feature or request label Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant