New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Does attention mask reduce computation cost？ #14

Open

brotherb opened this issue Dec 21, 2023 · 1 comment

brotherb commented Dec 21, 2023 •

edited

Loading

Hey, there.
After I read the code, I am confused that the computation cost can be reduced by mask more tokens. Did I miss anything?

PS. I see the FLOPS is calculated by the length of tokens retented at each layer which is counted during inference.

Do you have inference latency metrics?

brotherb changed the title ~~Can attention mask reduce computation cost？~~ Does attention mask reduce computation cost？

yaoyao926 commented Nov 11, 2024

Did you solve it in the end?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment