Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FlashInfer] Upgrade to 0.2.0 #11194

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open

[FlashInfer] Upgrade to 0.2.0 #11194

wants to merge 37 commits into from
+207 −36

Conversation

abmfy
Copy link

@abmfy abmfy commented Dec 14, 2024

This PR upgrades the FlashInfer attention backend to v0.2.0.

@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@abmfy
Copy link
Author

abmfy commented Dec 14, 2024

@youkaichao

@youkaichao
Copy link
Member

Looking forward to the update!

@Swipe4057
Copy link

https://github.com/flashinfer-ai/flashinfer/releases/tag/v0.2.0

@pavanimajety pavanimajety mentioned this pull request Dec 19, 2024
Open
@DarkLight1337 DarkLight1337 mentioned this pull request Dec 20, 2024
Closed
1 task
@youkaichao youkaichao mentioned this pull request Jan 21, 2025
Merged
youkaichao
youkaichao reviewed Jan 23, 2025
View reviewed changes
vllm/attention/backends/flashinfer.py Outdated
sm_scale: float


def infer_global_hyperparameters(model: nn.Module) -> GlobalHyperparameters:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this function can collect all per_layer_parameter, and only assert the results are the same.

youkaichao
youkaichao reviewed Jan 23, 2025
View reviewed changes
vllm/attention/backends/flashinfer.py
@@ -495,6 +597,8 @@ def __init__(self, input_builder: "ModelInputForGPUBuilder"):
self.sliding_window = input_builder.sliding_window
self.block_size = input_builder.block_size
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can remember the vllm_config here by calling get_current_vllm_config()

youkaichao
youkaichao reviewed Jan 23, 2025
View reviewed changes
vllm/attention/backends/flashinfer.py Outdated
# - `window_left`
# - `logits_soft_cap`
# - `sm_scale`
model = self.runner.model
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vllm_config.compilation_config.static_forward_context is a dict of layer prefix to attention layer. you can collect sliding window, etc. from there. no need to iterate over model's submodule.

youkaichao
youkaichao reviewed Jan 23, 2025
View reviewed changes
vllm/attention/backends/flashinfer.py
@@ -178,6 +179,9 @@ def __init__(self, runner):
self._decode_wrapper = None
self._prefill_wrapper = None

# Global hyperparameters shared by all attention layers
self.global_hyperparameters: Optional[PerLayerParameters] = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remember the vllm_config here?

youkaichao
youkaichao reviewed Jan 23, 2025
View reviewed changes
vllm/worker/model_runner.py Outdated
Comment on lines 1501 to 1509
with set_current_vllm_config(self.vllm_config):
# To make vLLM config available during
# worker initialization
attn_metadata = (self.attn_state.
graph_capture_get_metadata_for_batch(
batch_size,
is_encoder_decoder_model=self.
model_config.is_encoder_decoder,
))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then we don't need this change.

@youkaichao
Copy link
Member

also need to update this line to pass the ci:

vllm/Dockerfile

Line 200 in f0ef372

python3 -m pip install https://github.com/flashinfer-ai/flashinfer/releases/download/v0.1.6/flashinfer-0.1.6+cu121torch2.4-cp${PYTHON_VERSION_STR}-cp${PYTHON_VERSION_STR}-linux_x86_64.whl; \

@mergify mergify bot added the ci/build label Jan 23, 2025
youkaichao
youkaichao approved these changes Jan 23, 2025
View reviewed changes
Copy link
Member

@youkaichao youkaichao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the contribution!

@youkaichao youkaichao marked this pull request as ready for review January 23, 2025 13:55
@mergify mergify bot removed the needs-rebase label Jan 24, 2025
abmfy and others added 7 commits January 24, 2025 12:22
tlrmchlsmth
tlrmchlsmth reviewed Jan 26, 2025
View reviewed changes
Dockerfile Outdated
Comment on lines 216 to 220
# RUN --mount=type=cache,target=/root/.cache/pip \
# . /etc/environment && \
# if [ "$TARGETPLATFORM" != "linux/arm64" ]; then \
# python3 -m pip install https://github.com/flashinfer-ai/flashinfer/releases/download/v0.2.0.post1/flashinfer-0.2.0.post1+cu121torch2.4-cp${PYTHON_VERSION_STR}-cp${PYTHON_VERSION_STR}-linux_x86_64.whl; \
# fi
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delete prior to landing?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commented in bb44221 . we might need it soon, so I just comment it out right now.

@youkaichao youkaichao enabled auto-merge (squash) January 26, 2025 13:28
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Jan 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@youkaichao youkaichao youkaichao approved these changes

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

@zhuohan123 zhuohan123 Awaiting requested review from zhuohan123 zhuohan123 is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

@comaniac comaniac Awaiting requested review from comaniac comaniac is a code owner

@njhill njhill Awaiting requested review from njhill njhill is a code owner

@mgoin mgoin Awaiting requested review from mgoin

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon

@LiuXiaoxuanPKU LiuXiaoxuanPKU Awaiting requested review from LiuXiaoxuanPKU

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337

@ywang96 ywang96 Awaiting requested review from ywang96

@simon-mo simon-mo Awaiting requested review from simon-mo

Assignees
No one assigned
Labels
ci/build documentation Improvements or additions to documentation frontend ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@abmfy @youkaichao @Swipe4057 @tlrmchlsmth