Release v0.7.0

[simon-mo]

• Alpha release for vLLM V1 architecture, ETA 1/23-1/24
• Pending V1 items
  • Performance numbers @ywang96 @robertgshaw2-redhat @WoosukKwon
  • Documentation @WoosukKwon
  • Blog post @WoosukKwon
• Other Pending PRs
  • [core] add wake_up doc and some sanity check #12361
  • [torch.compile] decouple compile sizes and cudagraph sizes #12243
  • Revert "[core] separate builder init and builder prepare for each batch" #12377 [perf] fix perf regression from #12253 #12380
  • [Bugfix][Kernel] Fix CUDA 11.8 being broken by FA3 build #12375
  • [Bugfix][Kernel] FA3 Fix - RuntimeError: This flash attention build only supports pack_gqa (for build size reasons). #12405

[njhill]

#12298 should also now be ready. merged

[ywang96]

I should have some numbers ready by EoD

[njhill]

THe perf dashboard shows a potential regression (see slack thread), which looks most likely to be caused by #12253.

[wangxiyuan]

And this one please #11324, the last PR for platform pluggable.

[comaniac]

THe perf dashboard shows a potential regression (see slack thread), which looks most likely to be caused by #12253.

Revert: #12377

[youkaichao]

THe perf dashboard shows a potential regression (see slack thread), which looks most likely to be caused by #12253.

Revert: #12377

Fixed in #12380

[robertgshaw2-redhat]

@ElizaWszola - can you post the Mixtral issue

[DarkLight1337]

There are still a few consistent CI failures on main:

• Language Models Tests (Extended)
  • Accuracy error for Mistral model: Tests started failing between https://buildkite.com/vllm/ci/builds/11418#01942acd-a27b-477a-809b-1fd358291b41 and https://buildkite.com/vllm/ci/builds/11468#01942fb0-18f3-4deb-8bb4-8dbcaa2987e1
  • Unable to load Granite MoE model: Will be fixed by [Bugfix] Fix Granite 3.0 MoE model loading #12446
  • Memory profiling error for Mamba model: Tests started failing between https://buildkite.com/vllm/ci/builds/11861#019458e3-1b40-45b0-9a45-d5cd2975ec1a and https://buildkite.com/vllm/ci/builds/11926#0194632f-b3a9-43e2-9d5f-a01bf7233bca
  • Shape mismatch error in GGUF tests: Tests started failing between https://buildkite.com/vllm/ci/builds/11861#019458e3-1b40-45b0-9a45-d5cd2975ec1a and https://buildkite.com/vllm/ci/builds/11926#0194632f-b3a9-43e2-9d5f-a01bf7233bca
• Multi-Modal Models Tests (Extended)
  • Unable to run BLIP-2 model: Will be fixed by [Bugfix] Fix BLIP-2 processing #12412

[ElizaWszola]

There are some issues in the moe_align_block_size_kernel causing illegal memory access in Mixtral models. Here is a PR that fixes this #12413 (I still haven't figured out why the current code is broken though).

[tlrmchlsmth]

#12417 would be good to pick up as well

[DarkLight1337]

Anyone available to review #12251? The authors made the effort to open the PR before the model release so we should handle this ASAP.

[rainkert]

Anyone available to review #12251? The authors made the effort to open the PR before the model release so we should handle this ASAP.

Yes, this is very urgent. After we released the model, we have received many user inquiries about how to run this model within the vLLM framework. Please address this PR as soon as possible.

[robertgshaw2-redhat]

Anyone available to review #12251? The authors made the effort to open the PR before the model release so we should handle this ASAP.

Yes, this is very urgent. After we released the model, we have received many user inquiries about how to run this model within the vLLM framework. Please address this PR as soon as possible.

When is the model release planned for?

[DarkLight1337]

When is the model release planned for?

It has been released yesterday already. The PR was originally opened 4 days ago.

[simon-mo]

I plan to cut a release now for V1 alpha release now.

Currently, the knowns issues are

• To be investigated:
  • Accuracy error for Mistral model: Tests started failing between https://buildkite.com/vllm/ci/builds/11418#01942acd-a27b-477a-809b-1fd358291b41 and https://buildkite.com/vllm/ci/builds/11468#01942fb0-18f3-4deb-8bb4-8dbcaa2987e1
  • Memory profiling error for Mamba model: Tests started failing between https://buildkite.com/vllm/ci/builds/11861#019458e3-1b40-45b0-9a45-d5cd2975ec1a and https://buildkite.com/vllm/ci/builds/11926#0194632f-b3a9-43e2-9d5f-a01bf7233bca
  • Shape mismatch error in GGUF tests: Tests started failing between https://buildkite.com/vllm/ci/builds/11861#019458e3-1b40-45b0-9a45-d5cd2975ec1a and https://buildkite.com/vllm/ci/builds/11926#0194632f-b3a9-43e2-9d5f-a01bf7233bca
• To be merged:
  • Support for Qwen-1M: Implements dual-chunk-flash-attn backend for dual chunk attention with sparse attention support #11844
  • Support for Baichuan-M1: [Model] Enable Inference Support for the New Baichuan-M1 Model #12251

The next release will happen whenever the to model supports are merged, and/or update to Deepseek support pending FlashInfer kernels.