Upstream Open-Source Contributions

Upstream RFC: Accelerator Capability Descriptor (ACD) & Hardware-Adaptive Serving Policies

Motivation & Problem: Current serving engines enforce hardcoded chunk sizes (e.g. 512 tokens) and symmetric parallelism across disaggregated pools (prefill_tp == decode_tp). On compute-dense GPUs like NVIDIA L40S (423 FLOPs/Byte inflection), small chunks starve execution units; on lower-inflection GPUs like RTX 3090 (76 FLOPs/Byte), static chunks waste memory. Furthermore, disaggregated clusters cannot pair compute GPUs for prefill with memory GPUs for decode because KV head sharding layouts conflict.

Proposed Specification: (1) An EngineConfig hardware interrogation interface standardizing dynamic Roofline inflection calculations (inflection_point = peak_tflops / memory_bandwidth) to auto-tune batching and chunked prefill budgets; and (2) an Asymmetric Disaggregation Protocol enabling mismatched tensor parallelism via bipartite GQA head partitions and in-flight FP8 wire quantization.

[Bugfix][KV Connector] Backoff Last Prompt Token for Sliding Window in P/D (#65550)

Root Cause: In disaggregated Prefill/Decode serving architectures utilizing external KV connectors (such as NIXL or Mooncake), the decode worker recomputes token N-1 to prime its local KV cache state. When serving models with sliding window attention (e.g. Mistral, Qwen) where (N - W + 1) % block_size == 0, the decoder dropped the oldest block (N - W) because it fell strictly outside the sliding window boundary. This caused FlashAttention kernels to read unallocated memory from null_block, producing silent generation corruption and invalid tokens without triggering runtime exceptions.

Architectural Fix: Enforced prompt token backoff scoped strictly to sliding window models within the P/D connector path, ensuring valid KV block references are preserved without modifying full-attention execution pathways.

Unbounded Memory Growth in Prefill/Decode Disaggregation via Orphaned ZMQ Rooms

Root Cause: In SGLang's disaggregated prefill/decode architecture, ZeroMQ bootstrap rooms are allocated to coordinate inter-node tensor transfer. Under network partitions or abrupt client disconnects, room references were never garbage-collected, leading to linear memory growth and eventual host OOM.

Architectural Fix: Added an active O(1) min-heap TTL (Time-To-Live) sweeper thread that periodically cleans expired handshake tokens and evicts orphaned room states.

[Bugfix][Scheduler] Drain In-Flight Stale Output on Sync KV Load-Failure Recovery (#65131)

Root Cause: In vLLM V1's AsyncScheduler, during pipelined batch execution, synchronous KV cache load operations can fail due to network timeouts or storage transient errors. The failure recovery handler re-queued the request but failed to drain pending in-flight output token placeholders. In subsequent step iterations, the scheduler encountered negative placeholder counts, triggering fatal assertion failures (assert num_output_placeholders >= 0) and crashing the engine worker process.

Architectural Fix: Enforced explicit in-flight token draining via num_stale_output_tokens reconciliation during KV failure recovery, resetting scheduler placeholder tracking prior to request re-admission. (Note: Superseded duplicate attempt PR #65686 was closed and consolidated into this patch).

Preempted Requests Starvation & Re-Admission in PriorityRequestQueue

Origin & Root Cause: I filed the initial bug report (vLLM issue #41951), identifying that when vLLM's v1 engine experiences memory pressure and preempts in-flight requests, those preempted requests lose re-admission priority in the PriorityRequestQueue, causing indefinite starvation under continuous ingress traffic.

Fix Evolution: I authored two successive patches: PR #41952 (stale) and PR #56843 (which introduced a bounded starvation cap). The changes and regression test suite were subsequently cherry-picked into the consolidated PR #51574, where I hold co-authorship credit.

Refactoring RadixCache.total_size() from O(N) to O(1)

Code Change: Refactored the internal radix tree size accounting in RadixCache to maintain an incremental token count during insertion and eviction, eliminating recursive tree traversals on metric polling.

Marlin Quantized Backend Crash with Unquantized Draft Models

Root Cause: When running speculative decoding where the target model is Marlin-quantized (FP8/INT4) but the draft model contains unquantized layers, vLLM threw an unhandled execution exception because the Marlin backend failed to execute non-quantized tensor operations.

Architectural Fix: Implemented a dynamic fallback check in the speculative decoding model loader, gracefully defaulting unquantized draft layers to the auto backend execution path without degrading target kernel speed.

Qwen3.5 GDN Hybrid Model Crash Under Marlin Tile Divisibility

Root Cause: Attempting to load Qwen3.5 GDN hybrid architecture models under Marlin quantization resulted in silent tile divisibility violations, causing CUDA illegal memory access errors during tensor layout initialization.

Architectural Fix: Added pre-flight dimension validation checking tile-divisibility constraints prior to kernel invocation, triggering safe fallback to compatible kernel variants when non-standard dimensions are encountered.

[Bugfix][Quantization] Normalize Quark Exclude Entries with Parameter Suffixes (#65034)

Root Cause: When configuring AMD Quark quantization exclude lists, users specify layer patterns with or without parameter suffixes (.weight). vLLM's internal matching performed raw string equality, causing layers designated for exclusion to be inadvertently quantized or triggering unhandled key errors during quantization mapping.

Architectural Fix: Normalized Quark exclude dictionary entries by stripping parameter suffixes prior to module tree matching, bringing Quark configuration behavior into alignment with standard PyTorch quantization conventions.