Independent Research Practice · Founded by Aravind Sundaresan

Engineering the LLM Serving & GPU Systems Stack with Complete Provenance.

An independent lab building, benchmarking and fixing LLM serving and GPU infrastructure, with every result published openly.

I investigate bottlenecks across the entire inference pipeline — from scheduling theory and cluster placement down to PagedAttention internals and CUDA memory bandwidth. Every number published here states the exact hardware, workload, and distinction between measured facts and modeled projections.

Provenance Explicit Hardware Baseline

Every throughput and latency number is anchored to specific bare-metal or cloud hardware (RTX 4090, GCP L4, Vast.ai A100/3090). No merged benchmarks.

Discipline Measured vs. Modeled

Trace-driven simulations and analytical projections are explicitly stated in the exact same sentence as the number — never disguised as measured results.

Rigor Disconfirmed Results Published

When a kernel slows down, when a stateless heuristic beats a learned model, or when dataset brevity skews a classifier, I publish the disconfirmed result as a primary technical finding.

End-to-End Systems Architecture

The LLM Serving & Compute Stack

Every project in this practice attacks a different layer of the same overarching challenge: making LLM systems fast, reliable, and honestly measured.

Request Scheduling Edge / Ingress Proxy

Shortest-Job-First dispatch using an ONNX-exported XGBoost classifier predicting output token length at 0.029ms P99 before GPU admission.

Measured: -83.6% P50 arXiv:2606.07248

Clairvoyant →

Cluster / GPU Scheduling Kubernetes Scheduler Plugin

Native K8s scheduler plugin in Go with Spot capacity filtering, cost-ceiling enforcement, and Python sidecar gRPC confidence scoring.

1,252.6 pods/s @ 0.98ms Kube-Sentinel →

Serving Control Plane vLLM Ingress & Resilience

Streaming admission gateway managing concurrency, queue deadlines, and HTTP 429 load shedding under high-concurrency overload.

48 Paired Runs (58k reqs) Aether Control →

Cross-Hardware Serving Disaggregated Prefill / Decode

Disaggregated serving across asymmetric GPU nodes (Step 4B: RTX 3090 prefill → RTX 4090 decode) with 100% token match and Step 3 FP8 crossover.

Step 4B Cross-Machine HeteroDisagg →

Training & Data Contracts TrainSight Preflight & GRPO

Published PyPI preflight gate (TrainSight) auditing truncation starvation and collator loss leakage, paired with instrumented GRPO LoRA training.

PyPI: trainsight-preflight TrainSight & GRPO →

Upstream Engine Fixes & RFCs vLLM & SGLang Core

Hardening production serving engines via upstream PRs and architectural proposals: HeteroDisagg ACD RFC, speculative decoding fallback, and ZMQ memory sweepers.

RFC + 4 Open PRs Upstream Contributions →

Empirical Highlights

Selected Systems Disclosures

A sample of verified benchmarks across my research projects. Every metric carries its exact testbed hardware and methodological caveats.

Measured Benchmark arXiv:2606.07248 GCP L4 · RTX 4090 · M1

Clairvoyant: Predictive Request Scheduler

Go sidecar proxy sitting before upstream backends (Ollama, llama.cpp) without backend modification. Evaluates a 19-feature XGBoost model exported to ONNX to infer output token lengths at 0.029ms P99 for Shortest-Job-First dispatch.

Benchmark Results (Published in arXiv:2606.07248)

P50 Latency (GCP L4, trace replay) -83.6%

P50 Latency (RTX 4090, burst) -70% to -76%

Pairwise Ranking (LMSYS-Chat-1M) 95.1%

Empirical Finding & Limitation

Curated instruction datasets (Alpaca-style) proved completely unusable as training data due to GPT-imposed brevity bias. Real-world multi-turn conversational traces (ShareGPT, LMSYS) were required to achieve valid ranking.

arXiv:2606.07248 GitHub Dossier →

Measured Live K8s Simulated Trace

Kube-Sentinel: Native K8s GPU Scheduler

Native Kubernetes scheduler plugin in Go implementing PreFilter, Filter, PreScore, Score, and Reserve hooks. Protects latency-critical workloads from Spot interruptions, enforces explicit cost ceilings, and couples with a Python sidecar for telemetry-confidence scoring with circuit-breaker fallback.

Measured vs. Modeled Disclosures

Throughput (Live GKE CPU Host) 1,252.6 pods/s @ 0.98ms P99

Observation Matrix Coverage 39-Case Native Validation

Daemon Termination Failover 0 Failed Bindings (Fallback Active)

Alibaba 32-Node Cost Simulation 46.3% Modeled Saving

Explicit Provenance Caveat

The 46.3% GPU cost reduction is derived from an offline trace-replay simulation on Alibaba 2018 cluster logs across 32 nodes. It is a modeled projection. The scheduler plugin's 1,252 pods/s throughput and fail-closed circuit breaker were benchmarked on physical cluster nodes.

GitHub Dossier →

Step 4B Cross-Machine Compression Crossover

HeteroDisagg: Asymmetric Hardware Serving

Disaggregates prefill and decode across disparate GPU architectures via a 5-stage validation ladder. Maps compute-dense GPUs (RTX 3090 / L40S) to prefill and high-bandwidth GPUs (RTX 4090 / 3090) to decode, bridged by HeteroKVConnector asymmetric TP resharding and in-flight FP8 wire quantization.

Empirical Silicon Evidence (Steps 0–4B)

Step 4B Cross-Machine Token Agreement 100% Free-Running Match

Step 4B Cosine Similarity (3090 → 4090) 99.999% (Δlogit ≤ 0.0439)

Step 3 FP8 Wire Crossover (500 Mbps, S=1024) +25.73ms Speedup (1.104x)

Wire Payload Reduction (FP8 Block) 2.0x Byte Reduction

Compression Crossover & Disconfirmed Result

Isolated FP8 quantization kernels add +11ms to +32ms overhead, making uncompressed FP16 strictly optimal on loopback links. However, on 500 Mbps network links at $S=1024$, FP8 wire compression achieves a net +25.73ms speedup (1.104x), establishing the exact interconnect crossover frontier.

GitHub Dossier →

PyPI v0.2.4 Measured Silicon (RTX 3090)

TrainSight: Training Input & Loss Guard

Tokenizer-aware preflight inspection auditing silent data-contract failures before GPU allocation: eliminates truncation starvation, collator prompt/padding loss leakage, empty target supervision, and template drift across SFT and multi-turn DPO pipelines.

Published Packaging & Measured Impact

Official Distribution pip install trainsight-preflight

SFT Batch Padding Waste (RTX 3090) -25.76% ± 0.45% Padding Tokens

Useful-Input Token Throughput +29.28% ± 0.65% Higher Throughput

Loss-Mask Reconciliation Zero Mask Leakage (Labels != -100)

Guarding Silent Objective Corruption

Standard pipelines compile valid PyTorch tensors while quietly supervising prompt tokens or empty answers (<|im_end|>). TrainSight's in-memory batch reconciler enforces strict causal loss contracts before burning thousands of GPU-hours.

PyPI Package GitHub Dossier →

Production Engine Hardening

Upstream Contributions (vLLM & SGLang)

Four open PRs, plus one co-authored contribution — I fix race conditions, memory leaks, and architectural oversights directly inside the core engines the rest of the industry runs on.

vLLM #56964 Status: Open Speculative Decoding

Marlin quantization-backend fallback for unquantized draft layers

Fixed a crash where Marlin (quantization-only) backends couldn't execute unquantized draft-model layers in speculative decoding; added a dynamic fallback check to auto.

View PR #56964 →

SGLang #39741 Status: Open Kernel Divisibility

Marlin tile-divisibility fallback for Qwen3.5 GDN hybrid models

Fixed a crash loading Qwen3.5 GDN hybrid models from a silently-violated Marlin tile-divisibility requirement; added pre-flight tile dimension validation.

View PR #39741 →

SGLang #39623 Status: Open Disaggregated Serving

O(1) TTL sweeper fixing a memory leak in prefill/decode disaggregation

Fixed unbounded memory growth in prefill/decode disaggregation from orphaned ZMQ bootstrap-room state; added an active O(1) min-heap TTL sweeper.

View PR #39623 →

SGLang #37067 Status: Open RadixCache Hygiene

RadixCache.total_size() refactored from O(N) to O(1)

Maintainer-confirmed code-hygiene improvement, not a hot-path performance fix — eliminates recursive tree traversal on radix cache telemetry queries.

View PR #37067 →

vLLM #51574 Co-Authored v1 Engine Queue

Preempted Requests Starvation & Re-Admission Bug

Filed original bug (#41951) and wrote initial fixes (#41952, #56843); unit tests cherry-picked into this consolidated PR with co-authorship credit.

View PR #51574 →

View all 5 items — 4 open PRs, 1 co-authored →

Practice Principal

Aravind Sundaresan

Founder & LLM Inference / ML Infrastructure Engineer · Anavind Labs (Independent Research Practice)

With an engineering background rooted in large-scale distributed systems at Microsoft and Amazon, I founded Anavind Labs to investigate, benchmark, and harden the core LLM serving and GPU infrastructure stack.

Having designed systems ranging from multi-service enterprise governance platforms to real-time hardware test schedulers, I bring the same operational rigor to AI runtimes: declare every hardware baseline, distinguish empirical telemetry from analytical models, and treat negative results as primary systems insights.

Contact (aravind@anavindlabs.com) LinkedIn Profile Read Practice Profile →