Clairvoyant: Predictive Request Scheduler for LLM Serving
Preprint: arXiv:2606.07248 · Stack Layer: Request Scheduling (L1)
Standard LLM serving proxies process incoming requests under First-Come-First-Served (FCFS), leading to head-of-line blocking when high-token generations stall short conversational queries. Clairvoyant is a backend-agnostic Go sidecar proxy that intercepts incoming HTTP/gRPC requests before GPU engines (Ollama, llama.cpp) with zero modification to upstream code. It evaluates a 19-feature XGBoost classifier exported to ONNX to infer generated token length in 0.029ms P99 (approximately 70,000× faster than generation time) and re-orders dispatch using Shortest-Job-First (SJF).
| Hardware & Testbed | Workload Scenario | Metric Evaluated | Observed Reduction | Status |
|---|---|---|---|---|
| GCP L4 (24GB) | Real trace replay (ShareGPT) | P50 Queue Latency | -83.6% | Measured |
| Bare-metal RTX 4090 (24GB) | Bursty synthetic arrivals | P50 Queue Latency | -70.0% to -76.0% | Measured |
| Apple Silicon M1 (16GB) | Edge LLM pipeline (llama.cpp) | P50 Generation Latency | -68.1% | Measured |
| Offline Evaluation | LMSYS-Chat-1M dataset | Pairwise Ranking Accuracy | 95.1% | Measured |
Curated synthetic instruction datasets (Alpaca-style) are unusable for length-predictor training. Models trained on Alpaca exhibited severe prediction collapse because GPT-generated responses suffer from artificial brevity bias. The classifier learned to rank based on synthetic instruction patterns rather than natural token distribution. True predictive ranking required training on uncurated multi-turn human conversation logs (ShareGPT and LMSYS-Chat-1M).