发现信息
- 作者:Amey Agrawal, Haoran Qiu, Junda Chen, Íñigo Goiri, Rayyan Shahid, Ramachandran Ramjee, Alexey Tumanov, Esha Choukse
- 年份:2026
- 发表源:待核验
- OpenAlex ID:https://openalex.org/W4408955204
- DOI:https://doi.org/10.1145/3815789.3827962
- arXiv:待核验
- 检索主题:memory-evaluation
- OpenAlex 引用数快照:41(2026-11-12)
摘要线索
Deploying million-token Large Language Models (LLMs) is challenging because production workloads are highly heterogeneous, mixing short queries and long documents. This heterogeneity, combined with the quadratic complexity of attention, creates severe convoy effects where long-running requests stall short, interactive ones, degrading system responsiveness. We present Medha, a serving system that eliminates these convoys by introducing fine-grained, preemptive scheduling to LLM inference. Medha makes preemption practical with a co-designed set of mechanisms – including Adaptive Chunking and Stream Pipeline Parallelism – that overcome the perceived inefficiencies and scaling challenges of chunking. Additionally, we present a new parallelism strategy KV-Cache Parallelism to reduce the decode latency and afford interactivity despite very long context. These mechanisms are orchestrated by a Length-Aware Relative Slack (LARS) scheduler, a deadline- and heterogeneity-aware scheduling policy that prevents both the convoy effect and the starvation that plagues simpler policies. Under a heterogeneous workload, Medha improves throughput by 5.7 × while reducing median and 90th-percentile latency by 30 × and 174 ×, respectively, compared to state-of-the-art non-preemptive systems.
核验与筛选
- 当前状态:待筛选、去重和一手来源核验。
- OpenAlex 元数据和引用数只用于发现,不作为正式引文或质量判断。
- 记录正式版本、venue、方法细节、实验协议和局限后,才迁入稳定论文卡片。