发现信息
- 作者:Amey Agrawal, Nitin Kedia, Anmol Agarwal, Jayashree Mohan, Nipun Kwatra, Souvik Kundu, Ramachandran Ramjee, Alexey Tumanov
- 年份:2026
- 发表源:待核验
- OpenAlex ID:https://openalex.org/W4414689486
- DOI:https://doi.org/10.1145/3815789.3827963
- arXiv:待核验
- 检索主题:agent-memory
- OpenAlex 引用数快照:14(2026-11-12)
摘要线索
The rapid evolution of Large Language Model (LLM) inference systems has led to significant improvements in serving efficiency. However, our systematic analysis reveals that current evaluation methodologies suffer from fundamental flaws that mask critical performance issues and impede scientific progress. Through a comprehensive examination of recent systems, we identify recurring pitfalls in evaluation practices across three key dimensions: Baseline Fairness, Evaluation Setup, and Metric Design. These pitfalls are uniquely problematic for LLM inference due to its dual-phase nature combining prefill and decode operations, heterogeneous workloads, and strict temporal requirements. We demonstrate how implementation differences frequently obscure algorithmic contributions, how workload selection often fails to represent production scenarios, and how metric normalization can hide substantial performance variability that directly impacts user experience. Traditional metrics derived from the classical machine learning serving setup fail to capture the autoregressive generation characteristics essential to LLM inference, leading to misleading conclusions about system performance. To address these challenges, we provide a comprehensive checklist of best practices and methodological recommendations for robust LLM inference evaluations. To demonstrate the practical application of our methodology, we present a case study of speculative decoding evaluation - a technique whose non-uniform token generation patterns with bursts during accepted predictions and delays during verification failures fundamentally confound conventional metrics. Our work establishes a rigorous foundation for evaluation methodology that can accelerate progress in LLM inference systems by ensuring meaningful comparisons, reproducible results, and metrics that reflect real-world requirements.
核验与筛选
- 当前状态:待筛选、去重和一手来源核验。
- OpenAlex 元数据和引用数只用于发现,不作为正式引文或质量判断。
- 记录正式版本、venue、方法细节、实验协议和局限后,才迁入稳定论文卡片。