Skip to content

zenrran4nlp/Awesome-LLM-Inference-Serving

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
img
img
 
 
README.md
README.md
 
 

Repository files navigation

Awesome LLM Inference Serving

This repository contains the literature referenced in Taming the Titans: A Survey of Efficient LLM Inference Serving, and will be updated regularly.

Table of Contents

LLM Inference Serving in Instance

Model Placement

Model Parallelism

  • GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism [arxiv 2019.7] paper
  • PipeDream: Fast and Efficient Pipeline Parallel DNN Training [arxiv 2018.6] paper
  • Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM [arxiv 2021.4] paper code
  • Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism [arxiv 2020.3] paper code
  • Reducing Activation Recomputation in Large Transformer Models [arxiv 2022.5] paper
  • NVIDIA [2024] paper
  • Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity [arxiv 2022.6] paper code

Offloading

  • ZeRO-Offload: Democratizing Billion-Scale Model Training [arxiv 2021.1] paper
  • DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale [arxiv 2022.6] paper code
  • FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU [arxiv 2023.6] paper code
  • PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU [arxiv 2024.12] paper
  • TwinPilots: A New Computing Paradigm for GPU-CPU Parallel LLM Inference [SYSTOR 2024] paper
  • Improving Throughput-oriented LLM Inference with CPU Computations [PACT 2024] paper

Request Scheduling

Inter-Request Scheduling

  • Orca: A Distributed Serving System for Transformer-Based Generative Models [OSDI 2022] paper
  • DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency [arxiv 2024.8] paper
  • Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving [arxiv 2024.7] paper code
  • Shinjuku: Preemptive Scheduling for μsecond-scale Tail Latency [NSDI 2019] paper code
  • Fast Distributed Inference Serving for Large Language Models [arxiv 2024.9] paper
  • Efficient LLM Scheduling by Learning to Rank [arxiv 2024.8] paper code
  • Don't Stop Me Now: Embedding Based Scheduling for LLMs [arxiv 24.10] paper
  • Prophet: An LLM Inference Engine Optimized For Head-of-Line Blocking [scs.stanford.edu 2024] paper
  • The Effect of Scheduling and Preemption on the Efficiency of LLM Inference Serving [arxiv 2024.11] paper
  • BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching [arxiv 2025.1] paper

Intra-Request Scheduling

  • Orca: A Distributed Serving System for Transformer-Based Generative Models [OSDI 2022] paper)
  • DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference [arxiv 2024.1] paper code
  • Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve [OSDI 2024] paper
  • Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving [arxiv 2025.3] paper

Decoding Length Prediction

Exact Length Prediction

  • Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction [ICWS 2024] paper
  • Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads [arxiv 2024.1] paper
  • Efficient interactive llm serv ing with proxy model-based sequence length predic tion [arxiv 2024.11] paper code

Range-Based Classification

  • Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline [NeurIPS 2023] paper code
  • S3: Increasing GPU Utilization during Generative Inference for Higher Throughput [NeurIPS 2023] paper
  • Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing [arxiv 2025.1] paper
  • DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency [arxiv 2024.8] paper
  • Power-aware Deep Learning Model Serving with μ-Serve [ATC 2024] paper
  • Don't Stop Me Now: Embedding Based Scheduling for LLMs [arxiv 24.10] paper
  • SyncIntellects: Orchestrating LLM Inference with Progressive Prediction and QoS-Friendly Control [IWQoS 2024] paper

Relative Ranking Prediction

  • Efficient interactive llm serv ing with proxy model-based sequence length predic tion [arxiv 2024.11] paper code
  • Efficient LLM Scheduling by Learning to Rank [arxiv 2024.8] paper code
  • SkipPredict: When to Invest in Predictions for Scheduling [arxiv 2024.2] paper
  • BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching [arxiv 2025.1] paper
  • Predicting LLM Inference Latency: A Roofline-Driven MLMethod [NeurIPS 2024] paper

KV Cache Optimization

Memory Management

  • Efficient Memory Management for Large Language Model Serving with PagedAttention [arxiv 2023.9] paper code
  • Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache [arxiv 2024.7] paper
  • FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines [arxiv 2024.3] paper
  • LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management [arxiv 2024.10] paper
  • KunServe: Elastic and Efficient Large Language Model Serving with Parameter-centric Memory Management [arxiv 2024.12] paper
  • SYMPHONY: Improving Memory Management for LLM Inference Workloads [arxiv 2024.12] paper
  • InstCache: A Predictive Cache for LLM Serving [arxiv 2024.11] paper
  • PQCache: Product Quantization-based KVCache for Long Context LLM Inference [arxiv 2025.3] paper
  • InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management [arxiv 2024.6] paper

Reuse Strategies

  • Efficient Memory Management for Large Language Model Serving with PagedAttention [arxiv 2023.9] paper code
  • MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool [arxiv 2024.12] paper
  • Preble: Efficient Distributed Prompt Scheduling for LLM Serving [arxiv 2024.10] paper code
  • Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention [ATC 2024] paper
  • GPTCache: An Open-Source Semantic Cache for LLM Applications Enabling Faster Answers and Cost Savings [NLP-OSS 2023] paper
  • SCALM: Towards Semantic Caching for Automated Chat Services with Large Language Models [arxiv 2024.5] paper

Compression Techniques

  • Model Compression and Efficient Inference for Large Language Models: A Survey [arxiv 2024.2] paper
  • FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU [arxiv 2023.6] paper code
  • KIVI : Plug-and-play 2bit KV Cache Quantization with Streaming Asymmetric Quantization [2024] paper
  • MiniCache: KV Cache Compression in Depth Dimension for Large Language Models [arxiv 2024.9] paper code
  • AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration [MLSys 2024] paper code
  • Atom: Low-bit Quantization for Efficient and Accurate LLM Serving [arxiv 2024.4] paper
  • QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving [arxiv 2024.5] paper code
  • CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving [arxiv 2024.7] paper code

PD Disaggregation

  • DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving [OSDI 2024] paper code
  • Splitwise: Efficient Generative LLM Inference Using Phase Splitting [ISCA 2024] paper
  • DéjàVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving [arxiv 2024.3] paper
  • Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving [arxiv 2024.7] paper code
  • Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads [arxiv 2024.1] paper
  • P/D-Serve: Serving Disaggregated Large Language Model at Scale [arxiv 2024.8] paper

LLM Inference Serving in Cluster

Cluster Optimization

Architecture and Optimization for Heterogeneous Resources

  • Sia: Heterogeneity-aware, goodput-optimized ML-cluster scheduling [SOSP 2023] paper
  • Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow [ASPLOS 2025] paper code
  • LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization [arxiv 2024.3] paper code
  • HexGen: Generative Inference of Large Language Model over Heterogeneous Environment [ICML 2024] paper code
  • Splitwise: Efficient generative llm inference using phase splitting [ISCA 2024] paper
  • DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving [OSDI 2024] paper code
  • HexGen-2: Disaggregated Generative Inference of LLMs in Heterogeneous Environment [ICLR 2025] paper
  • Optimizing llm inference clusters for enhanced performance and energy efficiency [TechRxiv 2024.12] paper

Service-Aware Scheduling

  • DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency [arxiv 2024.8] paper
  • Splitwise: Efficient generative llm inference using phase splitting [ISCA 2024] paper

Load Balancing

  • Orca: A Distributed Serving System for Transformer-Based Generative Models [OSDI 2022] paper
  • Efficient Memory Management for Large Language Model Serving with PagedAttention [SOSP 2023] paper code
  • DeepSpeed-MII code

Heuristic Algorithm

  • Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving [arxiv 2024.6] paper
  • A Unified Framework for Max-Min and Min-Max Fairness With Applications [TNET 2007.8] paper
  • Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs [arxiv 2024.10] paper

Dynamic Scheduling

  • Llumnix: Dynamic Scheduling for Large Language Model Serving [OSDI 2024] paper code

Intelligent Predictive Scheduling

  • Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing [arxiv 2024.8] paper

Cloud-Based LLM Serving

Deployment and Computing Effective

  • SpotServe: Serving Generative Large Language Models on Preemptible Instances [ASPLOS 2024] paper code
  • ServerlessLLM: Low-Latency Serverless Inference for Large Language Models [OSDI 2024] paper code
  • Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity [arxiv 2024.4] paper code
  • Characterizing Power Management Opportunities for LLMs in the Cloud [ASPLOS 2024] paper
  • Predicting LLM Inference Latency: A Roofline-Driven ML Method [NeurIPS 2024] paper
  • Distributed Inference and Fine-tuning of Large Language Models Over The Internet [NeurIPS 2023] paper

Cooperation with Edge Device

  • EdgeShard: Efficient LLM Inference via Collaborative Edge Computing [JIOT 2024.12] paper
  • PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Services [arxiv 2024.5] paper
  • Hybrid SLM and LLM for Edge-Cloud Collaborative Inference [EdgeFM 2024] paper
  • Large Language Models (LLMs) Inference Offloading and Resource Allocation in Cloud-Edge Computing: An Active Inference Approach [TMC 2024.12] paper

Emerging Scenarios

Long Context

Parallel Processing

  • LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism [SOSP 2024] paper code

Attention Computation

  • Ring Attention with Blockwise Transformers for Near-Infinite Context [arxiv 2023.10] paper code
  • Striped Attention: Faster Ring Attention for Causal Transformers [arxiv 2023.10] paper code
  • Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache [arxiv 2024.1] paper
  • InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference [arxiv 2024.9] paper

KV Cache Management

  • Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache [arxiv 2024.1] paper
  • InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management [OSDI 2024] paper
  • Marconi: Prefix Caching for the Era of Hybrid LLMs [MLSys 2025] paper code

RAG

Workflow Scheduling

  • PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design [arxiv 2024.3] paper code
  • Teola: Towards End-to-End Optimization of LLM-based Applications [ASPLOS 2025] paper code
  • Accelerating Retrieval-Augmented Language Model Serving with Speculation [arxiv 2024.1] paper
  • RAGServe: Fast Quality-Aware RAG Systems with Configuration Adaptation [arxiv 2024.12] paper

Storage Optimization

  • RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation [arxiv 2024.4] paper
  • Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection [arxiv 2024.5] paper
  • CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion [EuroSys 2025] paper code
  • EPIC: Efficient Position-Independent Context Caching for Serving Large Language Models [arxiv 2024.10] paper

MoE

  • A Survey on Inference Optimization Techniques for Mixture of Experts Models [arxiv 2024.12] paper code

Expert Placement

  • Tutel: Adaptive Mixture-of-Experts at Scale [MLSys 2023] paper code
  • DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale [ICML 2022] paper code
  • FastMoE: A Fast Mixture-of-Expert Training System [arxiv 2021.3] paper code
  • GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding [ICLR 2021] paper code

Expert Load Balancing

  • Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference [arxiv 2023.3] paper
  • Optimizing Dynamic Neural Networks with Brainstorm [OSDI 2023] paper code
  • Lynx: Enabling Efficient MoE Inference through Dynamic Batch-Aware Expert Selection [arxiv 2024.11] paper
  • Mixture-of-Experts with Expert Choice Routing [NeurIPS 2022] paper

All-to-All Communication

  • Tutel: Adaptive Mixture-of-Experts at Scale [MLSys 2023] paper code
  • Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling [arxiv 2024.10] paper
  • Accelerating Distributed MoE Training and Inference with Lina [USENIX ATC 2023] paper

LoRA

  • LoRA: Low-Rank Adaptation of Large Language Models [ICLR 2022] paper code
  • LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models [ICLR 2024] paper code
  • QLoRA: Efficient Finetuning of Quantized LLMs [NeurIPS 2023] paper code
  • CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference [arxiv 2024.1] paper
  • dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving [OSDI 2024] paper code

Speculative Decoding

  • Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding [ACL 2024] paper code
  • OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure [TACL 2025] paper
  • SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification [ASPLOS 2024] paper code

Augmented LLMs

  • InferCept: Efficient Intercept Support for Augmented Large Language Model Inference [ICML 2024] paper code
  • Fast Inference for Augmented Large Language Models [arxiv 2024.10] paper
  • Parrot: Efficient Serving of LLM-based Applications with Semantic Variable [OSDI 2024] paper code

Test-Time Reasoning

  • Test-Time Compute: from System-1 Thinking to System-2 Thinking [arxiv 2025.1] paper code
  • Efficiently Serving LLM Reasoning Programs with Certaindex [arxiv 2024.10] paper code
  • Learning How Hard to Think: Input-Adaptive Allocation of LM Computation [ICLR 2025] paper

Miscellaneous Areas

Hardware

  • Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching [arxiv 2024.10] paper
  • Efficient LLM inference solution on Intel GPU [arxiv 2024.1] paper
  • LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Services [arxiv 2024.10] paper
  • Demystifying Platform Requirements for Diverse LLM Inference Use Cases [arxiv 2024.1] paper code
  • Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs [arxiv 2024.7] paper
  • LLM as a System Service on Mobile Devices [arxiv 2024.3] paper
  • Fast On-device LLM Inference with NPUs [arxiv 2024.12] paper

Privacy

  • A First Look At Efficient And Secure On-Device LLM Inference Against KV Leakage [MobiArch 2024] paper
  • No Free Lunch Theorem for Privacy-Preserving LLM Inference [AIJ 2025.4] paper
  • MPC-Minimized Secure LLM Inference [arxiv 2024.8] paper

Simulator

  • Vidur: A Large-Scale Simulation Framework For LLM Inference [MLSys 2024] paper code
  • Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow [ASPLOS 2025] paper code

Fairness

  • Fairness in Serving Large Language Models [OSDI 2024] paper code

Energy

  • Towards Sustainable Large Language Model Serving [HotCarbon 2024] paper

Reference

We would be grateful if you could cite our survey in your research if you find it useful:

@misc{zhen2025tamingtitanssurveyefficient,
      title={Taming the Titans: A Survey of Efficient LLM Inference Serving}, 
      author={Ranran Zhen and Juntao Li and Yixin Ji and Zhenlin Yang and Tong Liu and Qingrong Xia and Xinyu Duan and Zhefeng Wang and Baoxing Huai and Min Zhang},
      year={2025},
      eprint={2504.19720},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2504.19720}, 
}

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

50 stars

Watchers

4 watching

Forks

7 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors