Add comprehensive analysis of LLM training/inference pain points with detailed statistics and citations

[Copilot]

Researched GitHub issues across major AI/ML repositories to identify pain points in LLM training and inference, then developed a positioning strategy for Probing to attract project traffic.

Analysis Document

Created docs/analysis/llm-training-inference-painpoints.md covering:

Comprehensive Data Analysis (25 High-Engagement Issues)

• Statistical analysis: 2,746 total comments, 900 reactions across 25 issues
• Topic distribution: Hardware (24%), Model Support (20%), Training (16%), Distributed (12%), Memory (12%)
• Engagement metrics: Average 109.8 comments/issue, 36 reactions/issue
• Timespan: November 2022 - October 2025

Pain Points Identified (with Citations and Links)

• Memory management: GPU OOM, CPU leaks, multi-GPU imbalances (highest priority)
  • Flux LoRA OOM (489 comments) - bmaltais/kohya_ss #2701
  • Ray OOM in RLHF (47 comments) - volcengine/verl #429
  • vLLM 5x memory increase (92 comments) - ollama/ollama #9457
• Distributed training: Communication bottlenecks, parallel strategy integration complexity
  • Context Parallel bug (47 comments) - NVIDIA/Megatron-LM #673
  • FSDP RFC (82 comments, 65 reactions) - pytorch/pytorch #114299
  • Multi-GPU support (92 comments) - unslothai/unsloth #2435
• Hardware compatibility: New GPU support delays, cross-platform fragmentation
  • Qwen2-VL support (363 reactions) - ggml-org/llama.cpp #9246
  • RTX 5080/5090 (134 comments, 41 reactions) - vllm-project/vllm #14452
  • ROCm Windows (82 comments, 58 reactions) - pytorch/pytorch #106608
• Performance tuning: Bottleneck identification with high profiler overhead (5-20%)
  • DeepSeek V3 optimization (52 comments, 102 reactions) - sgl-project/sglang #2591
  • llama.cpp server improvements (123 comments, 18+ months) - ggml-org/llama.cpp #4216
  • Qwen3 best practices with 10x speedup data - modelscope/ms-swift #4030

Enhanced Documentation Features

• Complete reference table with all 25 analyzed issues including direct GitHub links
• Original user quotes with proper attribution for each pain point
• Quantitative performance comparisons from real cases (e.g., "Megatron-LM 10x faster than DeepSpeed")
• Detailed methodology section explaining data collection and search criteria
• Statistical breakdown by topic with engagement metrics

Probing Value Proposition

• Zero code modification via dynamic injection vs. traditional profilers requiring instrumentation
• <1% overhead for production use vs. 5-20% typical profiler cost
• Native distributed support with SQL-queryable cross-node metrics
• Interactive Python REPL for live process inspection

Traffic Attraction Strategy

• Target personas: LLM training engineers, MLOps engineers, algorithm researchers
• Content: Technical blog posts solving specific pain points (OOM debugging, distributed slowdowns)
• Community: Issue responses, case studies, workshops
• Partnerships: PyTorch ecosystem, hardware vendors, cloud providers

Deliverables

• Concrete use cases with command examples for memory debugging, performance optimization, distributed training
• Competitive analysis vs. PyTorch Profiler, Nsight Systems, TensorBoard
• 3-6-12 month roadmap with success metrics (GitHub stars, monthly active users, enterprise adoption)
• Comprehensive appendix with complete issue list, statistical tables, and data sources

Document provides actionable positioning based on rigorous analysis of real user pain points with full citations and quantitative backing.

Original prompt

我想搜索github的issue，看大模型的训练和推理中主要的痛点是什么，然后决策如何引入probing，为项目引流

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[reiase]

@copilot 我认为缺乏对真实数据的统计与案例典型引用

[Copilot]

@copilot 我认为缺乏对真实数据的统计与案例典型引用

已经补充了详细的统计数据和案例引用。主要更新：

统计数据：

• 分析了25个高参与度issues（2,746条评论，900个反应）
• 按主题分类统计：硬件24%、模型支持20%、训练16%、分布式12%、内存12%
• 平均参与度：109.8评论/issue，36反应/issue

完整引用：

• 添加了所有25个issues的完整引用表格，包含仓库、issue编号、标题、参与度和直接链接
• 每个痛点都有具体的issue引用和用户原始痛点描述
• 添加了定量数据来源说明（如"Megatron-LM比DeepSpeed快10倍"来自ms-swift #4030）

典型案例：

• Flux LoRA OOM：489评论，链接到bmaltais/kohya_ss #2701
• Ray OOM：47评论，包含内存使用率从20%到100%的监控数据
• DeepSeek V3优化：52评论102反应，详细优化清单
• Qwen3训练：性能对比表格，9.6s/it vs 91.2s/it

详见commit af774a7